Imagine that different types of data are different people that you are interviewing for a job. The job is to answer a specific research question. You want to know what their qualifications are—will they do a good job? Are their strengths a good match for the task? Like we’ve mentioned before, if there are issues with the underlying data, then there will be issues with the results. It’s important to consider if the data collected make sense for the question that was asked.
Today, the data “qualifications” we’ll be looking at are what kind of data were collected and how the data were collected. Next week, in part two of this post, we’ll cover definitions and what it means for data to be representative. Are you ready to get in there and interview some data? Come on, it’ll be fun!
1) What data were collected?
The data collected need to be appropriate for the research question. One step of deciding what data are most appropriate is selecting quantitative or qualitative data. Quantitative research is concerned with things we can count. How many books were checked out last year? Of all the people who have a library card, how many of them checked out at least one item last year? Qualitative research is concerned with capturing people’s experiences and perspectives. Why did someone check out an item? How did having that item make them feel? Qualitative research often aims to give us insight into the thinking or feeling behind an action.
Which type of data is most appropriate depends on the research question. If the researchers want to know how many children completed a summer learning program, the quantitative data from registration and completion work well. If researchers want to know what children got out of their experience, it would be more appropriate to collect qualitative data on a question like, “Please tell us more about your experience with our summer programming. What did you enjoy? What could be improved?”
What doesn’t work is if the researcher wants to know about the thinking behind an action, but asks for a number. Or if the researcher wants to know a number, but asks participants how they felt about their experience. The key is for the research question and the data collected to be a good match.
2) How were the data collected?
Once researchers figure out what they want to know, they have to decide how to collect the data. There are many ways to collect data—too many for this post. A few common ways that data are collected in libraries are to ask people (such as with surveys or interviews), observe people, or evaluate something they created. Each of these strategies is more appropriate in some situations than others.
Asking is great when you want to understand someone’s experience. People understand what’s going on internally for them. An example is asking participants what they wanted to get out of an experience and what they felt like they did get out of it. They are the experts on that. Asking, in particular using surveys, is also an effective strategy for collecting a large amount of information relatively quickly.
There are a few challenges with asking people. One issue is that we all want to feel good about ourselves and look good to others. Even if I receive an anonymous survey in the mail, I may put down an inaccurate answer if it fulfills these desires. We call this social desirability bias.
For example, if you want to know how many books I checked out last year, I would be happy to tell you. The problem is that I don’t remember. Maybe 50? Maybe 100? Or maybe I do remember, but I want to tell you it was 500 because that makes me feel good.
The second issue is that sometimes we simply don’t know, but we’ll guess if you ask us to. Our memory is unreliable. When asking people to report on something, it needs to be something that they can report relatively easily and accurately. A final issue is that sometimes people don’t understand the question as the researchers intended it.
Researchers keep these challenges in mind so they can take steps to mitigate them. Some common practices are to avoid asking questions that are highly sensitive, asking about things that are easy to remember, and testing surveys and interview questions in advance for understanding.
Observation enables researchers to directly witness people’s behaviors and interactions, rather than relying on their self-report. This can be very useful in situations where people don’t know or struggle to articulate the information researchers are trying to collect.
For example, let’s say a researcher wants to know more about children’s experiences at a storytime. Children are still learning words to express how they feel. If children who attended storytime were asked what they got out of the experience, they might say it was fun. They are probably not going to say they worked on fine motor skills and social skills. An observer, however, can collect data about the activities during storytime, watch children’s interactions with each other, and their facial expressions.
The challenge with any observation is that the information is captured or analyzed by a person. We are all biased and subjective in unique ways, and notice some things and miss others. To mitigate this challenge, researchers use a structure or guide while observing and, when possible, more than one observer.
Creating something is a good way to collect data in situations where researchers are evaluating skills. Let’s say there was a knitting class at the library. The main goal was for participants to knit a hat. At the end of the class, every participant selects their best work and those hats all go on display in the library. Those hats are the data. To determine if the class’s goal was achieved, you would look at all the hats and decide if they were “successful” hats.
As you read research, take note of the reason why researchers chose a particular method of collecting data. Does their reasoning make sense? Do the strengths of that method match well with the research question? What’s important here again is the match between the research question and how the data were collected.
Wow! That was a lot. How does your brain feel? A little gooey? Very understandable. Remember, you need the right data for the job. What data were collected and how they were collected are two ways to see if the data are a good fit for the research question. We’ll cover the other data qualifications next week: definitions and what it means for data to be representative. And we’ll talk more about those hats!