Does the (Data Collection Method) Shoe Fit?

You wouldn’t go hiking in a pair of dress shoes, right? Like the variety of shoes in your closet, there are a variety of data collection methods in all different shapes and sizes. The trick is finding which data collection method fits! Today’s post will help you determine which method is best for your evaluation.

What are Data Collection Methods?

Data collection is the process of gathering information from different sources with the goal of answering a specific question (your evaluation question). The method, or procedure, that you use to collect your data is your data collection method. Four common ones are: surveys, interviews, focus groups, and observations.

  • Survey: questionnaires that ask respondents to answer a set of questions. While these questions can be closed or open-ended, the purpose of surveys is to collect primarily quantitative data. Surveys can be administered online, by phone, by text, or in print. 
  • Interview: a conversation between two people—an interviewer and an interviewee—during which the interviewer asks primarily open-ended questions. Interviews may occur face-to-face, on the phone, or online. Interviews provide qualitative data.
  • Focus group: a dialogue between a group of specifically selected participants who discuss a particular topic. A moderator leads the focus group. Focus groups provide qualitative data.
  • Observation: a person (the researcher or evaluator) observes events, behaviors, and other characteristics associated with a particular topic in a natural setting. The observer records what they see or experience. Observations may yield quantitative or qualitative data.  
How to Pick the Right Data Collection Method

By this point in your evaluation you should have: 

Determined the goals and scope of your evaluation

  Written your evaluation question(s)

If not, you can circle back to those posts here and here, respectively. Now you’re almost ready to start collecting data—the fun part! First you need to decide which data collection method to use. Take a look at the pros and cons of each data collection method in the chart below. Use this to help you narrow down which methods might fit your evaluation.

To further narrow down your data collection method search, ask yourself the questions below. Do your answers rule out any of the methods? Reference the pros/cons chart for help. 

  What is most essential to you? Consider whether it is important for you to answer questions of how and why (more likely qualitative data) or what, how often, and to what extent (easier with quantitative data). 

  What will you be asking? Complex topics may lend themselves better to methods that allow for follow-up questions. Taboo topics may require additional anonymity. Think about what methods will make your participants feel most comfortable and safe responding to you.

  What are your constraints? Be realistic about the amount of time and resources you have. Choose a method that meets those constraints.

Conclusion

If none of these methods seem to fit your needs, don’t be afraid to branch out and find a collection method that is best for you or take a mixed-methods approach and use multiple techniques! For some other interesting ideas, here’s some additional articles on a collaborative photography method, oral histories, and other creative evaluation methods.

In our next post we’ll start our deep dive into the most popular data collection method—surveys. Stay tuned!

The Dynamic Data Duo: Quantitative and qualitative data, part 2

In our last post we introduced you to the dynamic data duo—quantitative (number) and qualitative (story) data. Like any good superhero squad, each have their own strengths and weaknesses. Quantitative data can usually be collected and analyzed quickly, but can’t really yield nuanced answers. Qualitative data is great at that! However, it often takes a lot of time and resources to collect qualitative data. So just like Batman and Robin, who balance out each other’s strengths and weaknesses when they’re together, both can also have successful solo careers. This post will walk you through a simple process to determine which data hero is right for the job!

Step 1: What is your evaluation question?

Let’s say we’re doing an evaluation where we want to find out if attending storytime helps caregivers use new literacy skills at home. If we go up to every caregiver and simply ask them, we’ll get a lot of yes/no answers, but not a whole lot of details. For example, imagine if we asked you right now: “Is this blog series helping you use new evaluation skills at work?” You might respond: “Uh…I don’t know. Maybe?” It’s a hard question to answer accurately. Often the evaluation question is too complex to directly ask participants.

Step 2: Break your evaluation question down into simple questions. 

Imagine calling up the Justice League and asking, “Hey, can you save the world?” They might answer yes, but will we know if they have the right skills or perhaps have other plans today? Similarly, our evaluation questions are often broad and abstract. We can’t always ask it outright and get a useful answer. So let’s look at some ways we can break our evaluation question down into simpler questions. 

As a reminder, our evaluation question is “does attending storytime help caregivers use new literacy skills at home?” Go word by word and see if you can come up with additional questions that would break the concepts down further. For instance, “does attending…” What are we assuming/what don’t we know? 

  • Did the caregiver attend a storytime session? 
  • Why or why not?
  • How many times did a caregiver attend a storytime session?
  • Which storytime sessions did the caregiver attend? 

Continue on with the rest of the evaluation question, keeping in mind you might not come up with simpler questions for every word or phrase. 

“Caregivers”

  • Who are the caregivers? 
  • Were they already using the literacy skills taught during storytime at home prior to attending a storytime? 

“New literacy skills” 

  • Are caregivers learning new literacy skills during storytime? (If caregivers aren’t learning new literacy skills at storytime, they can’t then use those skills at home!)
  • Why or why not? 
  • What new skills are they learning? 
  • How many new skills are they learning?

“At home”

  • Do caregivers use new literacy skills from storytime at home? 
  • Why or why not?
  • How often do they use new literacy skills from storytime at home? 

Step 3: Determine if each sub-question can be answered with numbers or a story

Go back through your list of sub-questions and try to answer each one with a number. Can you do it? If so, the question would give you quantitative data. If not, it might be a qualitative question. 

Let’s look at the question, “What new literacy skills are caregivers learning during storytime?” We need words to answer this question, not numbers—right? Not necessarily. We could create a list of 10 literacy skills that we taught during storytime and ask caregivers to check which ones they learned. By creating these parameters, we’re limiting the response options to a finite quantity (10 possible choices) and can count how many people choose each skill. This process transforms what would be an open-ended question yielding qualitative data into a question yielding quantitative data. 

You can generally apply this process to questions that either have a finite number of options or where a likert scale is appropriate. However, there are numerous (no pun intended) cases where you’ll want more nuanced, qualitative answers. For instance, try answering the question, “Why did you attend storytime today?” with a number! We could still create a list of possible answers, but it’s likely that someone would look at those choices and feel like none of them really fit. If we want to better understand our caregivers’ reasoning, then we don’t want to limit their responses. We want a story—we want qualitative data.

Step 4: Batman or Robin? Or both?

Now that you’ve classified your questions as quantitative or qualitative, do you have the means (capacity, resources, etc.) to collect data on all of them? Remember the pros and cons of each data type and review which questions are most important to you. Are a majority of them qualitative or quantitative? Knowing which type of data you need to collect will help you decide which data collection method to use. Our next several blog posts will address the different data collection methods you can use and their pros and cons, so keep reading!

Ready to meet your (data) match? Introducing number data and story data

Shows two different shapes to illustrate the two different categories of data

Hey, there! Welcome to 2021! We’re glad to see you here. It’s a new year and we’re ready to dive into research methods. Not what you expected to rejuvenate you in 2021? Well, hold on—research methods are actually pretty rad. First, though, what are they? 

Research Methods

Research methods are the different ways we can do the research or evaluation. If you’ve already tried out our tips on doing desk research, you may have found that the data you need is just not out there. You’re going to have to collect some data yourself! 

What kind of data should you collect? Two very broad categories of data are quantitative and qualitative data. Quantitative data are numbers data and qualitative data are story data. Wait—isn’t all data numbers? Nope! Story data are real! 

Quantitative Data: how much or what extent

What kind of information can quantitative data provide? Think about questions that you could answer with a number. Here are some examples from libraries:

  • How many books were checked out this month?
  • How often did families attend more than one storytime in a month?
  • What times for storytime have the highest attendance?
  • What percentage of our patrons rely on mobile services for library access?

You can see from the examples that quantitative data can answer questions about how much, how often, what, and to what extent. Quantitative data can often be collected by consulting data you already track within your library or by distributing a survey. This data can generally be collected and analyzed relatively quickly. The downside to quantitative data is that it can’t tell you how or why something is a particular way. If you collect data on how often families attended more than one storytime in a month, you still don’t know why some families came more often. That’s where qualitative data comes in. 

Qualitative Data: why or how

What kind of information can qualitative data provide? Think about questions that are difficult to answer with a number. The questions below cover the same topics as the quantitative questions above, but approached in a qualitative way:

  • Why are some patrons super-users? 
  • Why do some families attend storytime once and never return?
  • What reasons other than convenience determine whether families attend storytime? 
  • How do patrons who use the mobile services feel about the library in general?

You can collect some qualitative data on surveys by asking open-ended questions. You also can collect qualitative data from observations, interviews, and focus groups. While it yields detailed information, qualitative data collection and analysis can be complex and time-consuming. These data don’t always yield information that is actionable right away. Going back to our storytime example, if we ask why some families attend storytime once and never return, we may get a lot of different answers and need to spend time looking for common themes. 

How to choose?

Now that you know what both types of data could look like, how do you decide what data is the best to collect for a project? Did you notice how those quantitative and qualitative questions matched up on similar topics? That was on purpose! Different types of data can give you insight into different aspects of your evaluation question. 

To get the most meaningful results, it’s a great idea to collect both quantitative and qualitative data for your project. They can work together to provide a more complete picture of the topic. An easy way to incorporate both is to create a survey that includes mostly quantitative questions, but also a few key qualitative questions.

Now, is it always realistic that your organization has time and capacity to collect both types of data? Not really, right? That’s ok. The most important thing is to match the kind of data you collect with your evaluation question. 

Conclusion

Now you have a basic idea of how quantitative and qualitative data are different and how they can be used to find out different kinds of information. In our next post, we’ll show you a simple process for breaking down your evaluation question into smaller questions and determining if you need to use quantitative or qualitative methods. 

 

RIPL Data Boot Camp Webinar Series

Is one of your new year’s resolutions to get your library’s data in shape? Then, spend the winter with the Research Institute for Public Libraries (RIPL) and participate in our Data Boot Camp Series! This free webinar series features curriculum from the RIPL 2020 national event. These will NOT be webinars where you listen to a talking head the whole time; instead, please come ready to participate in a variety of interactive learning activities, some of which will occur in small groups in breakout rooms.

Here is the schedule – go to https://ripl.lrs.org/ripl-data-boot-camp/ to learn more about each webinar and register:

January 27 (1:00-2:30 ET/10:00-11:30 PT): Observations: Data Hiding in Plain Sight

February 2 (1:00-2:30 ET/10:00-11:30 PT): Can You Hear Me Now? Communicating Data to Stakeholders

February 23 (1:00-2:30 ET/10:00-11:30 PT): Nothing for Us, Without Us: Getting Started with Culturally Responsive Evaluation

March 2 (2:00-3:30 ET/11:00-12:30 PT): Meaningful Metrics for Your Organization

March 16 (2:00-3:30 ET/11:00-12:30 PT): Evaluation + Culture = Change

March 24 (1:00-2:30 ET/10:00-11:30 PT): Inclusive Data and Community Engagement: New Roles for Libraries to Shape Knowledge Creation and Use

All webinars will be recorded.