Reading (and Recording) the Room: Focus Groups

Ready to polish up your people skills? This month we are taking a step back from analysis and turning to data collection again. In previous posts we touched on different data collection methods, and in May we discussed the process of coding qualitative data. Understanding the basics of qualitative analysis opens up a world of possibilities for evaluation, and it definitely helps to know what you are taking on before beginning qualitative research since coding is an extensive process. With this background on coding qualitative data hopefully you feel more equipped to begin collecting it. Today we will focus our attention on a qualitative data collection method that we have mentioned several times but have yet to delve into—focus groups!

Introductions

Focus groups consist of several selected participants that partake in an intentional conversation directed by a moderator in order to gather community input. They are used to answer open-ended questions and gain a deeper understanding from diverse viewpoints. In this post we will outline what a successful focus group looks like as well as acknowledge their limitations.

If you have not participated in or led a focus group before, here’s a general rundown of how they start. Once everyone is brought together in the same space (everyone includes a moderator, usually an assistant moderator or note taker, and the participants), the moderator facilitates introductions. Icebreakers and friendly chatter can help the participants feel more relaxed, which will lead to a more productive session. Ground rules that foster a respectful, safe environment should be clearly established by the moderator, and consent to participate is received from all participants. As the moderator, you want to be mindful of time during the introductions. Remember, the participants are spending their valuable time to be there, and you want to make sure the resources you spent to bring them together yield valuable insights. 

Moving Past the Small Talk

Jumping into your predetermined questions will begin the true discussion. Throughout the allotted time, moderators have the incredibly important job of fostering a safe environment for participants to speak candidly. The quality of data you collect hinges on the ability of the moderator to maintain control of the conversation while ensuring participants are at ease. A level of trust needs to be established to ensure that the participants are comfortable speaking their mind, which is essential to gather reliable data. Moderators can foster this environment through these ten actions:

  • Giving verbal and nonverbal listening cues
  • Asking follow up questions
  • Asking for clarification before assuming what someone means
  • Being flexible if important, unanticipated points arise 
  • Steering the conversation back on topic if it goes astray
  • Emanating confidence
  • Staying neutral 
  • Always being respectful of differing opinions
  • Ensuring everyone has an equal chance to share
  • Avoiding leading questions

If you find yourself moderating a focus group, remember that you are ultimately the leader of the group, which gives you the power to politely direct the conversation. This is particularly important if one participant is dominating the conversation and others are being left out. 

All in all, leading a focus group requires impeccable social and leadership skills. 

Active Listening 

Conducting a focus group can reveal surprising information and perspectives that may be crucial to your research but you would not have thought to include in an interview or survey on your own. Selecting a diverse group of participants can reveal aspects of your study you never knew you were missing. While conducting a single focus group still ranks low on the Community-based participatory research (CBPR) continuum, it can be one tool out of many to begin incorporating more CBPR practices into your library. 

Because you may learn something significant that steers your research, it is best to conduct a focus group early on, so it is easier to incorporate your findings into your research moving forward. The focus group questions should all directly relate to your purpose, but the unique and most advantageous aspect of a focus group is the social dynamic and dialogue that prompts complex idea sharing. 

Inhibitions

Before you get too excited by all the thought provoking feedback that focus groups can produce, make sure you are also aware of some of their pitfalls as well. For starters, focus groups are often too small to be a statistically significant sample size for the population. If you are hoping to generalize your data to a large population or to make transformative planning decisions, a focus group should not be your only data source, but it can be a powerful source of data when combined with other methods. 

Although a single focus group may take less time and money to conduct than five separate interviews, focus groups also take a lot of planning to be well executed and should not be used as a rushed way to collect data. Similarly, focus groups require a skilled moderator in order to collect reliable data, and not having the right person available could cause your focus group to be less effective.

Possibly the most important thing to consider is that focus groups cannot guarantee anonymity due to the number of participants listening to each other. Sensitive topics that participants may not want to discuss in front of others are not well suited for focus groups since people are likely to hold back and not share their true internal reactions. In any focus group, regardless of topic, the moderator’s and participants’ expressed reactions can significantly influence what is subsequently shared. Therefore, focus groups can be susceptible to producing a consensus that may in fact be misleading, otherwise known as groupthink. This is another reason focus groups are often used in conjunction with other data collection methods. 

Takeaways

Clearly, focus groups have their strengths and weaknesses, but if well conducted they can bring invaluable, diverse perspectives to your research. Don’t forget to record the session so you have a complete transcript available to code! To wrap up, here are five key points to remember about focus groups. 

  1. It is essential to have a skilled moderator who will build a safe sharing space. 
  2. Remember to gain consent from each participant and lay ground rules to ensure everyone’s contributions are respected.
  3. Focus groups are generally not made up of a statistically significant sample size and therefore data from a single focus group should not be generalized to a large population. 
  4. Make sure participants are aware that confidentiality cannot be guaranteed. 
  5. Conduct focus groups early on in your study so that the unique insights they bring can inform your work moving forward. 

Thanks for reading! Next, we will cover some important logistical points such as how to select participants and ask the right questions. If you have any questions or comments we would love to hear from you. You can contact us at LRS@LRS.org.

Mapping the Methods: Content Analysis Part 2

Welcome back! I am excited to dive back into content analysis with you. It is no secret that content analysis can be far from a walk in the park and is possibly more comparable to following a treasure map across a remote island. Therefore, I will fill this post with a review of what we have already discussed, the final steps for analysis, obstacles to be aware of along the way, and a few helpful hints. 

Returning to Coding

We left off last week on coding, where short labels are applied to qualitative data which represent meaning within this data. Returning to this process throughout your analysis allows you to condense repetitive codes and make sure you are heading in a direction that is consistent with your data as a whole to avoid bias and better answer your research questions (or find the treasure, so to speak). 

Coding is actually the culmination of two steps. First, decoding data to find meaning and second, encoding it by applying a word or phrase that represents this meaning. Keeping these two steps in mind helps demystify the process of coding as a whole. The codes you create and how you apply them will depend solely on your data and the questions you are trying to answer. When creating codes it is helpful to think of them as not only labels but also links that piece your data together. 

Coding is a key step that organizes your data for further analysis. Once codes are applied to your data, you are ready to begin the more abstract part of analysis by categorizing the codes and searching for themes. 

Categorization of Codes

Categorizing codes is essentially synthesizing them into your analysis by identifying patterns in the coded data. Categories for codes are phrases that encompass an idea which multiple codes fall within. While searching for patterns of underlying meaning in your data, remember that patterns often develop by grouping similarities but can also develop through grouping data by outliers, frequency, order, causes, or other relationships. These possible pattern configurations are all helpful tools for categorizing your coded data. 

To continue with our previous example, in a response to the question “Do you feel that the library is an essential community resource, why or why not?” the code “family support” may fall into the category “highly valued early learning programs.” Other codes such as children’s programs, reading development, and storytime may also fall within this category.

Data outliers should not be viewed as problems but as points of interest and discovery. In this case, evidence that some patrons raising children do not use the library’s early learning resources does not necessarily make the previously mentioned category (“highly valued early learning programs”) wrong, but may lead to a new category entirely or reveal how these programs are more accessible to certain patrons than others. 

Developing Themes

Themes are a more abstract level of insight content analysis might reveal, meaning they are general and applicable beyond a single study. Themes may not always evolve from your coding and that is OK. Codes and categories can still be informative and point to paths for further research to answer your key questions accurately if themes do not develop.

Themes develop when you identify consistent patterns that stretch across the coded and categorized data and bring insight to your main research question. Once you have charted these patterns it is time to start digging for the treasure! Themes are not found by leaping to conclusions and away from your data, they develop through careful analysis of codes and categories to triangulate meaning based on evidence.

If multiple categories point to it, a theme developed from our example study could be “early learning programs bring people together from across the community” This is a concrete answer to the overarching research question “How can libraries increase a population’s sense of community?” and it could be used to inform decision making on future programming in your library.

Obstacles

Content analysis is a time intensive and sometimes frustrating process. You must be willing to dedicate time and effort to it for your conclusions to be accurate and limit bias. Also, it focuses solely on the content of your data without taking into consideration outside factors such as societal context. This limited focus, and the reduction of data to codes, categories and themes, may allow nuances of meaning to be lost. Condensing the data can be problematic if important aspects of it are ignored, or it can be exactly what you need to do to find the buried answers you are looking for. 

A Helpful Hint!

Coding does not have to be a lonely process. In fact, collaborating with a team can help you navigate this work and make the whole journey more enjoyable. As we discussed in the last post, each person will not apply the same codes to each excerpt and that is OK. Being open to a range of perspectives will bring insights to the data that you may never see alone. The possibilities for coding are enormous and narrowly focusing on one route can obscure key information and get you stuck.

Finally, make sure to take careful notes of your process and the codes you use. This will be helpful for you to refer back to throughout your analysis, and it will be helpful for those you share your study with to understand the work you put into it! 

Conclusion

Initially, qualitative data may feel overwhelming or ambiguous, but coding provides a map for condensing the data until you can categorize it and find meaning based within the text. It is rewarding when themes appear that were initially buried in the data. As you are putting time and effort into it, make sure to keep reminding yourself of the research question, the importance of your work, and your end goal. You may uncover key information when you least expect it! Coding reappears in other methods for qualitative analysis, so be sure to keep this information in mind as we continue this chapter.

Mapping the Methods: Content Analysis Part 1

Hello data enthusiasts! Let’s return to our exploration of qualitative analysis. Last time we uncovered a few ways qualitative analysis can expand research findings by looking beyond number data for better insight on human experiences. Now I want to explore strategies for putting qualitative analysis into practice.

Concentrating on Content Analysis

As we discussed last time, qualitative analysis is flexible and adaptive to different types of data. As you may have already guessed, this means there are multiple methods for qualitative analysis depending on the kind of research you are conducting, the form of your data and the questions you are asking. Content analysis is one of many methods of qualitative analysis. It carefully filters, categorizes and condenses qualitative data sets (often text based) to discover hidden (or not-so-hidden) meanings! It is one of the most common methods of conducting qualitative analysis, and so it is a great place for us to start this chapter.

The key question that content analysis helps answer is, how do you categorize this textual data to best identify important patterns, anomalies, and relationships that answer your research question?

Planning for Success

You can visualize the content analysis process as following a treasure map where the treasure (buried in the data) is the insights your analysis will eventually reveal!

First, there’s quite a bit of preparation that needs to take place to ensure your analysis goes as smoothly as possible. For content analysis you should clearly identify the main question you want your data to answer. In other words, what is the treasure that you want to find? Content analysis is a long, strenuous process and having a specific goal will help direct you along the way. To build off our example from the last post, a survey that asks the open ended question, “Do you feel that the library is an essential community resource, why or why not?” may have a driving analysis question of, “How can libraries increase a population’s sense of community?”

However, as the analyst, you may now be staring at fifty lengthy responses, all of which have a person behind them with their own unique perceptions and experiences they want to share with you. You know there is useful information within the responses, and you want to make sure you are considering everyone’s responses by using correct research methods. 

This means it’s time to read your data, then read it again! While it takes both patience and time, this step can also streamline the rest of the process. You don’t need to read it with any specific goal in mind except to be open minded, consciously consider any biases you have, and take notes of your general impressions. Instead of fixating on specific responses try to take a step back and look at the data as a whole. You want to know your data thoroughly as you embark on the next step, just as you would want to know the map before setting off on an adventure!

Next Steps

Once you know your data backward and forward it is FINALLY time to start your content analysis with a method called coding. Coding is essentially categorizing the text with descriptive labels, or codes.

Coding has multiple steps, but the process is also repetitive and cyclical. Remember, you can always return to previous steps and adjust something so your analysis better encompasses the data. It’s unlikely you will find the treasure immediately, so always be willing to backtrack if necessary! 

Meaning Units

Before you create codes for your data, it may help to condense text into sections that hold meaning, called meaning units. Don’t let this step intimidate you. You still want these meaning units to be close to, if not literally, the text of the data. For example, perhaps someone’s response to our example question includes the sentence, “I’ve always enjoyed the library, but it was a particularly great resource for me while raising my children.” A condensed meaning unit you may take from this is “a great resource while raising my children.”

It is important that your meaning units relate as directly to the text as possible, and you are careful not to over-interpret or otherwise misrepresent the responses in the data set . 

Codes

You begin to interpret the data and take it to a slightly more abstract form with the next step, which is applying codes. An example of coding is taking the meaning unit “a great resource while raising my children” and assigning the label “family support.” If these are not the specific words you would use to describe this meaning unit that is OK. Codes will vary person to person and also change depending on the focus of your driving question.

As long as you work diligently to keep the coding faithful to the text, while acknowledging and limiting your bias from previous experiences on the subject, codes will not be right or wrong. Many codes will be used repetitively throughout your data analysis. You may assign the code “family support” to other meaning units from other survey participants if appropriate. There will likely be certain codes that are used often and other outlying codes that are not. There is also no right or wrong answer for the number of codes that you use. It depends on the size of your data set and the variations within it. 

It can be helpful to use your intuition while creating codes as long as you are still basing these labels in the text and staying aware of how your biases will affect their selection. Similarly, there may be aspects of a map you intuitively understand, but it wouldn’t be very smart to throw the map away entirely and assume you know the way yourself.   

While coding you may need to change meaning units that suddenly don’t make sense moving forward. Remember this backtracking is a normal part of the process to make sure the codes you are using reflect the whole of the data the best that they can. 

Conclusion

After creating codes and applying codes to your data your analysis is off to a good start! Stay tuned next week to learn where to go from here. We will explore categorizing the codes you create to find themes and finish your analysis! 

Here is a quick reflection on what this post covered:

  1. Content analysis is a common method for qualitative analysis that categorizes data to reveal key research findings.
  2. There is a lot of preparation involved in this long process. Take notes to track your work and know your research question and data thoroughly!
  3. Meaning units help you identify the meaningful parts of the text that you will code. 
  4. Codes are descriptive labels you apply to meaningful parts of your data to make sense of it all. 
  5. Your intuition can be helpful, but only if you are aware of how your biases may affect your analysis. Find a balance!

New Season, New Chapter: Qualitative Analysis

Hello Everyone! I am honored to have the opportunity of continuing the LRS series staple Between a Graph and a Hard Place. Our last post began with “happy fall,” and now we are well on our way to Colorado’s mud season. Along with the change in seasons, a shift in topics feels like a great way to begin the series anew.

We left off discussing the use of observation as an important method for data collection and how to observe as unobtrusively as possible. To begin this new chapter, let’s take a step back to explore a type of research where observation is a crucial tool: qualitative analysis. 

The importance of numbers in research is impossible to overstate. While they can still be misleading, poorly displayed or simply inaccurate, we can all agree there is something reassuring about having the number data to back up your assertions. However, in a world filled with as many unique human experiences as ours, numbers alone (meaning quantitative research) can’t always give comprehensive and nuanced answers to every question, and that is where qualitative analysis shines.  

Background

In previous posts we established that qualitative analysis delves into data derived from stories and answers questions such as “why” and “how.” Now, let’s dig a little deeper. Qualitative analysis is not only a tool for stories. It is used to examine survey responses, feedback from focus groups, narratives, anecdotes, social media posts, secondary and primary sources and even artwork! In fact, practically anything that includes human expressions, perceptions, emotions, assumptions and/or experiences can be analyzed qualitatively. As muddy as this may seem at first glance, understanding the potential for qualitative analysis through multiple mediums is crucial to identifying where and how you can incorporate it into your own research.

What you might have picked up on already is that, in its broadest and most simplified sense, qualitative analysis is used to make sense of data that is not numerical. Qualitative analysis is a tool for qualitative research where language and behavior are studied to find patterns and/or anomalies that convey information about a data set.

Helpful Aspects of Qualitative Analysis

The social sciences have applied qualitative analysis to their research for over a century, but until the middle to late 20th century, qualitative data collection and analysis were thought to conflict with quantitative research methods. One reason for this is that qualitative research, as opposed to quantitative research, accepts that the researcher is never fully objective and detached from the data being examined. When analyzing language and behavior, it is important for researchers to be aware of and limit their biases by understanding and reflecting upon how their own experiences and assumptions are a lens through which the data is viewed. The researcher will be an aspect of the study in a qualitative analysis. 

The fact that qualitative analysis plays by different rules than quantitative analysis does not lessen the value of the insights that qualitative analysis uncovers. Additionally, quantitative and qualitative research do not have to be at odds with each other. These methods can work together to provide a better picture of the phenomenon under investigation. Here is an example of how qualitative analysis can bring new insights to a study:  

Let’s say a Likert scale, a quantitative tool, is used in an attempt to assess the extent to which library patrons view their library as an essential community resource. Participants in the study are asked how they feel about the statement “libraries are an essential community resource,” and one of many patrons selects the answer “strongly agree.” This is considered quantifiable because their selection of “strongly agree” counts as a tally toward the total number of participants selecting this answer. However, when the same question is asked and the same participant has the opportunity to give a narrative response they write, “I agree with this statement, but the library is also where I found my community. I met my closest friends through library programs and afterwards became involved in community outreach through them.” 

Of course, this is only one response, and a detailed qualitative analysis would include an extensive data set. However, themes such as finding community and community outreach appear in this short answer and imply that the library builds community as much as it acts as a resource for it. If this is a consistent theme through multiple participant responses, it could be crucial enough to shift the focus of the study or open up avenues to find support for the library system in the future. 

Hopefully this sheds some light on how qualitative analysis can give you insights where quantitative analysis, when used alone, might fall short. Qualitative analysis can feel subjective and potentially problematic when compared to quantitative analysis, but the important thing to remember is that the two methods exist separately as apples and oranges. In fact, the entire order of the research can be turned upside down in qualitative analysis because it is not always necessary to start with a single hypothesis that you are attempting to prove or disprove. Qualitative analysis allows for more flexibility throughout a study because you analyze data as data collection is taking place, not only at the end as is done in quantitative analysis. 

Conclusion

If you’re feeling a bit lost, or are just swamped with things to do today and need something quick to skim. Here’s a recap of five points we just covered:

  1. There are times when your research questions will not be comprehensively answered by quantitative data, let qualitative data help! 
  2. Qualitative data takes many different forms. Widening your vision of what qualifies as “data” can reveal new opportunities for learning. 
  3. There are a variety of methods for analyzing qualitative data, but whether the researcher is using their intuition or a computer software, they will never be fully removed from the research findings.
  4. Qualitative and quantitative analysis are two completely different approaches to data, viewing one through the lens of the other will only lead to frustration.
  5. Qualitative analysis allows for more flexibility to shift focus throughout the study as the data is analyzed.

These parameters for qualitative analysis will be used throughout the rest of this chapter as we begin to answer questions such as, how do you analyze this data accurately? When is it appropriate to incorporate qualitative analysis into your research? And what limitations does qualitative analysis have? I hope to explore the answers with you in future posts, and in the meantime, happy spring!  

How to observe without being totally awkward

Dog looks worried or confused in office

Happy fall to all you data nerds out there! We appreciate you being here with us. Last time we discussed how to get permission from your participants when you want to do an observation. You might be wondering how you can actually do the observation without it being completely awkward and perhaps even cringey. Today we are going to discuss just that!

First let’s review our goal for this project: We want to evaluate if caregivers are learning skills during storytime and using those skills with their children outside of storytime 

Based on this goal, we decided to do observations of caregivers and children in the library while they are not participating in storytime. Ideally from a research perspective, we would observe them at home, but that would not be practical or comfortable for anyone involved. Even in the library context, we are going to need to be careful to make sure that our participants feel as comfortable as possible. 

Being a participant observer

There are a variety of ways you can behave as an observer. For most library situations, I recommend a version of what researchers call “participant observation.” You’re observing while still interacting with the people you’re observing to a limited extent. This setup feels more comfortable while still giving you, as the observer, some distance from what you are observing. What would this look like for our example project? When the family you are observing tells the children’s desk that they are in the library, you would first introduce yourself to the family. Then during the observation you would talk with them only if it’s really important or necessary.

When is it really necessary to jump out of observer mode? A classic example I lived through with a team of librarian-observers was a child in the group we were observing getting a serious nosebleed. At the time there was only one library staff member who was teaching the group, but three of us were observing. One of us stopped observing and took the child to get medical attention. The instructor who actually knew the content that needed to be covered continued running the group. My best advice for when to break out of observing “mode” is to try to avoid it, but trust yourself when it feels like an appropriate time to spring into action. You are probably right!

Making people feel comfortable

When observing, we’re trying to balance getting quality data with making the participants feel comfortable. Every population, and every individual, has different needs to feel comfortable. It can help to start by thinking back to times you were in potentially awkward situations and someone made you feel more comfortable. What did they do? Remember in this case we want to go a step beyond that and treat people how they want to be treated, not just how we would want to be treated.

In a situation like this with caregivers, we should definitely reassure them that the library staff is not there to judge them. Parents feel judged a lot! It’s helpful to emphasize that you are evaluating storytime and not them. Nonetheless, don’t tell participants “We’re looking to see what early literacy skills you use outside of storytime.” Then they will inevitably show you every early literacy skill they have ever heard of! Instead, you might explain the project like this: “We want to make storytimes better. To do that, we need to understand how caregivers and children are interacting outside of storytime. We are watching so we can learn and make storytime as helpful and fun as possible. We are not evaluating you as a parent. Do you have any questions? Is there anything else I can do that would make you feel more comfortable?”

Working with children 

Children are going to want to interact with you while you’re observing them, especially if they know you. You should explain to them what you’re doing and why you are acting differently. For example: “Today, my job is to be very quiet and pay attention really carefully to the fun time you are having with your caregiver. You can look at me and I’m going to smile at you, but I’m not going to talk with you like I usually would. It doesn’t mean I’m mad at you. I’m just really focused on watching and listening today. I’ll tell you when we’re done and we can talk more then. Do you have any questions? ”

Conclusion

Having these kinds of conversations with participants before you start will help the observation go well. We observed teens for a project once, who are perhaps the most self-conscious creatures on the face of the earth. The staff observers introduced themselves to the teens at the beginning of their time together even though we already had informed consent. Although I can’t know for sure, I think we were able to collect valuable data on that project partly because the observers introduced themselves right before the observations and were very friendly and open. Remember, you set the tone for the whole interaction at the beginning.

Up next 

Next time we’ll talk about how to focus your observations and collect data that will be valuable to you. We’ll look forward to seeing you then!

LRS’s Between a Graph and a Hard Place blog series provides instruction on how to evaluate in a library context. Each post covers an aspect of evaluating. To receive posts via email, please complete this form.

How to observe: Ask first!

A kitten peaks out from between books

Welcome back! We left off talking about why you would use observations to collect data. Observation can be a great data collection tool when you want to see how different people interact with each other, a space, or a passive program. Observation is also helpful when it is difficult for someone to answer a question accurately, like when you ask them to remember something they did or—particularly with children—if you ask them to give critical or written feedback, both of which can be developmentally inappropriate.

To review, our big research question is: “Does attending storytime help caregivers use new literacy skills at home?” Our small questions within that big question are:

  • Were caregivers already using literacy skills at home prior to attending a storytime? 
  • Are caregivers learning new literacy skills during storytime? 
  • Do caregivers use new literacy skills from storytime at home?

After thinking through these in our last post, we decided that it would be both helpful and realistic to observe caregivers and their children in the library. This won’t tell us about caregivers’ behavior at home, but it’s not an option for us to follow them around their homes. While observing caregivers in the library won’t tell us what they are doing at home, it still helps us see if they’re learning skills during storytime that they’re using outside of storytime.

Ok, so how would we actually do this? Here are the key elements of any observation:

  • Get permission from your participants
  • Decide how you will approach the observation
  • Focus in on some specific things you’re looking for
  • Take notes or videos of what you’re observing
  • Code the notes or video to identify patterns

For the rest of this blog post, we will focus on the first point. It’s important to get started on the right foot. 

Why is getting permission important? 

We’ve discussed informed consent before in this blog series, and it is still important with observations. If you came to your library and a staff member followed you around the whole time without an explanation, it would feel weird and even invasive. Now, you may be thinking, “Wait a minute, if people know we’re watching, they’re going to act differently!” You’re exactly right. But the reality is, people are going to know you’re observing them no matter what. Even if you think you could soundlessly skulk around the stacks, most people are going to sense something weird is going on. 

Remember–we don’t want to be creepy when we’re doing research! Watching people without asking their permission is 1) pretty creepy, 2) not good for building and maintaining trust with our users, and 3) violates one of our core library values of privacy. Ethically, we have to ask people if it’s ok to observe them while they’re in the library, even if it does change their behavior.

How do you ask permission to observe?

In this example, you could explain at storytime that you’re doing a research project to improve storytime. You are looking for some volunteers who spend time at the library outside of storytime to let you observe them interact with their child. If they are interested, you have a form they can sign, and the next time they’re hanging out at the library, you’d like them to come say “hi” at the desk in the children’s area. That way your staff know when they are in the library and the staff member observing them can introduce themselves too, which makes the whole thing less awkward.

As part of the informed consent process, remember that you need to address both that the participants can stop participating at any time, for any reason, and how you will protect their privacy. These elements are particularly important during an observation of caregivers and children. What identifying information do you absolutely need? The more anonymous the data can be, the better. Make sure you also establish a clear and easy way for the caregiver to end the observation while it is happening. 

In the next blog post in this series, we will explore the different approaches to observation. After you’ve gotten permission, how do you actually sit, watch, and collect meaningful data? Join us next time to find out!

Nothing About Us, Without Us: Equitable evaluation through community engagement

 

This is a “guest post” from the Colorado Virtual Library Equity, Diversity, and Inclusion blog.

When you wake up, one of the first things you might do is open your weather app to see what the temperature is and if it’s supposed to rain that day. You then use that information—or data—to make important decisions, like what to wear and whether you should bring an umbrella when you go out. The fact is, we are all collecting data every day—and we use that data to inform what we do next.

It’s no different in libraries. We collect data about circulation, program attendance, the demographics of our community, and so on. When we collect the data in a formalized way and use it to make decisions, we call this evaluation. Simply put, “evaluation determines the merit, worth, or value of things,” according to evaluation expert Michael Scriven.

Equitable Evaluation

So what does this have to do with equity, diversity, and inclusion? Well…everything. If evaluation does in fact determine the merit, worth, or value of programs and services, what happens when your library’s evaluation excludes or overlooks certain groups from the data? Let’s take a look:

You are trying to evaluate patron satisfaction at your library, so you print off a stack of surveys and leave them on the lending desk for patrons to take. While everyone in your target audience may have equal access to the survey (or in other words, are being treated the same), they don’t all have equitable access. Sometimes people may need differing treatment in order to make their opportunities the same as others. In this case, how would someone who has a visual impairment be able to take a printed survey? What about someone who doesn’t speak English? These patrons would likely ignore your survey, and without demographic questions on language and disability, the omission of these identities might never be known. Upon analyzing your data, conclusions might be made to suggest, “X% of patrons felt this way about x, y, and z.” In reality, your results wouldn’t represent all patrons—only sighted, English-speaking patrons.

Inequities are perpetuated by evaluation when we fail to ensure our methods are inclusive and representative of everyone in our target group. The data will produce conclusions that amplify the experiences and perspectives of the dominating voice while simultaneously reproducing the idea that their narrative is representative of the entire population. Individuals who have historically been excluded will continue to be erased from our data and the overarching narrative, serving to maintain current power structures.

Evaluation With the Community, not On the Community

That’s a heavy burden to take on as an evaluator and a library professional, especially when taking part in people’s marginalization is the last thing you would want to do. Luckily, the research community has long been working on some answers to this problem. Community-based participatory research (CBPR) is contingent on the participation of those you are evaluating (your target population) and emphasizes democratization of the process. CBPR is defined as:

“focusing on social, structural, and physical environmental inequities through active involvement of community members, organizational representatives, and researchers in all aspects of the research process. Partners contribute their expertise to enhance understanding of a given phenomenon and integrate the knowledge gained with action to benefit the community involved.”

CBPR centers around seven key principles:

  1. Recognizes community as a unit of identity
  2. Builds on strengths and resources
  3. Facilitates collaborative partnerships in all phases of the research
  4. Integrates knowledge and action for mutual benefit of all partners
  5. Promotes a co-learning and empowering process that attends to social inequalities
  6. Involves a cyclical and iterative process
  7. Disseminates findings and knowledge gained to all partners

As one librarian put it, CBPR “dismantles the idea that the researcher is the expert and centers the knowledge of the community members.” When those that you are evaluating (whether it be patrons, non-users, people with a disability, non-English speakers, etc.) are involved in the entire process, your data will invariably become more equitable. As a result, your evaluation outcome will more effectively address real problems for your community. It’s a win-win for everyone.

However, if diving into a full community-based participation evaluation feels impossible given your time and resources, it’s okay. Think of CBPR as your ideal and then adjust to a level that is feasible for your library. The continuum of community engagement below outlines what some of those different levels might look like.

The continuum of community engagement ranges from total CBPR on the left end of the spectrum to community engagement on the right end of the spectrum. Total CBPR is full involvement in all parts of the study development and conduct. CBPR light is partial involvement in some or all parts of the study development and conduct. Community based research is research conducted in collaboration with members of the community. And community engagement is working with community members and agencies to reach community members.

The Big Takeaway

Evaluating your practices, policies, and programs in a library can lead to better outcomes for your library community. However, even the best of intentions can create harm for historically underrepresented groups when they are excluded from the very data used to make decisions that impact them. When undertaking an evaluation of any kind, think about the principles of CBPR and how you can incorporate them into your plan.

Why Observe? Watch and Learn

When I was a kid, one of my favorite summer activities was staring at hummingbirds. I would sit for hours, moving as little as possible, while I took notes about everything I saw. (Yes, I was a pretty weird eight year old.) I wanted to ask the hummingbirds so many questions, but I don’t speak hummingbird! Observing them was my only option for trying to understand their behavior. 

While it is literally impossible to ask a hummingbird to take a survey, there are many times with humans when a survey won’t work to collect the data you need either. Observation can be a great data collection tool when you want to see how different people interact with each other, a space, or a passive program. Observation is also helpful when it would be difficult for someone to answer a question accurately, like when you ask them to remember what they did or, particularly with children, if you ask them to give critical feedback or written feedback, both of which are sometimes developmentally inappropriate. 

In this post, I’m going to talk about why you might choose observation as a data collection method. Next time, I’ll talk about the logistics of observations and how you can use observational data. To better understand why you would collect data with observations, let’s use our example evaluation question from throughout this blog series: “Does attending storytime help caregivers use new literacy skills at home?” 

When we first outlined number data and story data, we talked about when to use each. We also outlined how to break your research question down into smaller questions. You really need to do that work to get to this point, so let’s go back and review what we did.  Here are some of the sub-questions we identified within our larger evaluation question:

  • Were caregivers already using literacy skills at home prior to attending a storytime? 
  • Are caregivers learning new literacy skills during storytime? 
  • Do caregivers use new literacy skills from storytime at home? 

Would a survey work to collect this data? We certainly could ask caregivers all of these questions. But we would immediately bump into some of the problems that come up when people self-report data: 1) we are not great at remembering things accurately and 2) we want to portray ourselves in the best possible light (social desirability bias). Let’s take a look at how those challenges would impact the data collection for our questions. 

  • Were caregivers already using literacy skills at home prior to attending a storytime? 
    • They may not know or accurately remember which skills they knew before attending storytime and which they learned at storytime. 
  • Are caregivers learning new literacy skills during storytime? 
    • They may report that they are learning new literacy skills at storytime because they don’t want to hurt anyone’s feelings—even if they aren’t actually learning those skills. 
  • Do caregivers use new literacy skills from storytime at home? 
    • They may report that they are using new literacy skills at home because that feels nice to say—even if they aren’t actually using those skills at home. 

So we could collect that data using a survey, but it may not be very accurate. We could get more accurate data by observing caregivers at home with children before they ever attended a library storytime and then continuing to observe after they started attending storytime. Then we could see for ourselves what skills they already knew and used at home, and which ones they learned at storytime. We could tally up how often they were using those skills too. Great! Let’s go follow people and their children around their homes 24 hours a day taking notes for several months. 

What? You don’t think that’s going to be a thrilling success? Unlike hummingbirds, who don’t seem to mind too much or alter their behavior a lot while I am watching, humans mind quite a bit and can change their behavior when they are being observed. Additionally, do you know any library staff who have the time to do this kind of intense observational study? Yeah, that’s what I thought. The time involved in observation and successfully navigating privacy concerns are two major elements that you always need to consider. 

What can we do that’s a little more realistic? Collecting data in the real world is often about doing what you can with what you have. In this case, it is unlikely anyone would let us come follow them around their home. We can, however, more easily observe caregivers and their children in the library. This would allow us to observe for indicators of caregivers learning skills during storytime and to observe if families are using early literacy skills while they are spending unstructured time in the library. Intrigued as to how we would do that? Come back for our next post where we’ll get into the nuts and bolts of how you can collect data using observation and pull out important takeaways from that data.

If you are an aspiring birdnerd, the two hummingbirds pictured are both species we have in Colorado and you can learn more about them here

Surveys: Don’t just set it and forget it!

Surveys are the rotisserie oven of the data collection methods. You simply “set it, and forget it!” That’s why it’s important to be strategic about how you’re reaching your target population. Otherwise, you may be leaving out key subsets of your audience—which are often voices that are already historically underrepresented.  

Is your survey equitable? 

Let’s say you want to send out a survey to library users, so you print off a stack of copies and leave them on the lending desk for patrons to take. While everyone in your target audience may have equal access to the survey (or in other words, are being treated the same), they don’t all have equitable access. Sometimes people may need differing treatment in order to make their opportunities the same as others. In this case, how would someone who has a visual impairment be able to take a printed survey? What about someone who doesn’t speak English? These patrons would likely ignore your survey, and without demographic questions on language and disability, the omission of these identities might never be known. Upon analyzing your data, conclusions might be made to suggest, “X% of patrons felt this way about x,y, and z.” In reality, your results wouldn’t represent all patrons—only sighted, English-speaking patrons. 

Who has access to your survey? 

Start by thinking about who you want to answer your survey—your target population. Where do they live? What do they do? What identities do they hold? Consider the diversity of people that might live within a more general population: racial and ethnic identities, sexual orientation, socio-economic status, age, religion, etc. Next, think through the needs and potential barriers for people in your target population, such as language, access to transportation, access to mail, color blindness, literacy, sightedness, other physical challenges, immigration status, etc. Create a distribution plan that ensures that everyone in your target population—whether they face barriers or not—can access your survey easily. Here are some common distribution methods you could use: 

  • Direct mail – Here’s more information about how to do a mail survey and it’s advantages and disadvantages. 
  • Online – For more information on how to make your online survey accessible, check out this article from Survey Monkey.
  • Telephone – In a telephone survey, someone calls the survey taker and reads them the questions over the phone while recording their answers. 
  • In-person – Surveys can also be administered in-person with a printed stack of surveys or a tablet. However, with this approach you might run into the dangers of convenience sampling

Depending on your target audience, surveys are rarely one-size-fits-all. The best plan is often a mixed-methods approach, where you employ multiple distribution strategies to ensure equitable access for all members of your target population. 

Who is and isn’t taking your survey?

Great! You’ve constructed a distribution plan that you feel can equitably reach your target population, but did it work? The only way to know for sure is by collecting certain demographic information as part of your survey. 

As library professionals, collecting identifying information can feel like a direct contradiction to our value of privacy. Yet, as a profession we are also committed to equity and inclusivity. When administering a survey, sometimes it’s necessary to collect demographic data to better understand who is and isn’t being represented in the results. Questions about someone’s race, ethnicity, income level, location, age, gender, sexual orientation, etc. not only allow us to determine if those characteristics impact someone’s responses, but also help combat the erasure of minority or disadvantaged voices from data. However, it’s important to note that: 

  1. You should always explicitly state on your survey that demographic questions are optional, 
  2. You should ensure responses remain anonymous either by not collecting personal identifying information or making sure access to that information is secure, and 
  3. Only collect demographic information that’s relevant and necessary to answer your particular research question. 

Compare the data from your demographic questions with who you intended to include in your target audience. Are there any gaps? If so, re-evaluate your distribution plan to better reach this sub-group(s), including speaking to representatives of the community or people that identify with the group for additional insight. Make additional efforts to distribute your survey, if necessary.

Conclusion

Inequities are perpetuated by research and evaluation when we fail to ensure our data collection methods are inclusive and representative of everyone in our target group. The absence of an equitable distribution plan and exclusion of relevant demographic questions on your survey runs the risk of generating data that maintains current power structures. The data will produce conclusions that amplify the experiences and perspectives of the dominating voice while simultaneously reproducing the idea that their narrative is representative of the entire population. Individuals who have historically been excluded will continue to be erased from our data and the overarching narrative.

Bad Survey Questions, part 2

Bad Survey Questions – pt. 2

Don’t let those bad survey questions go unpunished. Last time we talked about leading and loaded questions, which can inadvertently manipulate survey respondents. This week we’ll cover three question types that can just be downright confusing to someone taking your survey! Let’s dig in. 

Do you know what double-barreled questions are and how to avoid them?

When we design surveys it’s because we’re really curious about something and want a lot of information! Sometimes that eagerness causes us to jam too much into a single question and we end up with a double-barreled question. Let’s look at an example: 

         How satisfied are you with our selection of books and other materials? 

O    Very dissatisfied
O    Dissatisfied
O    Neither satisfied nor dissatisfied 
O    Satisfied
O    Very satisfied

Phrasing the question like this creates two problems. First, if a respondent selected “very dissatisfied,” when you analyzed the data you wouldn’t know if they were saying they were very dissatisfied with only the books, only the materials, or both. Second, if the respondent was dissatisfied with the book selection, but was very satisfied with the DVD selection, they wouldn’t know how to answer this question. They have to just choose an inaccurate response or stop the survey altogether.  

Survey questions should always be written in a way that only measures one thing at a time. So ask yourself, “What am I measuring here?” The double-barreled issue is in the second part of the survey question. What are you measuring the satisfaction of? Books and materials. 

Two ways of spotting a double-barreled question are: 

  1. Check if a single question contains two or more subjects, and is therefore measuring more than one thing.
  2. Check if the question contains the word “and.” Although not a foolproof test, the use of the word “and” is a good indicator that you should double check (pun intended) for a double-barreled question.

You can easily fix a double-barreled question by breaking it into two separate questions.

How satisfied are you with our selection of books?
How satisfied are you with our selection of other materials?

This may feel clunky and cause your survey to be longer, but a longer survey is better than making respondents feel confused or answer incorrectly. 

Do you only use good survey questions every day on all of your surveys, always?

Life isn’t black and white, therefore survey questions shouldn’t be either. Build flexibility into your response options by avoiding absolutes in questions and answer choices. Absolutes force respondents into a corner and the only way out is to give you useless data. 

When writing survey questions, avoid using words like “always,” “all,” “every,” etc. When writing response options, avoid giving only yes/no answer options. Let’s look at the examples below:

                    Have you attended all of our library programs this summer?  O Yes   O No

The way this question and response options are phrased would force almost any respondent to answer “no.” Read literally, you’re asking if someone went to every library program you’ve ever had, whether or not it was offered this summer or for their age group. Some respondents might interpret the question as you intended, but why leave it up to chance? Here’s how you might rewrite the absolute question:

How many of our library programs did you attend this summer?

Instead of only providing yes or no as answer choices, you should also use a variety of answer options, including ranges. For instance, if you also asked the survey respondent how many books they read during the summer, your answer options could be:

O    I have not attended any
O    1-3
O    4-6
O    7-9
O    10+
O    I do not know

Chances are, a respondent would feel like they easily fall into one of these categories and would feel comfortable choosing one that’s accurate.

Have you indexed this LRS text in your brain? 

In libraryland, we LOVE acronyms and jargon, but they don’t belong in a survey. Avoid using terms that your respondents might not be familiar with, even if they’re deeply familiar to you. If you use an acronym spell it out the first time you mention it, like this: Library Research Service (LRS). Be as clear and concise as possible while keeping the language uncomplicated. For instance, if asking how many times someone used a PC in the last week, be sure to explain what you mean by PC, and include examples like below: 

In the last week, how many times have you used a PC (ipad, laptop, android tablet, desktop computer)? 

Do you remember all the tools and tips we covered in our bad survey questions segment?

Hey, that’s ok if not! Here’s a quick review of things to do and don’t do in your surveys:

   Do use neutral language.

     Don’t use leading questions that push a respondent to answer a question in a certain way by using non-neutral language.  

   Do ask yourself who wouldn’t be able to answer each question honestly.

     Don’t use loaded questions that force a respondent to answer in a way that doesn’t accurately reflect their opinion or situation.

   Do break double-barreled questions down into two separate questions.

     Don’t use double-barreled questions that measure more than one thing in a question.

   Do build flexibility into questions by providing a variety of response options.

     Don’t use absolutes (only, all, every, always, etc.) that force respondents into a corner.

   Do keep language, clear, concise and easy to understand.

     Don’t use jargon or colloquial terms.