In our last post, we talked about when you should use a survey and what kind of data you can get from different question types. This week, we’re going to cover two of the big survey question mistakes evaluators make and how to avoid them so you don’t end up with biased and incomplete data. In other words—all your hard work straight into the trash!
Do you think a leading question is manipulative?
Including leading questions in a survey is a common mistake evaluators make. A leading question pushes a survey respondent to answer in a particular way by framing the question in a non-neutral manner. These responses therefore produce inaccurate information. Spot a leading question by looking for any of these characteristics:
- They are intentionally framed to elicit responses according to your preconceived notions.
- They have an element of conjecture or assumption.
- They contain unnecessary additions to the question.
Leading questions often contain information that a survey writer already believes to be true. The question is then phrased in a way that forces a respondent to confirm that belief. For instance, take a look at the question below.
Do you like our exciting new programs?
You might think your programs are exciting, but that’s because you’re biased! This question is also dichotomous, meaning they must answer yes or no. While dichotomous questions can be quick and easy to answer, they don’t allow any degree of ambivalence or emotional preference. Using the word “like” also puts a positive assumption right in the question, pushing the respondent in that direction. A better way to write this question would be:
How satisfied are you with our new programs?
In order to avoid leading questions, remember to do the following:
- Use neutral language.
- Keep questions clear and concise by removing any unnecessary words.
- Do not try to steer respondents toward answering in a specific way. Ask yourself if you think you know how most people will answer. This might highlight assumptions you’re making.
Why are loaded questions so bad?
Similar to leading questions, loaded questions force a respondent to answer in a way that doesn’t accurately reflect their opinion or situation. These types of questions often cause a respondent to abandon a survey entirely, especially if the loaded questions are required. Common characteristics of loaded questions are:
- Use words overcharged with positive or negative meaning.
- Questions that force respondents into a difficult position, such as forcing them to think in black and white terms.
- Presupposes the respondent has done something.
Let’s look at some examples of loaded questions. Put yourself in the shoes of different respondents. Can you think of someone that would have trouble or feel uncomfortable answering them?
How would someone who has never accrued late fees answer this question? This places someone in a logical fallacy. If they answer “yes,” they are saying that they once had late fees. If they answer “no” because they never started accruing late fees, then they are saying that they are still getting charged.
Why did you dislike our summer reading program?
How would someone who likes the summer reading program answer this question? This places someone in a logical fallacy. Any answer choices they select would be inaccurate. The question is loaded because it presupposes that respondents felt negatively about the program.
When you used our “ask a librarian” service, was the librarian knowledgeable enough to answer your question?
What if the librarian wasn’t knowledgeable, but was helpful? Maybe they didn’t know the answer, but they pointed you in the right direction so that you could find the answer. This phrasing causes the respondent to think in black and white terms, either they gave you the answer or nothing. Not to mention this question assumes you’ve used the service at all!
Here are some ways to avoid using loaded questions:
- Test your survey with a small sample of people and see if everyone is able to answer every question honestly.
- If you aren’t able to test it, try putting on multiple hats yourself and ask yourself who wouldn’t be able to answer this?
- You can also break questions down further and use what’s called “skip logic.” This means you would first ask respondents, “Have you used our ask a librarian service?” If they answer “yes,” then you would have them continue to a question about that service. If they answer “no,” they would skip to the next section.
How useful was this blog post for learning about surveys and helping you file your taxes?
As the bad question example above might allude to, we aren’t done with this topic! In our next post, we’ll talk about double-barreled questions and absolute questions, so stay tuned! As always, if you have any questions or feedback we’d love to hear from you at LRS@LRS.org.