Bad Survey Questions – pt. 2
Don’t let those bad survey questions go unpunished. Last time we talked about leading and loaded questions, which can inadvertently manipulate survey respondents. This week we’ll cover three question types that can just be downright confusing to someone taking your survey! Let’s dig in.
Do you know what double-barreled questions are and how to avoid them?
When we design surveys it’s because we’re really curious about something and want a lot of information! Sometimes that eagerness causes us to jam too much into a single question and we end up with a double-barreled question. Let’s look at an example:
How satisfied are you with our selection of books and other materials?
O Very dissatisfied
O Dissatisfied
O Neither satisfied nor dissatisfied
O Satisfied
O Very satisfied
Phrasing the question like this creates two problems. First, if a respondent selected “very dissatisfied,” when you analyzed the data you wouldn’t know if they were saying they were very dissatisfied with only the books, only the materials, or both. Second, if the respondent was dissatisfied with the book selection, but was very satisfied with the DVD selection, they wouldn’t know how to answer this question. They have to just choose an inaccurate response or stop the survey altogether.
Survey questions should always be written in a way that only measures one thing at a time. So ask yourself, “What am I measuring here?” The double-barreled issue is in the second part of the survey question. What are you measuring the satisfaction of? Books and materials.
Two ways of spotting a double-barreled question are:
- Check if a single question contains two or more subjects, and is therefore measuring more than one thing.
- Check if the question contains the word “and.” Although not a foolproof test, the use of the word “and” is a good indicator that you should double check (pun intended) for a double-barreled question.
You can easily fix a double-barreled question by breaking it into two separate questions.
How satisfied are you with our selection of books?
How satisfied are you with our selection of other materials?
This may feel clunky and cause your survey to be longer, but a longer survey is better than making respondents feel confused or answer incorrectly.
Do you only use good survey questions every day on all of your surveys, always?
Life isn’t black and white, therefore survey questions shouldn’t be either. Build flexibility into your response options by avoiding absolutes in questions and answer choices. Absolutes force respondents into a corner and the only way out is to give you useless data.
When writing survey questions, avoid using words like “always,” “all,” “every,” etc. When writing response options, avoid giving only yes/no answer options. Let’s look at the examples below:
Have you attended all of our library programs this summer? O Yes O No
The way this question and response options are phrased would force almost any respondent to answer “no.” Read literally, you’re asking if someone went to every library program you’ve ever had, whether or not it was offered this summer or for their age group. Some respondents might interpret the question as you intended, but why leave it up to chance? Here’s how you might rewrite the absolute question:
How many of our library programs did you attend this summer?
Instead of only providing yes or no as answer choices, you should also use a variety of answer options, including ranges. For instance, if you also asked the survey respondent how many books they read during the summer, your answer options could be:
O I have not attended any
O 1-3
O 4-6
O 7-9
O 10+
O I do not know
Chances are, a respondent would feel like they easily fall into one of these categories and would feel comfortable choosing one that’s accurate.
Have you indexed this LRS text in your brain?
In libraryland, we LOVE acronyms and jargon, but they don’t belong in a survey. Avoid using terms that your respondents might not be familiar with, even if they’re deeply familiar to you. If you use an acronym spell it out the first time you mention it, like this: Library Research Service (LRS). Be as clear and concise as possible while keeping the language uncomplicated. For instance, if asking how many times someone used a PC in the last week, be sure to explain what you mean by PC, and include examples like below:
In the last week, how many times have you used a PC (ipad, laptop, android tablet, desktop computer)?
Do you remember all the tools and tips we covered in our bad survey questions segment?
Hey, that’s ok if not! Here’s a quick review of things to do and don’t do in your surveys:
Do use neutral language.
Don’t use leading questions that push a respondent to answer a question in a certain way by using non-neutral language.
Do ask yourself who wouldn’t be able to answer each question honestly.
Don’t use loaded questions that force a respondent to answer in a way that doesn’t accurately reflect their opinion or situation.
Do break double-barreled questions down into two separate questions.
Don’t use double-barreled questions that measure more than one thing in a question.
Do build flexibility into questions by providing a variety of response options.
Don’t use absolutes (only, all, every, always, etc.) that force respondents into a corner.
Do keep language, clear, concise and easy to understand.
Don’t use jargon or colloquial terms.