In the novel Hitchhiker’s Guide to the Galaxy, a group of hyper-intelligent pan-dimensional beings build a supercomputer to ask the “ultimate question…the answer to life, the universe, and everything.” After waiting millions of years, the supercomputer tells them the answer to life, the universe, and everything is…42! Some might disagree, but the lesson here is simple—if you want a useful finding, you have to ask the right evaluation question! And you don’t have to be hyper-intelligent, pan-dimensional beings to learn how to do it.
Evaluation questions are developed to guide your evaluation. They allow you to focus your study, clarify your program/service outcomes, and help check or authenticate your work. Your mission is to answer them by collecting and analyzing data. Your findings should give you important insights about your program or service. Depending on the size of your evaluation, you may have anywhere from one to five main questions. To get started, follow these four steps:
Step 1: Clarify goals and objectives of program
You can’t develop an evaluation question if you aren’t clear about the intended outcomes of your program. Otherwise, you might research a question that ends up being entirely irrelevant. Also take the time to review your logic model—you want to ensure that each question ties to one of its components. For instance, if an activity on your logic model is STEM instruction, you might ask “to what extent did staff have adequate training and support to implement proper STEM instruction to children ages 6-12.” Luckily, we’ve recently covered both of these topics more in-depth. Learn more about outcomes here or logic models here.
Step 2: Identify key stakeholders and audiences
It’s helpful to make a list of your evaluation’s stakeholders and audiences, including taking note of their “stakes.” From this stakeholder list, identify who your evaluation serves. Is it to provide data to your library board? Do you intend to use the information to improve a program for library users? Think about whether your evaluation questions will give you answers that serve these groups of people. Additionally, consider whether your key stakeholders or audiences should have an opportunity to provide feedback on your evaluation questions.
Step 3: Write a list of evaluation questions
Now it’s time to put pen to paper and write some questions. Write as many as you can think of and then we’ll eliminate some in the next step. Here are some examples that frame the question around the objective of your evaluation:
Objective: To review the summer reading program.
Question: In what ways are participating children demonstrating interest in reading at home?
Objective: To provide information on non-library users in the community.
Question: For what reasons do residents within our library service area not use the library?
Objective: To examine library services directed at library users being affected by housing insecurity.
Question: To what extent are library programs and services directed at housing insecure patrons meeting their direct needs?
Sub-question: You can also have a sub-question like, “What need-gaps still exist that library services could provide?”
Step 4: Evaluate your evaluation questions
Ok, I know this might be starting to feel like the movie Inception, but bear with me. Now we need to evaluate each evaluation question based on these criteria:
Relevant: Does the question clearly apply to an aspect of the program (i.e. design, activities, outcomes)? Does it contribute valuable information to stakeholders?
Answerable: Is it possible to answer this question via empirical research methods? Can you obtain the necessary information ethically and respectfully?
Reasonable: Can the question be addressed given the resources and constraints (time, budget, staff, etc.) of your evaluation? Is it worth the effort?
Specific: Does the question distinctly target a program component? Are there any ambiguous phrases or undefined target groups?
Evaluative: Will data related to the evaluation question provide either formative information about the program or service for decision-making and improvement purposes, or summative information to determine the effectiveness? Is your question phrased objectively so that you are not making assumptions about your program or service prior to evaluating it?
Complete: Will the evaluation question give ample information for stakeholders to move forward?
If questions on your list don’t meet all of these criteria, consider revising or eliminating them. It’s possible you still have too many to be able to accomplish them all within your constraints. If so, go through each one and score them based on the criteria (1 = not very relevant, 5 = very relevant, etc.). Prioritize the questions that score the highest.
Still have questions about evaluation questions? Feel free to reach out to us at LRS@LRS.org. We’re always happy to shop talk and help you reach your library evaluation goals!