Measuring Social and Emotional Learning Competencies in a Summer Learning Program

 

Denver Public Library (DPL), in collaboration with Library Research Service (LRS), was recently featured in School Library Journal. The article highlights DPL’s evaluation of their summer learning program and use of data to inform programmatic decision making. Below is a summary of the results. To learn more about their data collection methods, analysis, and application of findings, you can read the full article here.

Denver Public Library (DPL) knew anecdotally they were positively affecting the social and emotion learning (SEL) of their youngest patrons, but needed to find a way to measure it. So in 2017, when they began shifting from a summer reading program to a summer learning program, they wanted to take the opportunity to evaluate the program’s impact. Their new program, titled Summer of Adventure, aimed to build relationships and facilitate social and emotional learning in addition to addressing summer learning loss.

With the help of Library Research Service (LRS)’s research analyst, Katie Fox, DPL began focusing on outcomes (the impact a program has) over outputs (registration and attendance). Their outcome goal for the program was: “After Summer Academy, participants will gain or enhance their social and emotional skills.” Knowing they could not likely see measurable change in SEL skills during the month-long program, their evaluation question became: “What social and emotional skills do youth participants currently have?” By understanding what skills youth needed to build upon, DPL could learn more about how different types of programming could encourage positive SEL behaviors.

During the evaluation, DPL utilized various data collection methods, which presented some limitations and challenges. An analysis of the data revealed two key findings:

  1. Relationship building occurred more during unstructured rather than structured activities; and
  2. Youth participants showed the most positive self-management during moderately challenging activities allowing many ways to complete the product.

Library staff used this information to help make strategic decisions about future programs and communicate with external stakeholders and funders about the program’s value. DPL continues to adjust the program to better support SEL, intrinsic motivation, and life-long learning. To learn more, read the full article here.

 

How to Compare Apples to Oranges

As our brains process information, we constantly make comparisons. It’s how we decide if something is good or bad—by it being better or worse than something else. However, like apples and oranges, not all things can readily be compared, even if they appear similar enough on the surface. We often make this mistake with data because we want to be able to draw simple conclusions. But when our goal is accurate information, it’s imperative to look at presentations of data  through a critical lens by applying these basic strategies.

So who’s better? 

Let’s say you wanted to determine whether Library A or B was doing a better job at reaching its community. To do so, you compare annual visits at both. This chart would lead you to conclude Library B has much more annual traffic and is therefore reaching more of its community than Library A. But are Library A and B comparable?

Library A serves a population of 5,400 while Library B serves a population of 30,500. When making comparisons among different populations, data should be represented in per capita measurements. Per capita simply means a number divided by the population. For instance, when we compare countries’ Gross Domestic Products (GDP), or value of economic activity, we usually express it as GDP per capita because it would be misleading to compare China’s GDP to that of Denmark. China’s GDP trounces Denmark’s, but that doesn’t mean Denmark’s economy is struggling. China is larger both in terms of the land it covers and the number of people that live there. It would be really weird if they had similar GDPs without the per capita adjustment. The same is true in this example. Take a look at how we draw an entirely different conclusion when total visits are expressed in a per capita measurement.

*Due to a 2-month closure, Library B’s data was only collected over a 10-month period

Now we can see that Library A has 18.5 visits per person i(100,000/5,400), whereas Library B only has 6.6 visits per person (200,000/30,500). These are the same data, but expressed in more comparable terms.

Let’s say Library B also closed for two months to do some construction on their building. Therefore, their annual visits account for 10 months of operation, not 12. Contextual information like this – which has a direct effect on the numbers – needs to be clearly called out and explained, like in the example above.

Breaking it down…

To check for comparability, it’s helpful to keep three things in mind: completeness, consistency, and clarity.

Completeness: are the data comparing at least two things? 

It would be incorrect to say “Library B has 100,000 more visits.” More visits than…Library A? Than last year? Also be wary of results indicating that  something is better, worse, etc. without stating what it is better or worse than.

Consistency: are the data being compared equivalent? And even if they appear equivalent , what information is needed to confirm this assumption? 

One of the best examples of inconsistency occurs when comparing data from different populations, particularly when we focus on total counts. “Totals” are often a default metric because it’s simple for a range of audiences to understand, but it can be very misleading, like in the first chart above. By expressing the data as per capita measurements, we can account for population differences and create a basis of similarity. Additionally, even if data appear similar enough to compare, you also need to review how they were collected. Any reliable research will include these details  (big red flag if it doesn’t !). For instance, it would be important to know that Library A and B were counting visits in the same way. If Library A is counting one week during the summer and multiplying that by 52 that wouldn’t be consistent with Library B who is counting during a week in the winter.

Clarity: Is it obvious and clear what is being compared? 

Data visualizations allow our brains to interpret information quickly, but that also means we may jump to conclusions. Be a critical data consumer by considering what underlying factors might also be at play. The second chart above clarifies that two months of data were missing from Library B. This could be one reason why Library B’s total visits per capita were so much lower than Library A’s. Also beware of unclear claims supposedly supported by the data, like “Library A has higher patron engagement than Library B.” Perhaps Library A defines engagement in terms of number of visits, but Library B’s definition is based on material circulation and program attendance. The data above do not provide enough information to support a comparable claim on engagement.

Comparisons are messy. Whether in library land or elsewhere, keep in mind that comparisons are always tricky, but also very useful. By engaging critically using the strategies above, we CAN compare apples and oranges. They are both fruit afterall…

LRS’s Between a Graph and a Hard Place blog series provides strategies for looking at data with a critical eye. Every week we’ll cover a different topic. You can use these strategies with any kind of data, so while the series may be inspired by the many COVID-19 statistics being reported, the examples we’ll share will focus on other topics. To receive posts via email, please complete this form.

New blog series: Between a Graph and a Hard Place

Hello, world!

We can all agree that these are strange times we are living through. Here at the Library Research Service, we’ve been thinking about how we can help. What skills could we share that might be useful to library staff and our communities?

As library and information professionals, before this pandemic we already spent a lot of time thinking about information, what it means, and how reliable it is. Here at LRS, we are data geeks in addition to being regular library geeks, so we think about data a lot too—the good, the bad, and the misleading.

Critically analyzing information is what librarians are trained to do. We can’t help ourselves. For me, this means every time I talk to my mom and she shares a statistic with me, I ask her about her source. I’m not trying to be a pain. This is just how my mind works.

Right now, we are all seeing a lot of data about the pandemic, and it can be challenging to understand. And this is where we come in.

Let’s be clear: we are not epidemiologists, we are not medical doctors, we are not experts in public health. We are not going to provide data about COVID-19 or interpretations. There are already good resources for both, and we don’t think it would help to add our voices.

What we can do—and we are going to do—is share strategies for looking at data with a critical eye. We’re going to cover a different strategy every two weeks, like thinking about the underlying data behind a visualization, identifying bias, evaluating the credentials of different experts, understanding that how the data are presented can impact how you perceive them, and how to find multiple perspectives on the same information.

We will also discuss how to engage with data carefully, with your mental well-being in mind. Data can make us feel a lot of things, and we all need to take care of ourselves.

This series is inspired by the current situation, but the examples we will share will focus on other topics. You can use these strategies with any kind of data.

We look forward to seeing you here every other Wednesday and hope that these strategies are helpful in this time of information overload. In the meantime, if you’d like some less serious data about a situation that many of us can relate to right now, check out this pie chart.

If you want to subscribe to receive the blog posts from this series by email, please complete this form.

Every Day is Earth Day in Libraries

Half a century ago, Earth Day began as a grassroots effort to bring attention to environmental issues. Now fifty years later, organizers are bringing the focus back to climate change, an admittedly enormous challenge, and urging everyone to take part in protecting and restoring our planet. As lending institutions, libraries have long understood their role as stewards of environmental responsibility. The Green Library Movement began in the early 1990s as a commitment to greening libraries by reducing their environmental impact on the planet. The movement gained popularity around 2003 and then in 2019, the American Library Association (ALA) Council adopted sustainability as a professional core value.

Is the Green Library Movement Growing?

In the years since the movement took off, libraries around the world have reviewed their operations and programming to identify ‘greener’ methods. Buildings have been rebuilt or remodeled to include energy efficient design and physical materials have been replaced with digital mediums. However, much debate still remains over whether libraries are fully embracing the challenge. A 2012 study conducted in Finland discovered that up to 60 percent of respondents believed the components of environmental management had not been taken into account enough in their own libraries. When asked about everyday routines, more than half of libraries were turning off lights after 10 minutes, switching computers off at the end of the working day, and sorting waste products. However, only about 10 percent of respondents said their library’s printers print on both sides of the paper by default. Less than five percent said laptops were preferred in computer acquisitions.

Measuring a Library’s Carbon Emission

Given limited resources, it can be difficult for a library to prioritize green initiatives, especially if they can’t pinpoint where they are expending the most energy consumption. To provide a better understanding of an institution’s Global Warming Potential (GWP), students at University of California Berkeley developed a carbon emissions calculator specifically for academic libraries. While it can be adapted to public and school libraries, the tool fails to take into account newer technologies (e-books, 3D printers, sewing machines, etc.), as well as waste produced from programming and outreach. Using the engineering library at UC Berkeley as a case study, researchers found that the HVAC was the largest abuser of annual power consumption (~145,000 kWh). Of materials, volumes had the greatest GWP at 5,394 metric tons of CO2 equivalent per year.

Greening a Community, Not Just the Library

Even if libraries are not huge carbon emission offenders, they still play a pivotal role in introducing sustainability initiatives to their communities. As green buildings, they can demonstrate the use of solar panels or reflective roofing, educate the community about residential use of rain barrels using their own rainwater collection systems, and incorporate native plants into landscaping to reduce reliance on irrigation.

E-books are another popular way of reducing the carbon footprint. A study at Boston College found that the majority of environmental waste for both e-books and paper books originates before reaching the hands of the intended audience. However, paper books contribute significantly more waste during distribution, making them less environmentally friendly. According to the study, a user would have to access 33 e-books on a device before offsetting the carbon footprint of one printed counterpart.

Libraries can also publicize green initiatives through creative programming. For adults, one librarian suggests screening a documentary related to sustainability. For children’s programming, another librarian tries to find materials that can be reused or repurposed. She also refuses programs that produce single-use waste. Being conscious of a program’s environmental impact—and highlighting that success—can be key takeaways for patrons.

Every Day is Earth Day in Libraries

Climate change may be the theme of this year’s Earth Day celebration, but more and more, it is serving as a foundation for libraries. Whether through building constructing, material use, or programming, multiple opportunities exist for libraries to become agents of change within their communities. Earth Day can be every day in a library.

Note: This post is part of our series, “The LRS Number.” In this series, we highlight statistics that help tell the story of the 21st-century library.

About 7 out of 10 US adults read a book last year, but among those without a high school degree it was 3 out of 10

About seven out of ten adults (72%) in the U.S. report that they read a book in the last 12 months. This percentage has stayed about the same since the Pew Research Center started conducting studies of adult reading habits in 2011, but it does vary depending on income and education.

In the most recent survey in 2019, nine out of ten college graduates (90%) said they read a book in the past year while only about three out of ten (32%) adults without a high school degree did. Higher percentages of women, Whites, those earning more than $75,000, and people living in urban areas reported reading a book in the past year. Males, Hispanics, those earning less than $30,000, and people living in rural areas reported lower rates of reading. The overall percentage of men who read a book decreased from 73% in 2018 to 67% in 2019.

The portion of people reading audio books is on the rise, and increased from 14% in 2016 to 20% in 2019. This increase is particularly strong for college graduates and those who earn more than $75,000. Print books, however, are still the most popular way for people to read: 65% of the people who read a book in the last year read a print book. Another 25% of people reported reading an e-book in the past year. About 37% of adults read only print books.

The full report can be found here.

Note: This post is part of our series, “The LRS Number.” In this series, we highlight statistics that help tell the story of the 21st-century library.

How much is your library worth?

We can all agree that libraries are valuable to our communities, but exactly how much are they worth? Libraries are under increasing pressure to translate qualitative services into quantifiable impact. One approach is to determine the Return on Investment (ROI) a library provides to community members. Doing so communicates the value of public libraries in terms of dollars and cents.

Traditionally a business metric, ROI measures a business’s profitability. Simply put, it compares costs to profits and expresses it as a ratio or percentage. For a public institution like a library, ROI demonstrates how much “value” is realized by the community for each dollar spent on services and materials. This includes:

  • The cost to use alternatives: the estimated amount of money that would have been spent to use an alternative if the library did not exist;
  • Lost use: for patrons who indicated they would not have tried to meet their needs with another source or would not have known where else to go, the estimated value of the direct benefit that they would not have received if the library didn’t exist;
  • Direct local expenditures: dollar figures for expenditures on goods and services within the library’s legal service area;
  • Compensation for library staff: the amount of annual compensation that staff members would not have received if the library didn’t exist; and
  • Halo spending: purchases made by library patrons from vendors and businesses that are located close to the library.
  • Some ROI methodologies also apply a dollar amount to patrons’ time and take the amount saved seeking materials or services elsewhere into account.

Two approaches are commonly used to calculate a library’s ROI: contingent valuation or market valuation, both of which have their strengths and weaknesses. Contingent valuation bases dollar values on subjective perceptions of responding library users. However, within those subjective perceptions, patrons may include a more holistic experience that takes into account the value of having various needs being met in one place. This method acknowledges that the value of a library is likely greater than the sum of the value of its individual resources and services. In contrast, market valuation bases dollar values on objective, “real world” values such as the use of electronic resources, material and book circulation, program attendance, reference services, and meeting room use. Perhaps the greatest advantage of this approach is that it can be pursued using readily available data, as opposed to contingent valuation that relies on patron surveys and interviews.

A meta-analysis of findings from 38 previous library ROI studies found that, on average, the return value for public libraries is 4 to 5 times the amount invested. A study conducted by Library Research Service in 2009 found similar results in Colorado using a contingent valuation methodology. Although valuation findings should not necessarily be extrapolated out to a state or national level, overall they can—and do—show decision makers, patrons, and the public that libraries are a wise investment.

Note: This post is part of our series, “The LRS Number.” In this series, we highlight statistics that help tell the story of the 21st-century library.

Loneliness Reaches Epidemic Levels. Are Public Libraries the Cure?

Libraries are not traditionally thought of as social spaces. Stereotypes of older women glaring over thick-rimmed glasses to shush talkative individuals pervade our pop-culture references. However, studies show that public libraries foster social support and decrease isolation. At a time when loneliness is being deemed a public health crisis in the United States, libraries are uniquely positioned to offer up a cure.

Cigna, the global health service company, reported epidemic levels of loneliness in 2019 that continue to intensify. After surveying 10,500 adults, they found that three in five (61%) classify as lonely, a seven percentage-point increase from 2018. The results are based on a 20-item questionnaire developed to assess self-reported, subjective feelings of loneliness or social isolation. A score of 43 or higher indicates loneliness. The report’s findings show a national average of 45.7 out of a possible 80.

Loneliness is affecting people of all ages, all demographics, and across socio-economic divides. Gen Z (18-22) is the loneliest age bracket, with levels decreasing as people get older. Additionally, entry-level employees and executives are the two most likely groups to report always or sometimes feeling there is no one they can turn to, not feeling close to anyone, and that no one really knows them well. Hispanic and African American workers agree in higher numbers that they feel abandoned by coworkers when under pressure at work. So how do we reach such a wide-ranging cross-section of American society to address this epidemic? Open the door to a public library.

Libraries’ extensive population reach, their access to diverse sectors of the US population, the public trust they command, and their diverse geographic coverage favorably position them as a multi-sectoral strategy to advance public health. 95 percent of the US population live within a public library service area and as Donald Barclay writes, “Public libraries are perhaps the last remaining indoor public spaces where an individual can remain from opening until closing without needing any reason to be there and without having to spend any money.”

Research published in the Journal of Community Health shows that libraries can address social exclusion among structurally vulnerable groups, from homeless individuals to new parents. In Denver, a Community Technology Center team regularly visits the local day shelter to give participants bus tokens, a tour of the main library, and library cards. In New Jersey, a new parents’ support group meets weekly at the local library. LGBTQ youth who may not feel safe at home or on the streets can turn to a library as a designated safe space. Library programs such as Drag Queen Story Hour also reduce social exclusion by increasing acceptance of the LGBTQ community. Additionally, libraries decrease social isolation by offering programs that build community and foster relationships. The Lifetime Arts’ program operates across 13 states and 80 public libraries, providing writing, painting, choir, and dancing classes for older adults. For newcomers such as refugees and immigrants, libraries serve as critical spaces to foster social integration. In Hartford, Connecticut, the public library provided services to promote immigrant civic engagement, including a core group of volunteer immigrants to help newly arrived individuals with tasks such as accessing community and legal services.

The 21st century library is an intersection of people and purposes. As national health data highlights a critical need for connection, the social role of libraries should not be overlooked. However, additional research is needed to evaluate the impact libraries have on the overall social wellbeing of patrons and the untapped potential for the wider—lonelier—public.

Note: This post is part of our series, “The LRS Number.” In this series, we highlight statistics that help tell the story of the 21st-century library.

Evaluation Training Opportunity for Colorado Public Libraries

Are you wondering how to assess your library’s impact? Do you have a brilliant idea for evaluating a library program or service but don’t know how to get started? Did you participate in the Research Institute for Public Libraries (RIPL) and need some ongoing support to realize your big ideas? Do you supervise a public library evaluation coordinator or data librarian and find it challenging to position them for success? The IMLS-funded project Embedding Evaluation in Libraries: Developing Internal Evaluators is the training program for your library.

During a nine-month period from summer 2020 through spring 2021, teams from up to six Colorado public libraries will complete robust, outcome-based evaluation projects supported through a peer learning process that includes in-person and virtual training, coaching, and technical assistance.

To be eligible, participants must work in a Colorado public library (all library sizes and locations welcome!) and participate as a team of 2-4 staff from the same library, including the supervisor of the primary person working on the evaluation. At least one member of the team must have some previous education and/or experience in evaluation, assessment, or research methods, such as participating in a RIPL event. All team members must attend the Kickoff Institute July 27-28, 2020 at the Arapahoe Libraries Administration (Support Services) Building in Englewood, and commit to attending the 2-day Wrap-Up Institute the week of April 5-9, 2021 (exact location and dates TBD after the cohort is formed).

Want to learn more about this opportunity? We held a webinar on January 29 to provide information about participating in Embedding Evaluation and chat with a current California cohort member, Lisa Dale (Collection Services Manager, Sacramento Public Library), about her experience. The recording is linked from this webpage.

For more information about Embedding Evaluation in Libraries: Developing Internal Evaluators, see https://www.libraryeval.org/.

Questions? Please contact info@libraryeval.org.

This project is made possible in part by the Institute of Museum and Library Services grant # RE-13-19-0076-19.

How Libraries Help Students Evaluate Information in the Era of “Fake News”

In the digital age, algorithms based on personal data allow information to find us as opposed to the other way around. Have we adequately prepared the next generation, who are predominantly accessing the world through social media platforms and search engines, to filter through the noise?

In a 2019 survey of US secondary school librarians, 96 percent said they teach some form of information literacy. These research skills allow students to “discover and evaluate credible information effectively and ethically by thinking critically.” In other words, students learn to identify biased reporting and suspicious sources, which can be applied to everything from Instagram posts to citizen news sites. Yet, according to a 2017 survey of academic librarians, the knowledge is not being retained—only 28 percent of first-year students enter their institution prepared for college-level research. If information literacy is widely taught in schools, why is there such a large gap?

Librarians site a “lack of time” (69 percent) and “lack of faculty support” (59 percent) as the biggest challenges to instruction. “I don’t think [faculty] see these skills as important. They also feel so pressed for time covering their curriculum that these skills fall to the wayside.” Survey respondents candidly admitted that higher-ups in the educational food chain don’t see a critical value in information literacy skills. The lack of prioritization from administrators trickles down to teachers who often fail to prioritize “non-tested” material. Librarians note the difficulty in finding instruction time for students and the lack of integration and reinforcement of these skills across all curricula.

Amid the frustration, some respondents offered that one solution could be to start younger. “Students are so hands on with tech, even BEFORE entering preschool, focusing on these skills at the high school level seems too late,” noted one librarian—and they’re right. A 2015 study by Pearson found that 53 percent of 4th and 5th graders and 66 percent of middle school students regularly used a smartphone. Yet, only 28 percent of children learned about “seeking multiple perspectives” prior to entering high school. How to effectively use open web resources was more likely to be introduced in grades 10 and above, meaning there are years of access to information without proper education on how to appraise it.

One librarian who participated in the research study offered this as a final thought, “In our world of ‘fake news,’ teaching our students how to find accurate news sources and how to evaluate them is critical to have well-informed citizens.” Information literacy education needs to be addressed by understanding the critical role librarians play in laying the foundations for information consumption. Librarians are necessary for teaching students how to evaluate sources and make informed decisions to navigate a world increasingly embedded in the internet and social media.

Note: This post is part of our series, “The LRS Number.” In this series, we highlight statistics that help tell the story of the 21st-century library.

International survey shows large adult literacy gap between the more and less educated in the United States

The Organisation for Economic Co-operation and Development (OECD) assesses adult competency on an international scale, using the Survey of Adult Skills. The survey assesses adults aged 16-65 on literacy, numeracy, and problem-solving in technology-rich environments. The OECD focuses on these three skills because they are necessary for full participation in society, relevant to a variety of contexts, and learnable. The United States as well as Ecuador, Hungary, Kazakhstan, Mexico, and Peru participated in the most recent round of data collection in 2017, and the United States also administered the assessment in 2012-2014.

The results of the survey allow researchers to calculate averages across all OECD countries for several measures. In addition to skills in literacy, numeracy, and problem-solving, the relationships between these skills and other variables are calculated on average across all participating OECD countries. These averages provide a valuable baseline for comparing country-specific results.

The research found that the gap in literacy proficiency between high- and low-educated adults is greater in the United States than on average for other OECD countries. The research also shows that the relationship between parental education and the skills of adult children is stronger than it is in other OECD countries on average. Finally, the study found that higher proficiency in literacy has a positive impact on other beliefs and behaviors, like trust in others, political efficacy, and self-reported health.

What do these findings mean about literacy in the US relative to the rest of the world? While adult literacy proficiency is impacted by a person’s education and their parents’ education across the countries included in the study, those relationships are even stronger in the US. US adults who did not pursue further education and had parents with lower education levels had lower literacy proficiency than they might if they lived elsewhere in the world. Libraries strive to provide additional access to literacy skills, and these results show that the need for that access is still great, particularly for those who have less educational experience.

Note: This post is part of our series, “The LRS Number.” In this series, we highlight statistics that help tell the story of the 21st-century library.