Measuring COVID Learning Loss

“A lot of the reason I got into testing policy was because the seemingly arcane decisions made by statisticians in back rooms actually have really profound implications for equity and fairness,” he said. “It’s pretty technical work, but it’s always driven by these questions of equity in the education system.”

In addition to his faculty role, Soland holds a position as a research fellow at NWEA, an education nonprofit that administers a widely used test called MAP Growth. MAP Growth was designed to measure how students progress in math and reading skills. Three times a year, about one out of every 10 schools in the country give MAP Growth tests to third- through eighth-graders, providing a rich dataset for researchers.

Last year, Soland and colleagues at NWEA dug into past data on learning loss – including studies on school absences and closures caused by natural disasters – to estimate the potential impacts of COVID on student learning. They predicted learning losses in both reading and math, with steeper losses in math.

Jim Soland, an assistant professor of education, is driven to understand how seemingly minor decisions about educational measures ultimately affect students. (Photo by Sanjay Suchak, University Communications)

Now, they finally have the data to see how real results compared to their predictions. Using data from fall testing of more than 4 million students in public schools across the country, Soland and his colleagues looked at how much students’ learning has grown since schools closed in March, as well as how students performed relative to a typical school year.

The results were mixed. On the bright side, the data showed that students did make learning gains on average during the pandemic, and reading scores in 2020 were mostly comparable to students in the same grade in 2019. However, scores on math achievement were more concerning. On average, students across the board scored about five to 10 percentile points lower compared to the prior year.

“Math definitely seems to be where people should … be most worried,” Soland said.

The data are preliminary, and much is still unknown about the extent of learning loss and how it may vary within different communities. Critically, researchers are concerned about signs that many students have dropped out of the education system altogether – meaning some of the most at-risk students likely aren’t showing up in the data at all. The less severe drops than anticipated in reading could, for example, be related as much to who is being tested as to how those students are performing.

Still, Soland said the early analysis is a valuable first step toward recovery.

“I think, deep down, everybody knows and suspects that [the pandemic] hasn’t been good for student learning,” he said. “But having concrete data that show just how big the drop-off has been in math is important not just for parents and teachers to know, but also for policymakers as they’re debating how to support schools.”

Plus, early data can also inform how states, districts and teachers begin to plan for how to move forward and address learning loss – both specific to COVID and in general. Evidence pointing to steeper losses in math can help shape how educators approach the task of helping students who are falling behind.

“If COVID has shown anything, it’s shown that the vast majority of teachers are bringing their A-game,” Soland said. “They are working hard for the kids and want to be as good as they can be. I think this is a chance to switch the accountability framework to one that starts with that assumption, and then asks, ‘How can we give teachers data to help them improve teaching and learning as much as we can?’”

As more data roll in during the months ahead, Soland hopes to see more research focused on equity – particularly achievement gaps related to income and race. While researchers weren’t able to drill down into specific demographics in this particular study, “It’s reasonable to suspect that this is affecting low-income communities more than it’s affecting high-income communities.”

Then, there are the bigger-picture questions related to the many other ways the pandemic has altered life for students and their families that researchers will be exploring for years to come. “If we look at challenges related to poverty and food insecurity and job loss and achievement and social-emotional outcomes, how do we connect the dots and get a more comprehensive picture of what this pandemic has meant for kids in K-12 schools?”

For now, however, early studies like this can provide some light in the darkness as educators navigate crucial next steps.

“There are some very fast-approaching decisions that will be really important for schools that we all need to think through,” Soland said. “Trying to use these data to help with that is part of the goal.”