In the last two years, the U.S. Department of Education has awarded $383 million in Race to the Top competition funds to districts implementing blended learning models. But amid the models of success, “how to” guides, and lessons learned, one simple question remains: Do blended schools produce better student outcomes?
The preliminary answer: on average, early blended schools appear to perform as well or better on California statewide standardized tests than do non-blended schools serving demographically similar student populations. These findings are based on a study of 35 rotation-model blended learning charter and district-run public schools in California. Several of the early blended schools are operated by national and regional charter management organizations, including Aspire, Rocketship Education, and Summit in the San Francisco Bay Area, Alliance College-Ready Public Schools in Los Angeles, and KIPP schools in both LA and the Bay Area.
Number and distribution of schools in the study.
Results and Methodologies
While student outcomes still vary widely across the blended schools studied, blended learning, on average, appears to be associated with gains of up to 0.84 standard deviations in math achievement (with a standard error of 0.18) and up to 0.42 standard deviations in ELA achievement (standard error of 0.17) on California statewide assessments. These gains translate to an increase in scaled scores on the California Standards Tests (CSTs) of approximately 40 points and 20 points on math and ELA tests, respectively, scored on a 600-point scale. These gains represent the difference between a mid-range score in the “Below Basic” level and a score reaching the “Basic” level, or the difference between “Basic” and “Proficient.”
The findings are based on public data obtained from the California Standardized Testing and Reporting (STAR) Program for the school years 2006-07 to 2012-13.
The main methodology used in the current study was a matching estimator comparing the average CST scores of blended schools to a “matched” pool of similar non-blended schools selected based on demographic traits--such as gender, ethnicity, English learning status, and eligibility for free and reduced price meals--of each school’s student population. For each of the 35 blended schools in the study, I found up to three non-blended schools that serve the most similar student demographic.
To supplement the matching methodology, the study also compared student test score outcomes for schools that had converted from a traditional instructional model to a blended model, as well as schools operated by charter management organizations that ran both blended and non-blended schools within the same district, such as KIPP, Aspire, and Alliance. In contrast to the matching methodology, which showed strong positive gains associated with blended learning in both math and ELA, the supplemental methods did not find a statistically significant difference between performance in blended and non-blended schools in either subject.
Click for larger image
The graph above shows the range and average performance of the blended schools and non-blended comparison groups. The CST test score data are standardized to a California statewide mean of 0 and standard deviation of 1 for each test year. A standardized score of “0” represents the California statewide average. Each of the plotted points represents the average test score (in math or ELA) for one school in a particular test year, and each of the lines represents the average test score for one of the comparison groups (green for all 35 blended schools, yellow for the “matched” pool of non-blended schools, and red for all non-blended schools within the districts studied).
A word of caution
Before we start cheering, know that these gains are both encouraging and cautionary. Yes, we see that some blended schools produce student outcomes significantly better than those of their non-blended peers serving similar students. Yes, we can at least say that blended schools, on average, performed no worse than their non-blended peers.
However, these above-average test scores are by no means guaranteed. As the number of blended schools has increased sharply in the past few years, so too has the variation in test scores. As a result, the average test score performance of blended schools has decreased. Even for blended learning pioneer Rocketship Education, early successes have been difficult to replicate. As its network of charter schools has grown, scores on California statewide assessments have declined, with the number of students scoring “proficient” or above in math and ELA declining by 14 and 30 percentage points, respectively, over the past five years.
In addition, in the absence of a more controlled research setting, it is difficult to determine exactly how much of the achievement gains can be attributed simply to “blended learning,” an umbrella term that encapsulates a wide range of practices and resources, both human and material. The methodologies used did not look at school culture, principal leadership, teacher ability, and other factors that can affect student outcomes.
“Blended learning has tremendous potential to improve the quality of education and lower the cost,” noted Alan Krueger, former White House Chief Economist and Bendheim Professor of Economics and Public Affairs at Princeton University, where this study was conducted. “Given the encouraging results from this study, I think it would be helpful if researchers could conduct a larger evaluation of blended learning techniques using some element of randomization.”
The main takeaway? That research on blended learning must keep pace with the innovation. “It is critically important that studies like this gather and analyze evidence to empirically evaluate the efficacy of blended learning and that we build on what works,” said Krueger.
