How Can We Measure Edtech's Return on Investment?

School Infrastructure

How Can We Measure Edtech's Return on Investment?

By Stacey Wang     Aug 18, 2016

How Can We Measure Edtech's Return on Investment?

One day at Stanford D.School (Institute of Design at Stanford), an Oakland team and I were building paper planes as part of a design thinking activity. We thought our goal was to build the “best” plane so that’s what we built; in actuality, the D.School wanted us to land the plane in a 2x2 target on the other side of an opague wall. When we flew all six planes over the wall, we had no idea how well we performed - did anything land in the target zone? Were we even remotely successful?

We quickly realized this experience was an analogy to how school districts normally design initiatives. Districts design a well thought out solution to schools’ needs (like a paper plane), work with the best intentions to tirelessly get something perfect (a plane that flies far), and then try to get these solutions to “land” well at schools (in the target zone). Unfortunately, it is often challenging to design a feedback mechanism to understand if the new initiative – whether it be a new curriculum adoption, device rollout, or new assessments - will really work. How do systems know when their initiatives “landed” successfully to support teachers and enable students to thrive?

In Oakland, our teams have been working to avoid the “flying a plane blind” approach to school support. Instead, many teams have adopted a teacher-centric approach to designing initiatives, such as those that are technology-related, and we have started assessing the ROI of these initiatives based on educator and student feedback. For example, with technology we have engaged with BrightBytes’ Technology and Learning module to measure the progress of our tech programs over time and get evidence to inform future initiatives.

Based on data from a majority of schools with a statistically significant sample size, we see that the tremendous efforts of the Technology Services team has paid off. The foundational technology pieces – Chromebooks, internet, assessment platforms – are stabilized at most Oakland schools. We’re doing well strong in terms of technology access at schools and home and around foundational skills. However, the school environment, in terms of teacher support and space for using technology to support learning, is not as strong.This impacts the level of capacity teachers have to bring technology into the classroom. As a district we have done the heavy lifting of shifting our curriculum to meet CCSS in ELA and in Math, but the technology integration of digital skills is still a work in progress.

We’re seeing plans for the future in this feedback, but the most important pieces of this work are the lessons we have learned about how to roll out technology initiatives. They include:

Access is just the first piece of the puzzle. A tool without a purpose doesn’t do any good. As the Tech Team was planning to roll out devices, the team immediately recruited a teacher at every school site to be an Instructional Technology Teacher Leader to support the adoption of devices, prep for SBAC and the integration of devices to support classroom instruction. These teacher leaders were critical because they ensured that technology had a space for conversation at school sites, and that teachers were using the devices for classroom activities beyond just SBAC and formative assessments. They were the pioneers who created trainings for their fellow teachers around tech usage, pushed Chromebook carts around the school and managed cart scheduling, ensured students had the right access by being the Google admins at the site level, and much more.

Ask your teachers for feedback and really listen. Our Technology Team has always engaged in surveys to understand teacher needs as they planned their technology roadmap for the year. Back in 2013, Oakland teachers told us that wifi was inconsistent and too slow to use any ed tech programs and they didn’t have devices outside of the occasional lab time to really build consistent use of programs for students. So the next year, the district purchased 10,000 chromebook devices on wifi-enabled carts and rolled them out equitably across our 86 schools and 36,000 students. Concurrently, the Tech Team also worked with a contractor to install new wireless – that is 3,333 wireless access points in every classroom and instructional space. This year, teachers told us that they still needed more devices as they were increasingly implementing blended learning models, so schools received another 8,000 Chromebooks. Our total change over a short three years? State of the art wifi in every school and 18,000 devices across 37,000 students (just about a 2:1 ratio when you factor in other site-based purchases of technology).

Create time for teachers to learn from teachers. In the Instructional Technology Teacher Leader monthly meetings, the focus was on how to integrate technology to support rigorous academics. Understanding of how to do this well was nascent when we started in 2014, but over time, as teachers developed strong practices, we started spotlighting teachers and sharing their practices with other teacher leaders during these meetings. For example, one of our rock star elementary teachers effectively uses Google Classroom to support balanced literacy. He ran a model lesson at one of these meetings with his fellow teachers who were interested in Classroom and it was rated one of the best trainings the teachers ever received in the monthly meetings. Teachers learn best from one another because teachers learning new skills often struggle with certain nuances that only others in their shoes will be able to help troubleshoot and advise.

We've certainly come a long way in Oakland. And yet we know we have plenty of work left to do to continue to support our teachers. Only by taking the time to create systems for teacher and leader feedback will we be able to successfully ensure intentions translate to effective support for teachers as they shift towards a digital learning environment. Here is to another year of landing more planes in the target zone so every student thrives.

Stacey Wang is the Advisor to the Superintendent at the Oakland Unified School District.

Learn more about EdSurge operations, ethics and policies here. Learn more about EdSurge supporters here.

More from EdSurge

Get our email newsletterSign me up
Keep up to date with our email newsletterSign me up