School decision-makers are often faced with making choices amongst a variety of educational programs to serve their students. What criteria should apply in making such decisions?
First, the decision-maker needs to assess whether the program is likely to help his or her particular students succeed. Ideally, evidence of program efficacy gathered from experimental studies demonstrates that students engaging in the program achieve better educational outcomes than students not engaging in the program. Second, the program must be affordable given prevailing budget constraints. The decision-maker needs a good estimate of what the program costs to implement in practice—which is different from the “sticker” price advertised by the vendor. Third, the required personnel, equipment and facilities must be available to allow for high-fidelity implementation. Implementing a program often requires diverting existing personnel, facilities, and other resources from current activities to new ones. It is important to recognize these resource demands to avoid inadvertent strain on currently operating programs. But how to think about all three things at once?
Cost-effectiveness analysis combines all this information in one analysis, simultaneously allowing decision-makers to compare the resource requirements, costs and effectiveness of programs and make informed choices among them. It is a useful evaluation method for assessing how efficiently resources are being used in schools and which programs provide the greatest educational returns on investment.
At the Center for Benefit-Cost Studies of Education ( CBCSE), based at Teachers College, Columbia University, we use a free online tool, CostOut, to facilitate this kind of analysis. CostOut helps estimate the costs of educational programs so that education decision-makers can identify financially feasible programs. It allows us to compare the resource requirements and costs of several programs at once.
To ensure that full costs of implementation are captured, CostOut follows the “ingredients method” of estimating costs. This involves listing all the resources or “ingredients”—personnel, facilities, materials, equipment, transportation, and so on—needed to implement each educational program under consideration, and how much of each is needed. Drawing from a database of around 700 prices of educational resources, a price is assigned to each ingredient. For example, the price of a half-time 5th grade teacher might be half the national average annual salary for a K-6 teacher, plus fringe benefits. CostOut takes care of any necessary adjustments such as inflation, present value, amortization, deprecation, and geographical location. It presents us with a total program cost and cost per program participant.
We are also able to spread the costs across various “funding agencies.” For example, a school may be responsible for 100% of the costs of its teachers, but maybe the district pays partially for teaching aides and facilities costs. And perhaps a local foundation contributes laptops. This feature allows the decision-maker to account for all the resources required for successful program implementation even if there is no actual financial outlay on his or her part for some of the ingredients.
In addition, if we have data on how much the program improves student achievement, CostOut calculates cost-effectiveness ratios for each program. For example, an analysis of math programs could show how much each program costs per unit of gain on a math test, and an analysis of dropout prevention programs could show how much each program costs per extra student who graduates. The results of such an analysis can be represented visually (see chart below) so that the decision-maker can see the potential trade-offs between costs and effectiveness. While it would always be ideal to find programs that cost less than business-as-usual but still yield better results, this rarely happens. It is more likely that we have to pay more for better results. Depending on prevailing constraints and priorities, the decision-maker may aim for the program that yields the most improvement regardless of costs, or may need to find the least costly program that still provides acceptable results.
Let's take one analysis as an example:
The user is deciding among four different after-school math enrichment programs. Program 1 (the orange triangle) produces the greatest gains in math achievement for students but it is also the most costly at $10,000 per participant. If money is not an issue and the School Board is determined to improve math achievement, this may be the best bet. Program 2 (the green square) produces reasonable gains compared with business as usual, which is represented by the mid-point (0,0), but not as many as Program 1. However, it actually saves a little money. It falls in the “cost-effective quadrant” because it costs less than business as usual but still produces better outcomes. That could be the safest bet. Program 3 (the black diamond) is slightly less effective than business-as-usual but it saves a lot of money, around $6,000 per student. This may be worth considering if a budget cut is looming. Program 4 (the blue dot) is an all-out loser—the decision-maker would be paying up to get worse outcomes. Anything in the “cost-ineffective quadrant” can be wiped off the list right away.
