Education is rife with contenious issues, starting with what constitutes "evidence" of the effectiveness of different teaching models. To this end, we offer a few definitions aimed at clarifying what different people mean when they referece to research.
Reports: Reports are descriptions or case studies of scenarios. Some reports take a bird's eye view, describing opinions of experts and thought leaders. Well-researched newspaper stories can be reports; so, too, are analyses put together by private or non-profit think-tanks or university groups.
Although some reports may aim to present "all sides" of an issue, many reports have a political agenda and may be making the case for a particular point of view. Readers need to be mindful of the organization creating the report and whether the group is advancing a cause.
Results: Results typically are reports rooted in quantitative information such as a survey or a study. There may not be deep methodological or statistical rigor to a results. "Results" can be tricky to assess because data often sounds convincing but there may be subtle biases or weaknesses in how the results were gathered. "SurveyMonkey" polls, for instance, are "results" that provide an indication of trends.
Like reports, "results" can often be used to prove a point. Once again, readers need to be mindful of the source of the results and should investigate how the results were gathered: how many people contributed to the results? Over what period of time? Readers should also be aware that "results" may omit or merge data that don't support a particular thesis.
Research: "Research" refers to studies conducted by independent organizations using methodologies that are recognized as key to establishing credible, statistically valid and unbiased conclusions -- in this case on the effectiveness or efficacy of a product or service on educational outcomes.
Among the factors that readers should consider when evaluating research reports:
* Does the study include enough data to draw meaningful conclusions?
* Is the duration of the study long enough to draw meaningful conclusions?
* Is there a "control" group (comparable to the group studied) to help gauge effectiveness?
* Who paid for the research?
* How much support and training did the company provide for teachers using the technology?
* Has the organization made the full report available so that all the data can be independently reviewed?
There are a growing number of organizations supporting independent assessments of education technology. Among them:
* The Education Innovation Laboratory at Harvard University (run by Dr. Roland G. Fryer, Jr.) seeks ways to use data to explore claims about education effectiveness. ("We squeeze truth from data."
* The Software & Information Industry Association, which in November 2011, published a report: "Conducting and Reporting Product Evaluation Research." (Here's the executive summary; the report is available for free to teachers and to SIIA members; it costs $125 for others.) It includes a list of suggested guidelines for companies planning, designing, conducting (or commissioning) and reporting on their products and services.