(1).jpg)
When global education rankings are released, the results are often used to boast of educational excellence, or to argue for reforms aimed at lifting teaching and learning standards.
However, new research – published in Science by Judith Singer of Harvard University and Henry Braun of Boston College – shows that these rankings can often be misleading, creating false comparisons and risking policy confusion.
The study was based on work undertaken by a National Academy of Education steering committee that examined tests such as the Program for International Student Assessment (PISA) and the Progress in International Reading Literacy Study (PIRLS).
According to Singer and Braun, the biggest message for principals and education policy makers is the need to avoid over-emphasising global education rankings when assessments are released.
Singer said that while these tests provide crucial insights into how education systems are performing, the rankings are held in such high regard that “they become the statement of truth”, and the data underlying the rankings – which is much more nuanced than the rankings themselves suggest – gets lost.
So what does this mean for those breaking down the data and reporting the results?
The study suggests that instead of drawing from the simplistic conclusions of top-level rankings, more credence should be given to reports that break ILSA data down on a regional or local level and combine it with another source of data, such as surveys.
Another suggestion included comparing national analyses across different, but culturally similar, countries and looking at relationships between different indicators of achievement.
For example, the National Academy of Education steering committee reviewed studies that found while Hong Kong and Taiwan had similar scores on the 2012 PISA test, the relationship between scores and socio-economic status (SES) was three times greater in Taiwan than Hong Kong.
The committee also found similarities in a study examining Canada and the United States. Canada’s mean score was 37 points higher than that of the United States, but the relationship between scores and SES was much weaker in Canada.
Singer and Braun said that these types of analyses are more likely than rankings to lead to productive exploration of these equity issues, but they often don’t have the same public impact.
The study laid out five strategies for improving ILSAs and enhancing impact:
- All partners – the assessment organizations, testing contractors, policymakers, and media – should move to deemphasize rankings.
- When ILSA data is released, they should be linked to other sources of data, so that more nuanced and meaningful analysis can happen more quickly.
- Researchers and policymakers should take advantage of the move to digitally based assessments and the ability to glean more useful and accurate data.
- Build pilot programs to add longitudinal tracking components that measure learning over time.
- When an ILSA release spurs consideration of policy changes, use the ILSA data to trigger randomized field trials among like countries to test the effects of specific interventions.
The original version of this article appeared in Usable Knowledge.