A new education research brief from public school advocates, Save Our Schools (SOS), says PISA international league table rankings are misleading due to a large proportion of the 15-year-old population in many countries not being in school.
The brief pointed to Vietnam – one of the standout performers in the results from PISA 2015 – as an example.
It said that while Vietnam achieved a ranking of 8th in science, half of its 15-year-old population was not covered because they were not in school.
The OECD’s PISA 2015 report shows that the sample of Vietnam students participating in PISA was taken from only 48.5% of Vietnam’s 15-year-olds. This was the lowest coverage of 15-year-olds in all the countries participating in PISA.
The next lowest coverage was Mexico at 61.7%. Australia’s coverage was 90.6% and most OECD countries had over 90% coverage.
‘Artificially inflated scores’
SOS national convenor, Trevor Cobbold, pointed to an analysis of the results by John Jerrim, an academic at the Institute of Education at the University College of London.
Jerrim suggested that Vietnam’s PISA science scores are artificially inflated because those who were not covered by the sample are likely to be academically weaker than those who were tested.
“He [Jerrim] assessed the impact of the low coverage of 15-year-old students by comparing its science results with other PISA data where it is assumed that 15-year-olds who are not in school perform at or below the national median,” Cobbold said.
“He estimates that the ‘real’ performance of Vietnam is probably 50 to 60 points below its mean science score. This would put Vietnam’s ranking in science at 35-40th – a far cry from 8th.”
Cobbold pointed out that Australia’s coverage of 15-year-old students at 90.6% is in the mid-range of OECD and top 20 country rankings by mean scores. Its ranking on median scores is slightly affected.
Australia’s median score of 515 excluding 15-year-old students not in school drops to 500 when they are included.
As a result, Australia’s international ranking on median scores falls slightly from 13th to 15th, Cobbold pointed out.
However, he says Australia is doing much better than Vietnam, the four Chinese cities/provinces and the UK, despite the perception given by the published PISA rankings.
“Australia has high test scores and a high proportion of 15-year-olds in schools, although there is room for improvement on both measures,” Cobbold said.
“The analysis demonstrates that caution is needed in interpreting the PISA international league tables.”
‘OECD should rethink scoring method’
Cobbold said that not only do scores and rankings change according to whether mean or median scores are used, but they can be significantly different if the coverage of 15-year-olds varies significantly between countries/cities/regions.
“Country and city test scores that include 15-year-olds not in school are lower than when they are excluded,” he explained.
“In some cases, such as Vietnam and the four Chinese cities/provinces, there is a very large difference in scores that amounts up to two or more years of schooling.”
Vietnam, the Chinese cities/provinces and the UK all drop out of the top 20 PISA rankings when account is taken of the proportion of 15-year-olds not in school.
Cobbold said this was “a stark reminder” of how the proportion of 15-year-olds in schools can have a very large impact on country/city test scores and international rankings.
“A country, region or city with high test scores can hardly be considered successful education performers if a large proportion of its 15-year-olds are not in education,” he said.
“The OECD should consider not including countries/regions/cities with a high proportion of 15-year-olds not in school in its international rankings because they can significantly distort the rankings and make them highly misleading.”