A newcomer to the PISA tests went on to take top spot in all three categories – Shanghai. Not only did they take top spot but they did it by a huge margin. Let's take a look at the top 8 scores in each category, and the 32nd place country (i.e. in the middle of the 65 countries/regions whose scores were released):
Reading
556 Shanghai
539 Korea
536 Finland
533 Hong Kong
526 Singapore
524 Canada (yay!!)
521 New Zealand
520 Japan
. . . .
483 Greece (32nd)
Mathematics
600 Shanghai
562 Singapore
555 Hong Kong
546 Korea
543 Taiwan
541 Finland
536 Liechtenstein
534 Switzerland
. . . .
487 United States/Ireland/Portugal (tied for 32nd)
Science
575 Shanghai
554 Finland
549 Hong Kong
542 Singapore
539 Japan
538 Korea
532 New Zealand
. . . .
489 Italy (32nd)
Shanghai beat the second-place entrant by 17-38 points, yet the difference between 2nd to 3rd, 3rd to 4th etc, was a mere 1-9 points, with an average of 4.1, and a change of 56-73 points is enough to take you from 2nd all the way down to 32nd. No country was consistently 2nd or 3rd etc in all three categories. That is just to give you an idea of how wide a margin Shanghai did on the PISA test. It's a fantastic result, an incredible result.
It's too good of a result. Waaaaaaaay too good. Something doesn't quite add up here.
The sample size for these tests is huge. According to the PISA website, countries are required to test at least 5,000 students, or an entire cohort if the country is small and does not have that many students in the age category. What are the odds that a group of 5,000 students (or more) would be so dramatically better on average than other groups of a similar size? I'd say near-impossible odds. (quick, someone call the author of
Freakonomics!). One cannot have such a large data pool of random students and yet have such a dramatic variance at the top end.
Now before I continue I just want to make one thing clear --
it is not unreasonable for Shanghai to have taken the top spot. East Asian countries have performed incredibly well on the PISA test every time it has been given, and countries like Japan, Korea, Hong Kong, and Singapore are regularly in the top five in all categories. Education is highly valued in these countries, expectations are high, and there is a lot of parental involvement and monitoring. Children in these countries are expected to study hard and do well in school and PISA results have consistently reflected that. And Shanghai is China's biggest and wealthiest city so I have no doubt that they have some of the best schools in China. There is no reason why Shanghai couldn't be leading the pack.
But no way by that much.
So what happened? Well, I'm not entirely sure. I'm not involved with PISA and I do not know what all of their processes are in administering the test. But let me speculate . . .
Perhaps the educational ministers or officials in Shanghai were under a lot of pressure to see good results.
Countries tend to see the PISA test as a way to see how their students are performing and try to see if there are weaknesses in the educational system for which a government can introduce initiatives to try to fix, or to identify good performing countries so that other countries can look to see what exactly those countries do and perhaps emulate it. It is a great learning tool for governments. If you take that approach to the PISA test then it is highly unlikely you are going to try to manipulate the results in order to look better than other countries. You need real, honest data about how your country is doing. Because of this I'm not sure if PISA’s controls are particularly strict to ensure countries don't manipulate the results. What would be the point of taking the test if you are not interested in honest results, right? Well, if you were worried that your career was at stake maybe you'd be more inclined to provide the results you think your boss wants.
So it is possible that officials in Shanghai were very worried about repercussions from the Central Government if results were less than stellar, which would give them an incentive to improve the results. This is not uncommon with one-party centralized governments, officials who fear repercussions give the story that their superiors want to hear. Since this is an outside test perhaps some officials, worried that the students would show poor results compared to countries in their region, tweaked things a little somehow, and tweaked things a bit too much.
How? Here are some possible ways:
1) Have students older than the required age take the test.
2) Select only specialized schools or top-end schools
3) Review the test before handing them in and correct a few mistakes (I believe
Freakonomics had a tale on this regarding standardized testing in the US)
4) Review the test beforehand and inform students of some of the answers, or make sure that material is covered in classes before the exam is given.
5) Pretest the students on a similar exam (perhaps PISA 2003 or PISA 2006) and only have the higher-performing students take the 2009 exam.
6) Some or all of the above
Now maybe PISA has some strong controls or prevent many of these, but they would not be able to prevent all of them (number 1 or 5 for example). Anyway, if a country or an education Ministry really wanted to manipulate the results they could.
I would love if someone could get a good statistician and run the odds that Shanghai would pull so far ahead in all three categories given the size of the student pool. If anyone does this let me know. Until such time I will always be skeptical about the results posted for Shanghai.