A newcomer to the PISA tests went on to take top spot in all three categories – Shanghai. Not only did they take top spot but they did it by a huge margin. Let's take a look at the top 8 scores in each category, and the 32nd place country (i.e. in the middle of the 65 countries/regions whose scores were released):
Reading
556 Shanghai
539 Korea
536 Finland
533 Hong Kong
526 Singapore
524 Canada (yay!!)
521 New Zealand
520 Japan
. . . .
483 Greece (32nd)
Mathematics
600 Shanghai
562 Singapore
555 Hong Kong
546 Korea
543 Taiwan
541 Finland
536 Liechtenstein
534 Switzerland
. . . .
487 United States/Ireland/Portugal (tied for 32nd)
Science
575 Shanghai
554 Finland
549 Hong Kong
542 Singapore
539 Japan
538 Korea
532 New Zealand
. . . .
489 Italy (32nd)
Shanghai beat the second-place entrant by 17-38 points, yet the difference between 2nd to 3rd, 3rd to 4th etc, was a mere 1-9 points, with an average of 4.1, and a change of 56-73 points is enough to take you from 2nd all the way down to 32nd. No country was consistently 2nd or 3rd etc in all three categories. That is just to give you an idea of how wide a margin Shanghai did on the PISA test. It's a fantastic result, an incredible result.
It's too good of a result. Waaaaaaaay too good. Something doesn't quite add up here.
The sample size for these tests is huge. According to the PISA website, countries are required to test at least 5,000 students, or an entire cohort if the country is small and does not have that many students in the age category. What are the odds that a group of 5,000 students (or more) would be so dramatically better on average than other groups of a similar size? I'd say near-impossible odds. (quick, someone call the author of Freakonomics!). One cannot have such a large data pool of random students and yet have such a dramatic variance at the top end.
Now before I continue I just want to make one thing clear -- it is not unreasonable for Shanghai to have taken the top spot. East Asian countries have performed incredibly well on the PISA test every time it has been given, and countries like Japan, Korea, Hong Kong, and Singapore are regularly in the top five in all categories. Education is highly valued in these countries, expectations are high, and there is a lot of parental involvement and monitoring. Children in these countries are expected to study hard and do well in school and PISA results have consistently reflected that. And Shanghai is China's biggest and wealthiest city so I have no doubt that they have some of the best schools in China. There is no reason why Shanghai couldn't be leading the pack.
But no way by that much.
So what happened? Well, I'm not entirely sure. I'm not involved with PISA and I do not know what all of their processes are in administering the test. But let me speculate . . .
Perhaps the educational ministers or officials in Shanghai were under a lot of pressure to see good results.
Countries tend to see the PISA test as a way to see how their students are performing and try to see if there are weaknesses in the educational system for which a government can introduce initiatives to try to fix, or to identify good performing countries so that other countries can look to see what exactly those countries do and perhaps emulate it. It is a great learning tool for governments. If you take that approach to the PISA test then it is highly unlikely you are going to try to manipulate the results in order to look better than other countries. You need real, honest data about how your country is doing. Because of this I'm not sure if PISA’s controls are particularly strict to ensure countries don't manipulate the results. What would be the point of taking the test if you are not interested in honest results, right? Well, if you were worried that your career was at stake maybe you'd be more inclined to provide the results you think your boss wants.
So it is possible that officials in Shanghai were very worried about repercussions from the Central Government if results were less than stellar, which would give them an incentive to improve the results. This is not uncommon with one-party centralized governments, officials who fear repercussions give the story that their superiors want to hear. Since this is an outside test perhaps some officials, worried that the students would show poor results compared to countries in their region, tweaked things a little somehow, and tweaked things a bit too much.
How? Here are some possible ways:
1) Have students older than the required age take the test.
2) Select only specialized schools or top-end schools
3) Review the test before handing them in and correct a few mistakes (I believe Freakonomics had a tale on this regarding standardized testing in the US)
4) Review the test beforehand and inform students of some of the answers, or make sure that material is covered in classes before the exam is given.
5) Pretest the students on a similar exam (perhaps PISA 2003 or PISA 2006) and only have the higher-performing students take the 2009 exam.
6) Some or all of the above
Now maybe PISA has some strong controls or prevent many of these, but they would not be able to prevent all of them (number 1 or 5 for example). Anyway, if a country or an education Ministry really wanted to manipulate the results they could.
I would love if someone could get a good statistician and run the odds that Shanghai would pull so far ahead in all three categories given the size of the student pool. If anyone does this let me know. Until such time I will always be skeptical about the results posted for Shanghai.
8 comments:
Hey pal, I am a twenty-something Chinese guy who have gone through all the education processes within mainland China, after reading your "Skeptical about Shanghai" post, I have a few thoughts to share.
1st, I am not surprised at all that the Shanghai students have taken the top spot in the PISA test. By that, I mean Chinese students generally do work much harder (years of longer school-hour and higher intensity of academic activities)than most students in other countries, not because they are so very self-motivated that they voluntarily apply themselves to the hard work and sometimes grueling pain of hitting the books when you are just 1-page away from vomiting, but the pressure from the peers, teachers, parents and the entire society. The thing is, little kids in China were told that if they don't work hard and do well in school, he/she is gonna live a mediocre and miserable life. Not that I am going to check the validity of the point made above, but you get a sense of competition going on in China. Because most Chinese are considered poor by International Standards, excellence in academics is a "golden ticket" out of poverty. When your life is literally is on the line, you (or the people around you) really push you(self) hard, I would assume the same principle applies to other developing countries with huge population(i am assuming huge population equals severe competition) like say, India.
2nd, I don't really agree with the notion that academic levels achieved by students in Shanghai can be copied by students elsewhere in China later. It is common sense in China that students in big cities in China like Beijing & Shanghai are usually out-performed by students from other less-developed provinces as far as standard test scores goes. I guess it's got something to do with the fact that students in big cities are generally more comfortable than their less-fortunate peers (financially speaking), the drive for excellence may not be that intense. It'd be interesting to see students from different provinces from China battle it out on this PISA test. Just a thought. :)
3rd, about your concern with the possibility of manipulation of test results. I don't really know, and I understand that when it comes to international image, the word China usually doesn’t remind you of nice memories, and I am with your guys on this one.
4th, back-flash of the results in China. Don't get me wrong, Chinese love to be No. 1. But on the main news portal sites in China, you'll find criticism appear much more often then appraisals (I actually haven’t found a positive review of this result yet). Good for nothing except for super test-takers... the usual stuff they say about the things that are fundamentally wrong with the Chinese education system.
Hi JustTodd.
On your first point I agree, which is why I have no issue with the possibility of Shanghai topping the list. Patrental pressure and expectations to do well in school is common throughout East Asia, and is far more so than you will find in North America, and is a big factor in why East Asian countries place so highly on the test scores. My concern was that Shanghai posted scores so high above other East Asian nations. Had no other East Asian nation been tested I probably wouldn't have batted an eye at Shanghai scoring 50 to 80 points above European and North American nations. But 20 to 70 points over Japan, Korea, or other Chinese populations (Singapore, Hong Kong)?
I also really wish countries like India would sign on to PISA so we can see how their curriculum really does score. Given Qatar's low score despite Qataris not making up the majority of the student population it might be possible that students in the Indian curriculum schools might also be performing below OECD averages. I've no way to know unless India were to sign-up to PISA.
2nd. I too would love to see scores from other parts of China. The media reporting on the Shanghai results also tends to note that Shanghai would unlikely be representative of the country as a whole. In their defense I don't think the Chinese media or the Chinese government has ever stated that Shanghai represented typical Chinese student performance. Perhaps the Chinese government would be willing to expand the testing for the 2012 test.
4th. I didn't realize that! I'll have to take a look and see what I can find. I have seen some Western criticisms about how the Chinese curriculum district students are taking tests but because of its rote-memorization focus does little to encourage entrepreneurialship, the ability to work well in teams, and other social skills that would be necessary in office and professional environments. Sour grapes by the Western media perhaps?
INTERNATIONAL MATH OLYMPIAD 1990-2010
http://imo-official.org/results.aspx
1990 China 1st
1991 China 2nd
1992 China 1st
1993 China 1st
1994 China 2nd
1995 China 1st
1996 China 6th
1997 China 1st
1998 China Not Attending
1999 China 1st
2000 China 1st
2001 China 1st
2002 China 1st
2003 China 2nd
2004 China 1st
2005 China 1st
2006 China 1st
2007 China 2nd
2008 China 1st
2009 China 1st
2010 China 1st
No other nation even come close. China and Russia are the only nations that have achieved an all-members-gold IMO multiple times (China: 10 times in total, including years 1992, 1993, 1997, 2000, 2001, 2002, 2004, 2006, 2009, 2010; Russia: 2 times in 2002 and 2008).
While it is impressive what China has done in the mathematics Olympics, it still does not address the point because the math Olympics only tests the best minds in countries against each other. It would be the same as saying that the United States should have the highest PISA science scores because they've had the most Nobel Prize winners in science (and the United States is nowhere near the top in the science score). You have also noted that Russia is the only other country to have won more than one math Olympics, yet their Mathematics scores on the PISA test were not that impressive.
When you test large groups, in this case over 5000, to have a significant statistical difference from all other groups is so highly unlikely that I still suspect that something is going on with the Shanghai scores. I will reiterate that I do not have an issue with Shanghai being top on the PISA test as there are a number of factors as to why they could be, but the sheer difference in their scores between them and all other countries is just too good to be true.
Hi Glen,
A Chinese here.
Your firm assumption that they must be almost interchangeable peers (racially, intellectually and culturally) amongst Shanghai and other North East Asian entities, led you to reach the hypothesis that perhaps there's something fishy with the results.
To answer your question, I have to challenge that implicit assumption of yours: in fact they are not totally interchangeable, even not at all particularly when competing at this high level.
Now I have to list some related IQ averages here as IQ is the most relevant factor, though politically incorrectly speaking, in PISA and SAT etc highly g-loaded tests. And these high correlations among them have been well documented, researched and quantitatively established in HBD field. There are actually almost no surprises in those results from angle of the average IQs
-- e.g. it's almost impossible for Mexico somehow surpassed, say, Austria (or even North Korea to make my point) in maths/science scores no matter how many train loads of $$$ you pour into the former. I know that it’s not PC, but still. To respond one of your other curiosities here, so is with India to be brutally honest. It’s impossible for India to surpass the OECD average in PISA score ( IF it is a statistically VALID random sample of India) as India's mean IQ being 81 whereas OECD mean IQ being about 100. That's the base line supported by mountains of stats over decades. Don't believe it? Just wait and see the future scores.
That being said and argued, now Shanghai average IQ is about 112; HK /Singapore are about 108; Korea is about 107; Japan and Taiwan are about 105. See the general pattern here with the tests results? On top of these extra points on average IQ, Shanghai kids are more driven and hence likely worked harder as Just Todd said. Ask any older generation non-liberal & non-PC HongKonger or Chinese Singaporean, they would tell you, if they are perfectly honest, that if Shanghai were an independent entity after WWII, its GDP per cap could well be the top of the world by now. Empirically as an unspoken rule within Chinese communities, a group of Shanghainese would have more chance in winning over a group of Cantonese ( i.e. HKers, Singaporeans)on many areas due to being "smarter". I bet Singapore's current PM would also tell you such. On Ground Zero, ask yourself who is starting to challenge HK& Singapore's positions as the Asian Financial Centre? Yeah, Shanghai. And when Singapore wanted to invest in China, where in China it has set up high-tech industrial parks for R&D? Yeah, Suzhou ( the capital of JiangSu province which has basically the same people - Wu people - as nearby Shanghainese). Such examples are endless...Yet it's not your fault that you don't and can't see, simplely because you are not Chinese.
Therefore, the combination of above 2 factors ( aka IQ and study harder/more driven) could likely be enough to explain this test difference. Actually, your kind of "surprises" have mostly come from the western world. I, and most other Chinese (mainland + overseas) I believe, even couldn't imagine if Shanghai didn't score that high.
The second issue here is that "is Shanghai the high-end test score exception of China? ". NO, according to my knowledge and experiences in the field (and likely most other Chinese as well). GaoKao ( China's Annual College Entrance Exam) is the de facto and statistically valid yardstick to support my view, since Shanghai though being often among the top quartile , has been almost always surpassed by stronger provinces such as JiangSu, ZheJiang, FuJian, Sichuan,ShanDong, Beijing region, etc. Therefore, it wouldn't surprise me at all that some of these provinces ( if they also took PISA 2009) got even better Maths and/or Science test scores - to be published in the coming months.
Cheers!
Hi Glen,
A Chinese here.
Your firm assumption that they must be almost interchangeable peers (racially, intellectually and culturally) amongst Shanghai and other North East Asian entities, led you to reach the hypothesis that perhaps there's something fishy with the results.
To answer your question, I have to challenge that implicit assumption of yours: in fact they are not totally interchangeable, even not at all particularly when competing at this high level.
Now I have to list some related IQ averages here as IQ is the most relevant factor, though politically incorrectly speaking, in PISA and SAT etc highly g-loaded tests. And these high correlations among them have been well documented, researched and quantitatively established in HBD field. There are actually almost no surprises in those results from angle of the average IQs
-- e.g. it's almost impossible for Mexico somehow surpassed, say, Austria (or even North Korea to make my point) in maths/science scores no matter how many train loads of $$$ you pour into the former. I know that it’s not PC, but still. To respond one of your other curiosities here, so is with India to be brutally honest. It’s impossible for India to surpass the OECD average in PISA score ( IF it is a statistically VALID random sample of India) as India's mean IQ being 81 whereas OECD mean IQ being about 100. That's the base line supported by mountains of stats over decades. Don't believe it? Just wait and see the future scores.
That being said and argued, now Shanghai average IQ is about 112; HK /Singapore are about 108; Korea is about 107; Japan and Taiwan are about 105. See the general pattern here with the tests results? On top of these extra points on average IQ, Shanghai kids are more driven and hence likely worked harder as Just Todd said. Ask any older generation non-liberal & non-PC HongKonger or Chinese Singaporean, they would tell you, if they are perfectly honest, that if Shanghai were an independent entity after WWII, its GDP per cap could well be the top of the world by now. Empirically as an unspoken rule within Chinese communities, a group of Shanghainese would have more chance in winning over a group of Cantonese ( i.e. HKers, Singaporeans)on many areas due to being "smarter". I bet Singapore's current PM would also tell you such. On Ground Zero, ask yourself who is starting to challenge HK& Singapore's positions as the Asian Financial Centre? Yeah, Shanghai. And when Singapore wanted to invest in China, where in China it has set up high-tech industrial parks for R&D? Yeah, Suzhou ( the capital of JiangSu province which has basically the same people - Wu people - as nearby Shanghainese). Such examples are endless...Yet it's not your fault that you don't and can't see, simplely because you are not Chinese.
Therefore, the combination of above 2 factors ( aka IQ and study harder/more driven) could likely be enough to explain this test difference. Actually, your kind of "surprises" have mostly come from the western world. I, and most other Chinese (mainland + overseas) I believe, even couldn't imagine if Shanghai didn't score that high.
The second issue here is that "is Shanghai the high-end test score exception of China? ". NO, according to my knowledge and experiences in the field (and likely most other Chinese as well). GaoKao ( China's Annual College Entrance Exam) is the de facto and statistically valid yardstick to support my view, since Shanghai though being often among the top quartile , has been almost always surpassed by stronger provinces such as JiangSu, ZheJiang, FuJian, Sichuan,ShanDong, Beijing region, etc. Therefore, it wouldn't surprise me at all that some of these provinces ( if they also took PISA 2009) got even better Maths and/or Science test scores - to be published in the coming months.
Cheers!
O.K., I know I'm gonna come across as some kind of anti-Chinese fanatic, but experience is a hard teacher and it tells me that when something sounds too good to be true, it's probably Chinese-made. That's all.
Further analysis later in the blog (about a month later) seems to lend weight that the scores are legit. Time will tell -- what will the next test show?
Post a Comment