OECD PISA test -- looking again at the Shanghai controversy

My posts skeptical of Shanghai scores on the PISA test has generated a bit of interest, including some input from Chinese commentators. A couple of hypotheses have been floated: IQ scores, the longer hours Chinese students spend on school work and studying, work ethic, impressive results by China at Mathematics Olympiads etc.. The general assertion is that the Shangahi scores are reasonable and not somehow overinflated as I asserted based on a statistical “Freakonomics” proposal given the large pool of students taking the test.

[update: at least an anonymous someone did post a lengthy comment regarding the Chinese scores, IQ, testing, etc but I can't find the comment now?! For the record I didn't delete it!]

Like any good critical thinker you have to be willing to revisit issues if new information comes up. Unfortunately I'm in Qatar and do not have access to the granular Shanghai PISA data so the best I can do is look around the Internet and see what the media has uncovered. Most of the articles that I looked at just accepted the results at face value without much digging into the underlying reasons for the good scores. There were also some veiled assertions of fixing the scores but without any elaboration, which is also of little value.

So here's some of the items I could find:

Let's start with support for my assertion. A blog post by a writer working for The Atlantic quotes a scientist who pretty much came up with the same rationales that I did, statistical anomalies etc. and suggested that Shanghai officials were “gaming the exam”:

http://www.theatlantic.com/national/archive/2010/12/on-those-stunning-shanghai-test-scores/67654/

Downside: this scientist is not named so I cannot figure out the source for the (rather lengthy) analysis. I tried looking on the Internet but could not find anything other than another article that was referencing The Atlantic article. The article seems comprehensive though so I gain some comfort that it is for real. However, the article is actually critical of almost any type of standardized testing to compare groups and points out that there are other significant differences between countries that seem unexplained so questions the overall validity.

Overall opinion I get from the article -- Shanghai scores are probably overinflated, despite that Shanghai students are still probably way better than (almost?) everyone at these things due do their educational system focusing on standardized testing, and everyone is overanalyzing the PISA results and needs to chill out.


Solidly against my assertion: the New York Times. They had a few articles about the PISA test but unlike many other newspapers did some level of work looking at what Shanghai did:

http://www.nytimes.com/2010/12/07/education/07education.html?ref=europe

Much of their work focused on interviews with education professionals who are familiar with Chinese system. Such as . . .

“Mark Schneider, a commissioner of the Department of Education’s research arm in the George W. Bush administration, who returned from an educational research visit to China on Friday, said he had been skeptical about some PISA results in the past. But Mr. Schneider said he considered the accuracy of these results to be unassailable.
“The technical side of this was well regulated, the sampling was O.K., and there was no evidence of cheating,” he said.

How about . . .

Secretary of Education Arne Duncan said in an interview on Monday.
“I know skeptics will want to argue with the results, but we consider them to be accurate and reliable, and we have to see them as a challenge to get better,” he added.
(though I'm not clear whether he was referring to the Shanghai scores or the US scores)

Also NYT asserts . . .

“The testing in Shanghai was carried out by an international contractor, working with Chinese authorities, and overseen by the Australian Council for Educational Research, a nonprofit testing group, said Andreas Schleicher, who directs the Organization for Economic Cooperation and Development’s international educational testing program. “


So it appears that there was some independence in administering the test, so controls may have been tighter than I had assumed.


The Christian Science Monitor also took a look at Shanghai and spoke to two American professors with some experience with the Chinese educational system. Both provided rationales for how the results could have happened, and discussed issues regarding the possible strengths with the current Chinese educational system, including attention spans, focus on education, and recent changes to education in Shanghai. Neither of them dismissed the results or alluded that they may have been overinflated:

http://www.csmonitor.com/World/Asia-Pacific/2010/1209/Shanghai-test-scores-have-everyone-asking-How-did-students-do-it

The Telegraph had an article that did not challenge the results but instead looked at some of the factors as to why Shanghai students would do so well. Long hours of class and studying with a focus on specific subjects that are tested appears to be their conclusion. They even quote a Chinese critic of the educational system that states that the singular focus on test preparation is actually a problem as it goes too far:

http://www.telegraph.co.uk/news/worldnews/asia/china/8187967/Shanghai-students-ranked-best-in-the-world-at-maths-and-science.html


So far the evidence is definitely stronger for the “Shanghai did not inflate the results” camp. I'll keep monitoring to see if anything else comes up in the media but it would need to be pretty strong to offset what has been reported in some of the articles. I would be interested in finding some articles from other East Asian countries to see what they have to say given that those countries also have very strong focuses on education.