Tuesday, December 14, 2010

OECD PISA test -- looking again at the Shanghai controversy

My posts skeptical of Shanghai scores on the PISA test has generated a bit of interest, including some input from Chinese commentators. A couple of hypotheses have been floated: IQ scores, the longer hours Chinese students spend on school work and studying, work ethic, impressive results by China at Mathematics Olympiads etc.. The general assertion is that the Shangahi scores are reasonable and not somehow overinflated as I asserted based on a statistical “Freakonomics” proposal given the large pool of students taking the test.

[update: at least an anonymous someone did post a lengthy comment regarding the Chinese scores, IQ, testing, etc but I can't find the comment now?! For the record I didn't delete it!]

Like any good critical thinker you have to be willing to revisit issues if new information comes up. Unfortunately I'm in Qatar and do not have access to the granular Shanghai PISA data so the best I can do is look around the Internet and see what the media has uncovered. Most of the articles that I looked at just accepted the results at face value without much digging into the underlying reasons for the good scores. There were also some veiled assertions of fixing the scores but without any elaboration, which is also of little value.

So here's some of the items I could find:

Let's start with support for my assertion. A blog post by a writer working for The Atlantic quotes a scientist who pretty much came up with the same rationales that I did, statistical anomalies etc. and suggested that Shanghai officials were “gaming the exam”:

http://www.theatlantic.com/national/archive/2010/12/on-those-stunning-shanghai-test-scores/67654/

Downside: this scientist is not named so I cannot figure out the source for the (rather lengthy) analysis. I tried looking on the Internet but could not find anything other than another article that was referencing The Atlantic article. The article seems comprehensive though so I gain some comfort that it is for real. However, the article is actually critical of almost any type of standardized testing to compare groups and points out that there are other significant differences between countries that seem unexplained so questions the overall validity.

Overall opinion I get from the article -- Shanghai scores are probably overinflated, despite that Shanghai students are still probably way better than (almost?) everyone at these things due do their educational system focusing on standardized testing, and everyone is overanalyzing the PISA results and needs to chill out.


Solidly against my assertion: the New York Times. They had a few articles about the PISA test but unlike many other newspapers did some level of work looking at what Shanghai did:

http://www.nytimes.com/2010/12/07/education/07education.html?ref=europe

Much of their work focused on interviews with education professionals who are familiar with Chinese system. Such as . . .

“Mark Schneider, a commissioner of the Department of Education’s research arm in the George W. Bush administration, who returned from an educational research visit to China on Friday, said he had been skeptical about some PISA results in the past. But Mr. Schneider said he considered the accuracy of these results to be unassailable.
“The technical side of this was well regulated, the sampling was O.K., and there was no evidence of cheating,” he said.

How about . . .

Secretary of Education Arne Duncan said in an interview on Monday.
“I know skeptics will want to argue with the results, but we consider them to be accurate and reliable, and we have to see them as a challenge to get better,” he added.
(though I'm not clear whether he was referring to the Shanghai scores or the US scores)

Also NYT asserts . . .

“The testing in Shanghai was carried out by an international contractor, working with Chinese authorities, and overseen by the Australian Council for Educational Research, a nonprofit testing group, said Andreas Schleicher, who directs the Organization for Economic Cooperation and Development’s international educational testing program. “


So it appears that there was some independence in administering the test, so controls may have been tighter than I had assumed.


The Christian Science Monitor also took a look at Shanghai and spoke to two American professors with some experience with the Chinese educational system. Both provided rationales for how the results could have happened, and discussed issues regarding the possible strengths with the current Chinese educational system, including attention spans, focus on education, and recent changes to education in Shanghai. Neither of them dismissed the results or alluded that they may have been overinflated:

http://www.csmonitor.com/World/Asia-Pacific/2010/1209/Shanghai-test-scores-have-everyone-asking-How-did-students-do-it

The Telegraph had an article that did not challenge the results but instead looked at some of the factors as to why Shanghai students would do so well. Long hours of class and studying with a focus on specific subjects that are tested appears to be their conclusion. They even quote a Chinese critic of the educational system that states that the singular focus on test preparation is actually a problem as it goes too far:

http://www.telegraph.co.uk/news/worldnews/asia/china/8187967/Shanghai-students-ranked-best-in-the-world-at-maths-and-science.html


So far the evidence is definitely stronger for the “Shanghai did not inflate the results” camp. I'll keep monitoring to see if anything else comes up in the media but it would need to be pretty strong to offset what has been reported in some of the articles. I would be interested in finding some articles from other East Asian countries to see what they have to say given that those countries also have very strong focuses on education.

5 comments:

Anonymous said...

Well Glen, I was that "Anonymous Chinese guy" who got my previous comment (I posted twice) lost as you mentioned. I don't have a copy of it thus let's leave it as such.

Regarding Shanghai score, you will have to have a good working knowledge on HBD/ IQ stuff before you could ever make a sense of 1SD defference without resorting to "cheating" etc abormal explainations.

Randomly choosing 30 or so studs among 150 radomly seleted Shangahi schools as they did means that any deliberate cheating, if it were true, would be extremely hard if not impossible in pratice and it must be on large scale ( think massive amount of ppl knowing this info), hence would be almost impossible without being exposed to wider media one way or the other so far. Therefore, if what OECD claims is ture ( aka no cheating is noticed) then one must come up with a reason to explain this 1SD difference. Agree?

It's not that hard at all from IQ point of view.

Go see this blog of Steve Hsu (a Harvard PhD in Physics) for the explaination on this matter. BTW, I happen to have a postgrad degree in Finance hence know a little about what I am talking about.

Anonymous said...

Hi, Glen,
Regarding the Shanghai results, if you are able to read Chinese, you can use Baidu (Chinese search engine) to search to PISA test and there were some high school campus news about the PISA tests and also some online comments by students who actually participated the test and you would be able to get a glimpse of how test was conducted. It appears to me the test was conducted in 12 provinces apart from Shanghai and the school authorities were given a short notice (3-4 weeks before test) to collect the names of the students with birth dates within a certain window and then the students were randomly selected. There seemed to be some reluctance from student’s side to take the test as it interfered their preparations for the senior high school entrance exam which is very competitive. The students who took PISA tests found the PISA test was quite fun but not much helpful for their entrance exam preparation. Nevertheless, the students were told to finish the PISA exam for research purpose. I notice some of vocational schools in shanghai also participated in the PISA tests. In China, usually kids who finish poorly after compulsory education and who have little prospect of getting into college or kids from rural families would end up in vocational schools. So I would think sampling was okay and there was not much incentive for Chinese authorities to manipulate the PISA. In fact, after PISA results were published, there was quite limited media coverage in China and general attitude towards the results was that this is not surprising as Chinese kids are good at exams anyway (and are only good at exams).
The interesting part is the results of other provinces which will be published next year. Total 20,000 students from other 12 provinces participated in the PISA tests, including coastal areas, central China as well as extremely poor regions like Ningxia autonomous region. Shanghai scores might not be the highest as Haidian district of Beijing (with many nation-known elite schools) also participated and it usually score highest in any standard test in China. The result from Ningxia would probably be the lowest (as it usually scores among the lowest in college entrance exams and some kids there still suffer from malnutrition problem). Therefore, these results will give a floor and ceiling of the PISA performance of Chinese students. However, Andreas Schleicher told FT that even in very poor areas, the performance is close to OECD average.

So in the end, the question really is, how much should we read into these results and does the PISA test fairly measure the educational achievements given different cultural context?

Unknown said...

Hi, Glen,
Regarding the Shanghai results, if you are able to read Chinese, you can use Baidu (Chinese search engine) to search to PISA test and there were some high school campus news about the PISA tests and also some online comments by students who actually participated the test and you would be able to get a glimpse of how test was conducted. It appears to me the test was conducted in 12 provinces apart from Shanghai and the school authorities were given a short notice (3-4 weeks before test) to collect the names of the students with birth dates within a certain window and then the students were randomly selected. There seemed to be some reluctance from student’s side to take the test as it interfered their preparations for the senior high school entrance exam which is very competitive. The students who took PISA tests found the PISA test was quite fun but not much helpful for their entrance exam preparation. Nevertheless, the students were told to finish the PISA exam for research purpose. I notice some of vocational schools in shanghai also participated in the PISA tests. In China, usually kids who finish poorly after compulsory education and who have little prospect of getting into college or kids from rural families would end up in vocational schools. So I would think sampling was okay and there was not much incentive for Chinese authorities to manipulate the PISA. In fact, after PISA results were published, there was quite limited media coverage in China and general attitude towards the results was that this is not surprising as Chinese kids are good at exams anyway (and are only good at exams).
The interesting part is the results of other provinces which will be published next year. Total 20,000 students from other 12 provinces participated in the PISA tests, including coastal areas, central China as well as extremely poor regions like Ningxia autonomous region. Shanghai scores might not be the highest as Haidian district of Beijing (with many nation-known elite schools) also participated and it usually score highest in any standard test in China. The result from Ningxia would probably be the lowest (as it usually scores among the lowest in college entrance exams and some kids there still suffer from malnutrition problem). Therefore, these results will give a floor and ceiling of the PISA performance of Chinese students. However, Andreas Schleicher told FT that even in very poor areas, the performance is close to OECD average.

So in the end, the question really is, how much should we read into these results and does the PISA test fairly measure the educational achievements given different cultural context?

Glen McKay said...

If you would like I can repost your original comment. Whenever someone sends a comment to the blog I receive an e-mail with its contents so I actually have your comment -- just for some reason it's gone from the blog itself.

Anonymous said...

Give you another article about Chinese PISA test result:
http://www.bbc.co.uk/news/business-17585201