Qatar and the PISA test -- further analysis

Back when the PISA 2009 results came out one commenter pointed out that while Qatar's marks were still low they had made significant improvements from 2006, which indicates that the reform measures taken by the Supreme Council of Education are making an impact. I'll reserve judgment on that until we analyze marks from a number of countries. The test changes from year to year so if countries at the bottom improved then it could indicate that the test was easier than the previous one. If all the countries improved than the test was definitely easier.

Unfortunately not a lot of countries in the bottom range had taken the test in 2000 or 2003, and some aspects of the test changed over the period, so I will only take the 2006 and 2009 results:

(Sorry about the mess, the table formatting didn't carry forward, the last number in the line is the change from 2006 to 2009)

Reading: 2006 2009 difference

Canada 527 524 -3
Japan 498 520 22
UK 495 494 -1
USA na 500 na
OECD avg 492 493 1
Portugal 472 489 17
Mexico 410 425 15
Tunisia 380 404 24
Uruguay 413 426 13
Qatar 312 372 60
Kyrgystan 285 314 29

Math: 2006 2009 difference

Canada 527 527 0
Japan 523 529 6
UK 495 492 -3
USA 474 487 13
OECD avg 499 496 -3
Portugal 466 487 21
Mexico 406 419 13
Tunisia 365 371 6
Uruguay 427 427 0
Qatar 318 368 50
Kyrgystan 311 331 20

Science: 2006 2009 difference

Canada 534 529 -5
Japan 531 539 8
UK 515 514 -1
USA 489 502 13
OECD avg 500 501 1
Portugal 474 493 19
Mexico 410 416 6
Tunisia 386 401 15
Uruguay 428 427 -1
Qatar 349 379 30
Kyrgystan 322 330 8

The results definitely support the view of the commentator, Qatar had easily the best improvements of the countries selected, in all three categories. Even countries near Qatar in terms of 2006 results had mixed improvements, although all of the bottom six improved to some degree in almost all categories.

Qatar still has a long way to go though. Scores below 400 are pretty bad, and as I noted in my previous post Qatar had a number of good performing schools which means there were a lot of schools performing even below the low-300s.

If you're not sure what exactly a score means I recommend you take a couple of the sample questions, which can be found here:

http://www.oecd.org/document/31/0,3746,en_32252351_32236191_41942687_1_1_1_1,00.html

Science question 12.2 was pretty enlightening, and one of the worst results for Qatar. (Summary the question: in an experiment on corn scientists chose 200 different fields to have the experiment in, why did they use so many fields?). 74% of 15-year-olds in the OECD got the question right, in Qatar it was 30%. Yet the question was multiple-choice with only four different answers, so even by random guessing you'd expect 25% to get it right. Yikes.