Lies, damn lies and UNICEF reports

Britons have been indulging in a bout of self-flagellation over our bottom-ranking in a recent UNICEF report on childhood well-being. Each person, of course, chooses to blame the result on their personal -bête noire-. No doubt there are many things wrong with British society and many complex causes, but before rushing to our knee-jerk reactions, no one seems to have bothered to consider how much water this report holds. It's from the international body representing children, so it must be impartial and accurate, right?

Well, not exactly....

1. This is supposed to assess "children and young people in the world’s advanced economies", and "shows that among all of the 21 OECD countries there are improvements to be made" (both quotes from UNICEF's press release accompanying the report). "The world's advanced economies"? "All of the 21 OECD countries"? There are 30 OECD countries. Where are Australia, Iceland, Japan, Luxembourg, Mexico, New Zealand, the Slovak Republic, South Korea, and Turkey? And what about the developed, non-OECD countries, like Singapore and Israel?

The authors of the press release must have been gratified by the dependable geographical ignorance of most of the British media. Not one report that I have seen has noticed that these countries were missing. Instead, they all found their own phrases to repeat the implication that this partial list was a comprehensive list of the rich/industrialised nations, thus making our shame complete:

• "the worst place in the industrialised world to grow up" - The Times

• "the worst country in the industrialised world in which to grow up" - The Telegraph

• The UNICEF report "placed the UK last in the survey of 21 nations, which included Europe as well as the United States, Canada and Japan" - The Independent (Japan was not one of the 21).

• "Children growing up in the United Kingdom suffer greater deprivation, worse relationships with their parents and are exposed to more risks from alcohol, drugs and unsafe sex than those in any other wealthy country in the world" - The Guardian (trust The Guardian to most overstate the case)

Even if they didn't notice the omissions, you would have thought the inclusion of Hungary and the Czech Republic would have given them pause for thought before using phrases such as "the industrialised world".

But perhaps it's excusable that they didn't notice. After all, the report's 52 pages barely mention these omissions. Only in a note in small print below the results table on page 2 is the omission of these countries from the overall results mentioned. They are listed as "OECD countries with insufficient data to be included in the overview". Well, the United States is included in the overview, even though insufficient data was available to give it a score in one of the six categories ("Subjective well-being"). Several other countries (Italy, Denmark, Canada, Ireland, Norway and France) in the overview were missing some aspects of the data used to compile the scores. And on some aspects on which the report claims not to have adequate data for the missing countries, such as the prevalence of single-parent families, that data is available with only a brief web-search. Perhaps not with identical dates or criteria, but close enough that it would have been significantly more helpful to an accurate comparison than simply ignoring the countries.

As the authors say:

"It is acknowledged throughout that the available data may be less than ideal and that there are prominent gaps. Children’s exposure to violence in the home both as victims and as witnesses, for example, could not be included because of problems of cross-national definition and measurement. Children’s mental health and emotional well-being may also be under-represented, though attempts have been made to reflect these difficult-to-measure dimensions. Age and gender differences are also insufficiently attended to, again reflecting a lack of disaggregated data and the fact that the majority of the available statistics relate to the lives of older children. A particularly important omission is the level of participation by three and four year-olds in early childhood education (for which, again, no internationally comparable data are available)."

2. Many, perhaps most, of the aspects assessed are of dubious utility for the assessment of childhood welfare. Finland scores comfortably the lowest of the 21 for the "Percentage of students age 11, 13 and 15 who report 'liking school a lot'". But they score comfortably the highest for the "Educational achievement of 15 year-olds, an overview of reading, mathematical and scientific literacy". These results are by no means incompatible, but how are we to interpret them, if treated as bare numbers to be added to the pot, as they are in this report? Perhaps the Finns achieve their great results through excessive discipline, which causes the children to be unhappy. Or perhaps there is greater emphasis on the basic skills assessed than on other subjects, compared to other countries, and it is the lack of breadth in the education which causes the dissatisfaction. Or perhaps Finns are more rebellious (by nature, culture or upbringing) than other races, providing the enquiring mind that produces good academic results, but the resentment of authority that leads to dislike of regimented education. Or perhaps the Finns just tend to exaggerate (both in marking student's papers and in estimating their emotional state). One could imagine many other generalisations, and the truth is probably that there is an aspect of several, and no valid overall generalisation. But what does this tell us about their well-being as children? Almost nothing.

3. The Poles have the lowest "percentage of young people age 13 and 15 who report being overweight". I'm sorry. I've been to Poland, met the people, and eaten the food. It is delicious, but in no way is it healthy. Amongst a group of more-than-adequately-girthed rugby-players, who were by no means averse to a decent portion of meat and fat, there was general agreement that it was surprising anyone could last more than a fortnight in the country without suffering complete congestion of the arteries. It is highly unlikely that young Poles are the thinnest in Europe, let alone in the industrialised world. So what do we learn from the fact that very few of them "report being overweight"?

The way this is phrased is key. The children were not measured, they were asked to provide their weights and heights. This question, apparently, "yielded low response rates, possibly indicating that the figures are underestimates". It's an interesting assumption that a small sample would have biased the results in one direction rather than the other. And embedded in the use of these figures is another assumption, that however much the results might be distorted in one direction or another, they were equally distorted in the same direction for all countries. Without this assumption, one could not responsibly use the figures to draw up a ranking. And yet a little reflection leads one to question whether that is a reliable assumption (that all races and cultures will be equally happy to be candid about their physiques), a doubt that is only reinforced by Poland's improbable presence at the top of this particular table. Is it completely improbable that the Poles are, for some reason, more embarassed to reveal their vital statistics if they differ from the hypothetical ideal, than are children of other nations?

4. Relative (not absolute) poverty was used to calculate the material well-being of children. This biases the results in favour of strongly redistributive economies and against more liberal economies. The justification for using relative poverty as a measure is that those below the median income may feel deprived, regardless of their absolute income. But this bears on the psychological well-being, not the material well-being of those from poorer backgrounds. A family on 50% of the median income in a country where the median income is $50,000 may be less satisfied with their lot than a family on 75% of the median income in a country where the median income is $30,000, but the former family is nevertheless materially better off than the latter family.

The same sort of argument can be made for the other components. Using the percentage of households without jobs assumes that joblessness has an equal impact on material well-being in all countries. Clearly, given different models of social protection, this is not the case. And using "reported deprivation" assumes that standards of deprivation are equal across all countries, or that sense of deprivation is somehow more important than actual deprivation to material well-being. That may be true for psychological well-being, but for material well-being, it is, by definition, untrue.

No wonder the countries with the most redistributive models did best in this category. They might as well have measured degree of redistribution in the tax and benefits system directly, for all the difference it would have made compared to the way they tried to dress it up.

5. The metric used for children's safety is "Deaths from accidents and injuries per 100,000 under 19 years". The report notes that four countries, including the UK, have made particularly strong progress in this regard, reducing deaths by 50% to less than 10 per 100,000. Whilst every life saved is precious, there has been a price to pay. Whilst some progress has doubtless been made in worthwhile areas, like reducing traffic speed in built-up areas, there can be equally little doubt that another contributing factor has been the removal of most opportunities for carrying out more exciting but risky activities. Thanks to liability laws, health-and-safety prosecutions, and time-pressures, the opportunities for children to take part in activities and adventures that carry even the smallest risk of harm have all but evaporated in Britain. This mollycoddling and cotton-wool-wrapping may be reducing the total number of deaths from a very small proportion to a very, very small proportion, but it is at the price of boredom and frustration for many children. Children need opportunities to challenge and look after themselves, and to work together against the odds, or they will grow up to be very much less healthy and self-sufficient than previous generations. This is very much a double-edged sword as a test of children's welfare, with the sharper edge being the one concealed.

6. The test of educational well-being again reveals the authors' prejudices. One aspect considered is the "percentage aged 15-19 remaining in education". The assumption is that this is automatically a good thing. That is only the case if one assumes that all children are equally well-suited and inclined to remain in education upto the age of 19. It also relies on a consistent definition of what constitutes education. Some countries may include vocational training, while others may not (the line may be blurred). If those countries lower down this league are releasing those children who are not academically-gifted to start doing something practical and earning money instead, are they being more or less considerate to those who in school might well be killing time?

Fortunately, the following aspect of educational well-being allows us to measure the extent to which this is a factor. This provides the "Percentage of 15-19 year-olds not in education, training or employment". In France and Belgium, these percentages combined with the percentages for the previous aspect come to 100% - in other words, if you are not in education, you are not doing anything. In Portugal, the UK and the USA, the combined percentages for the two aspects are significantly below 100%, indicating that many children of that age who are considered to be outside the educational system are either in training or employment. Particularly when we consider 18- and 19-year-olds, is that necessarily a bad thing? Are we to conclude, as the authors do, that the well-being of Belgian children is greater than that of Portuguese children because many more of them are obliged to stay in some form of education until at least the age of 19?

Many more similar points could be made. But in short, some things that matter could not be measured, some things were inadequately measured, some things were subjective and liable to competing and often contrasting interpretation, some things were fudged, whilst in other cases standards were applied rigidly where a little flexibility would have yielded a broader comparison. That is the nature of trying to compare one country with another. It does not invalidate the report, but it does mean (as doubtless the authors intended) that the results should be treated with extreme caution, rather than the hysteria and appropriation to every cause that has greeted them in the UK.

When the report is weighed with that necessary caution, the unavoidable conclusion is that it measures not well-being, but compliance with a particularly European, egalitarian, social-democratic economic and social model. Which makes it rather less surprising that those countries whose models most closely comply with the authors' preferences are near the top of the league, and those countries whose models contrast most strongly with those preferences are near the bottom. The authors have successfully proved that A is A.