Wages, Simpson’s paradox and the complexity of statistical analysis

I spotted this good description of Simpson’s paradox in the trend data on wages over at the Inside-R blog today. Revolution Analytics did a good job explaining how Simpson’s paradox applies. At every education level median wages declined. However, for all people median wages have increased over the same time period. The reason for the difference is in the changes in the composition of the various groups over time.

The source article was at the New York Times. The New Statesman also gives another perspective explanation of what was going on utilizing the New York Times piece.

But where the New York Times creates some confusion was in saying “Low-Paid Workers Lose.” This implies that it is group of people who all lost out at an individual level. It is not until the very end of the piece that they clarify that they are not referring to the same people at both ends of the time period. But it is this point that is key to understanding what is happening. And it is at this point that the politicians take advantage of the statistics to make their points. They can say that workers at all education levels are losing out, and ignore the true situation.

Here what is likely happening is, particularly during the recession of the past few years, jobs that used to require only a high school diploma or some college now are occupied by those with a college degree. But the wages for the job have not changed to reflect the higher level of education of the job holder. As a result the median wages for those with college degrees has gone down.

The New York Times captured this phenomenon a couple of years ago in 2011 with an article titled “The Master’s as the New Bachelor’s.” They called it “Call it credential inflation.” With one person quoted as saying “Several years ago it became very clear to us that master’s education was moving very rapidly to become the entry degree in many professions.” That article goes on to claim without any real data that we are sending people to college who should not be there and do not need to be there for the jobs they end up with. That may be true, but we are also in a situation where many jobs require more skill than they used to. Many manufacturing jobs have moved from manual assembly of a product to operating the robotics that do the assembly. Both produce the same end product but the skill set required for the job has changed dramatically.

It is these kind of complexities that make dealing with number and statistics a challenging endeavor. It requires careful thought to understand what is the real story the statistics are telling us. It is all to easy to be led astray by looking at the numbers in haste or listing to someone tell a very smooth story.