Most of us have heard the phrase “There are three kinds of lies: lies, damn lies, and statistics” which was quoted by Mark Twain and perhaps erroneously attributed to Benjamin Disraeli. It seems that the one of the earliest occurrences of a similar remark was “fibs, lies, and statistics” by Sir Charles Dilke (at least according to Wikipedia, the Deep Thought of the 21st century). So, the notion that statistics are evil goes back at least 100 years. And yet we keep using them. Why? Because the actual world is very very hard to understand.
So, the notion that statistics are evil goes back at least 100 years. And yet we keep using them. Why? Because the actual world is very very hard to understand.
“The world is everything that is the case” begins the Tractatus Logico-Philosophicus, one of the most important philosophical works of the 20th century. The world is made up of an uncountable number of individual facts which, of course, our small brains can not keep track of. So, we summarize by various means, including statistics.
Every time we summarize, we lose precision. By analogy, image that you have a high resolution photograph and you reduce it by 50% on a photocopier. It looks fine, but if you take the reduced photo and enlarge it by 100% (putting it back to the same original size), you will not get back all of the detail — it’s just gone. In the same way, we lose detail whenever we apply statistics. This is okay if we’re measuring, say, golf balls on a driving range (to optimize where to run the golf-ball-picker-upper) — even though some golf balls will be left to rot on the fringes. This is not okay when we’re dealing with people.
Educational research is obsessed with statistics; in fact, it’s very difficult to get an educational research paper published if it doesn’t include some bullshit statistics in it. This is because educational researchers suffer from feelings of inadequacy, often with good reason. Most educational “treatments” are useless and pointless — otherwise, after over a hundred years, we would be a lot further along in understanding how to assist people in learning. If you pick a paper out of an educational journal from 50 years ago, brush it off, and change the pictures to look more “inclusive”, you could re-publish it today, perhaps to great acclaim. Try doing that in physics or medicine.
Most educational “treatments” are useless and pointless — otherwise, after over a hundred years, we would be a lot further along in understanding how to assist people in learning.
The first thing that an educational researcher is taught to do is to divide people into clumps. I know of papers that were rejected from publication because the author refused to segregate the participants by race and gender. Hey — here’s an idea — how about we treat everyone as an individual? No? Okay, sorry. I thought we were supposed to treat everyone the same.
What does it mean if “52.238% of students performed better on standardized test X after treatment Y”? What it means is that 47.762% didn’t. Educational research is stuck in Zeno’s Paradox: if you keep going half the distance to the door, you never get out of the room. If we keep treating half the educational group, we will never reach everyone.
If we remove statistics from educational research, what will we replace it with? The answer is individualized treatments. If we treat people as individuals, we have the opportunity to actually effect a change in their lives. There are only two things holding us back from this: (1) will and (2) effort.