Quote of the Day | March 10, 2009: Random Data

From Bill James' 2005 SABR article, "Underestimating the Fog":

The first problem with comparison offshoots is that they have the combined instability of all of their components. Every statistic in baseball is to a certain degree a measurement of a skill, to a certain degree a statement about the circumstances, and to a certain degree simply a product of luck. A pitcher goes 20-8—he goes 20-8 to a certain degree because he is a good pitcher, to a certain degree because he pitches for a good team, and to a certain degree because he is lucky (or unlucky). There is luck in everything, and baseball fans are always engaged in a perpetual struggle to figure out what is real and what is just luck.

. . .

We ran astray because we have been assuming that random data is proof of nothingness, when in reality random data proves nothing. In essence, starting with Dick Cramer’s article, Cramer argued, “I did an analysis which should have identified clutch hitters, if clutch hitting exists. I got random data; therefore, clutch hitters don’t exist.”

Cramer was using random data as proof of nothingness—and I did the same, many times, and many other people also have done the same. But I’m saying now that’s not right; random data proves nothing—and it cannot be used as proof of nothingness.

Why? Because whenever you do a study, if your study completely fails, you will get random data. Therefore, when you get random data, all you may conclude is that your study has failed. Cramer’s study may have failed to identify clutch hitters because clutch hitters don’t exist—as he concluded—or it may have failed to identify clutch hitters because the method doesn’t work—as I now believe. We don’t know. All we can say is that the study has failed.

Tags: Bill James, baseball, Moneyball