Do you want a sample?
I'm a not a professional statistician, but that fact doesn't spare me from being repeatedly dumbstruck by invalid inferences made by people who should know better.
The most egregious errors are those stemming from bad population sampling. I've seen a lot of obliviousness when it comes to understanding the importance of random sampling and how misleading a biased sample can be.
Once I attended a presentation of a research study that concluded the majority of customers for a particular software company were corporate customers. This finding was completely contradictory to conventional wisdom within the company (technical support call demographics, representation on the Web, etc.).
When I asked about the evidence used to support the conclusion, the researcher revealed that the basis for the conclusion came from returned software registration cards. My next question asked if corporations might be more likely to register software than home users.
The researcher, chagrined, admitted the study assumed home users and corporate users were equally likely to register, but he had no basis for believing it, and it was a seriously flawed assumption.
Wikipedia notes a similar snafu in the history of statistical sampling:
"...[T]he 1936 Literary Digest prediction of a Republican win in the presidential election went badly awry, due to severe bias. A sample size of one million was obtained through magazine subscription lists and telephone directories. It was not appreciated that these lists were heavily biased towards Republicans and the resulting sample, though very large, was deeply flawed."
There's a right way and a wrong way to collect population sample data. The right way is to collect the data by randomly sampling the population. The wrong way is to simply let the data come to you. The data that come to you might not be representative of the general population.
If you only consider data coming to you, you might conclude the only sorts of people in the world are beggars and salesmen. Even if it's true in some profound sense, there are many other occupations.
The most egregious errors are those stemming from bad population sampling. I've seen a lot of obliviousness when it comes to understanding the importance of random sampling and how misleading a biased sample can be.
Once I attended a presentation of a research study that concluded the majority of customers for a particular software company were corporate customers. This finding was completely contradictory to conventional wisdom within the company (technical support call demographics, representation on the Web, etc.).
When I asked about the evidence used to support the conclusion, the researcher revealed that the basis for the conclusion came from returned software registration cards. My next question asked if corporations might be more likely to register software than home users.
The researcher, chagrined, admitted the study assumed home users and corporate users were equally likely to register, but he had no basis for believing it, and it was a seriously flawed assumption.
Wikipedia notes a similar snafu in the history of statistical sampling:
"...[T]he 1936 Literary Digest prediction of a Republican win in the presidential election went badly awry, due to severe bias. A sample size of one million was obtained through magazine subscription lists and telephone directories. It was not appreciated that these lists were heavily biased towards Republicans and the resulting sample, though very large, was deeply flawed."
There's a right way and a wrong way to collect population sample data. The right way is to collect the data by randomly sampling the population. The wrong way is to simply let the data come to you. The data that come to you might not be representative of the general population.
If you only consider data coming to you, you might conclude the only sorts of people in the world are beggars and salesmen. Even if it's true in some profound sense, there are many other occupations.
posted by metamerist at
10:43 AM
0 Comments:
Post a Comment
Subscribe to Post Comments [Atom]
<< Home