I was amused to come across the toupée fallacy recently. According to the RationalWiki page on it, the idea is that it’s (informally) fallacious to assert that “all toupées look fake” if I am basing that conclusion solely on the “evidence” that, thus far, I have only ever seen fake-looking toupées. The problem should be clear – I haven’t noticed any non-fake-looking toupées because they’re not fake-looking, so I thought they were just hair!
The toupée fallacy seems to me to be the fallacy of hasty generalization, and it’s not clear why hasty generalization isn’t referenced on the RationalWiki page. Perhaps the folks over there are hoping the snazzier “toupée fallacy” moniker will catch on, but they’re up against some stiff competition already. The (Irrational?) Wikipedia article lists a few, none of which refer to hairpieces:
The fallacy is also known as the fallacy of insufficient statistics, fallacy of insufficient sample, generalization from the particular, leaping to a conclusion, hasty induction, law of small numbers, unrepresentative sample, and secundum quid. When referring to a generalization made from a single example it has been called the fallacy of the lonely fact or the proof by example fallacy.
I am partial to “the fallacy of the lonely fact.”
As noted in both wiki articles, hasty generalization is part of the more general problem of induction. I’ve written about induction before, at which time I brought up the distinction between the plebian and aristocratic problems of induction. The plebian problem is stated (by Larry Laudan) as:
Given a universal empirical generalization and a certain number of positive instances of it, to what degree do the latter constitute evidence for the warranted assertion of the former?
And the aristocratic problem is stated as:
Given a theory, and a certain number of confirming instances of it, to what degree do the latter constitute evidence for the warranted assertion of the former?
As I noted in my previous post, Laudan states (in a footnote) that “a ‘theory’ in this sense must postulate one or more unobservable entities, i.e., statements which could arise as empirical generalizations do not count as theories for these purposes.” He illustrates the aristocratic problem of induction with a discussion of whether or not pressure-volume relations in gasses count as evidence for a particle-based theory of gasses. The aristocratic problem is that, even if we observe a pressure-volume relationship that is predicted by our particle-based theory, because other theories might make the same prediction, too, it doesn’t provide use with evidence for our theory. Jaynes describes it as “plausible reasoning,” but strictly speaking, the premises “If P, then Q” and “Q” license nothing more than “Maybe P, maybe not P.”
Both of the wiki articles about hasty generalization are concerned only with plebian induction, and neither consider probabilistic relationships.
Let be the hypothesis “all toupées look fake.” Considering , we’re faced with the classic, non-probabilistic plebian induction problem. We can’t ever completely confirm unless we can observe every toupée. And all it takes is one non-fake toupée to disconfirm . Any generalization from positive instances is hasty, and any observation of a non-fake-looking toupée falsifies .
Let be “we expect k out of any N toupées to be fake-looking.” Again, we could (dis)confirm by observing all toupées, but that’s not a realistic possibility. So, suppose we test by looking at a sample of toupées, in which m are fake-looking.
Under many – maybe most – of the statistical probes of , no observation provides an unambiguous answer. Observing m fake-looking toupées will be more or less consistent with our hypothesis, and we will need some extra machinery to make any decisions with respect to .
Of course, we could use good ol’ statistical hypothesis testing, in which case we can maybe get an unambiguous answer that depends, in part, on the observed value of m, the hypothesized value of k, the size of our sample, and any number of choices regarding statistical test procedures (e.g., the level, the test statistic, etc…). And to the extent that we do get an unambiguous answer, it will be “reject ,” and it will be based on the fact that out test statistic exceeded some more or less arbitrary criterion.
But and aren’t theories. Suppose we develop a theory that posits unobserved entities, and suppose further that this theory implies that, among other things, (some proportion of) toupées are fake-looking. The aristocratic problem tells that no number of observed fake-looking toupées licenses the conclusion that our theory is true.
As discussed at length by Laudan, it’s this kind of (aristocratic) problem with induction that led to the dominance of hypothetico-deductive methods in science. I’ve been reading some papers recently (e.g., Gelman & Shalizi; Box’s chapter in this book) that present a sort of cyclic picture of science, with induction (or induction-like) reasoning (e.g., statistical model building) alternating with hypothetico-deductive reasoning (e.g., statistical testing, model fit evaluation).
This picture seems more or less correct to me, in the sense that it accurately describes the kind of things (social?) scientists do, at least some of the time (I’ll leave open the question of whether or not this is what scientists should do). But to the extent that the model-building phase in this picture of (statistically-oriented) science is based on induction from observation, it doesn’t make direct contact with any theory (in the unobserved-entity sense of theory used above).
I think cognitive science writ large can still benefit substantially from more or less atheoretical inductive model building. But one of the most difficult – and important – parts of conducting good science is figuring out exactly how theory makes contact with observation. Theories of, e.g., speech perception pretty much never make predictions about empirical phenomena as directly and unambiguously as particle-based theories of gas do about pressure-volume relationships, but it seems exceedingly unlikely to me that we’ll ever have anything like a complete understanding of speech perception without detailed, well-tested, unobserved-entity-positing theories.