Resuscitation of the Demarcation Problem?

An edited volume on (and called) The Philosophy of Pseudoscience (TPoP) came out last year. I would be hard pressed to think of a topic better suited to get me to pony up a few bucks and then spend time reading and thinking about something not directly related to my work.

The basic idea motivating the book is that Larry Laudan was wrong (or at least premature) in announcing The Demise of the Demarcation Problem. The Demarcation Problem is, in case you don’t know, the problem of differentiating between science and non-science or between science and pseudoscience (and maybe also between non-science and pseudoscience). As it happens, I’ve blogged about this Laudan essay before, though in a way that isn’t directly addressed at this new book, so in this post, I’ll review the basics of Laudan’s argument. I’ll follow up in later posts with reviews of the first few chapters of TPoP.

Spoiler alert: I remain unconvinced by the arguments in TPoP that Laudan is wrong. In order to see why (I think) they’re wrong, it will be useful to make reference to Laudan’s original essay. I’ve got it as a hard copy of a book chapter, and I can’t find it in freely-available digital format (here it is behind a rather pricey paywall, and here is most, but not all, of it on google books), so I will cover the main points made in a highly abbreviated format (and organized using a subset of Laudan’s section headers).

The Old Demarcationist Tradition

Ancient concerns with knowledge/reality vs opinion/appearance lead to Aristotle arguing that scientific knowledge is certain, deals in causes, and follows logically from first principles, which are themselves derived directly from sensory input. Thus, Laudan writes that, according to Aristotle, “science is distinguished from opinion and superstition by the certainty of its principles; it is marked off from the crafts by its comprehension of first causes.” (p. 212)

In the 17th century, the latter criterion fell out of favor. Many of the people we think of as the founding fathers of modern science (e.g., Galileo, Newton) explicitly repudiated the idea that a science necessarily addresses causes. By the 19th century, certainty was discarded, as well: “the unambiguous implication of fallibilism is that there is no difference between knowledge and opinion: within a fallibilist framework, scientific belief turns out to be just a species of the genus opinion.” (p. 213)

With certainty and causation no longer able to demarcate science from non-science, folks turned methodology to (try to) do the job. In order for methodology to do the job, philosophers had to establish that there is a single, unified scientific method and that this method is epistemically better than other, non-scientific methods. Attempts to establish the unity and epistemic superiority of method led to disagreement about what the one, true method is and ambiguity or outright falsity with respect to whether or not practicing scientists actually employ any particular proposed method.

A Metaphilosophical Interlude

Laudan says that we should ask three questions (quoted verbatim from the essay):

  1. What conditions of adequacy should a proposed demarcation criterion satisfy?
  2. Is the criterion under consideration offering necessary or sufficient conditions, or both, for scientific status?
  3. What actions or judgments are implied by the claim that a certain belief or activity is ‘scientific’ or ‘nonscientific’?

With respect to #1, Laudan argues that a demarcation criterion should (a) accord with common usage of the label ‘science’ – it should capture the paradigmatic cases of science and non-science, regardless of how it deals with more difficult, borderline cases; (b) identify the epistemic and/or methodological properties that science has and that non-science does not; and (c) be precise enough so that we can, in fact, use it to demarcate science and non-science.

With respect to #2, Laudan argues that a demarcation criterion must provide both necessary and sufficient conditions. Necessary conditions alone will only allow us to say if something isn’t science, but do not allow us to say if something is, while sufficient conditions alone only allow us to say if something is science, but do not allow us to determine what is not scientific.

With respect to #3, Laudan points out that, because it will have numerous social, political, and, in general, practical implications, “any demarcation criterion we intend to take seriously should be especially compelling.”

The New Demarcationist Tradition

More recent efforts to develop demarcation criteria have focused on what Laudan calls potential epistemic scrutability rather than actual epistemic warrant. Verificationist, falsificationist, and various approaches based on testability, well-testedness, the production of surprising predictions, and so forth, all serve very poorly as demarcation criteria for various reasons, including their frequent failure to serve as both necessary and sufficient conditions for demarcation. Plenty of obviously scientific claims aren’t verifiable, and plenty of obviously non-scientific claims are; plenty of obviously scientific claims aren’t falsifiable, and plenty of obviously non-scientific claims are. And so on.

It’s worth quoting from Laudan’s conclusion at length (emphasis in the original):

Some scientific theories are well-tested; some are not. Some branches of science are presently showing high rates of growth; others are not. Some scientific theories have made a host of successful predictions of surprisingly phenomena; some have made few if any such predictions. Some scientific hypotheses are ad hoc; others are not. Some have achieved a ‘consilience of inductions’; others have not. (Similar remarks could be made about several nonscientific theories and disciplines.) The evident epistemic heterogeneity of the activities and beliefs customarily regarded as scientific should alert us to the probable futility of seeking an epistemic version of a demarcation criterion. Where, even after detailed analysis, there appear to be no epistemic invariants, one is well advised not to take their existence for granted. But to say as much is in effect to say that the problem of demarcation… is spurious, for that problem presupposes the existence of just such invariants.

Laudan ends the essay with a brief discussion of the sorts of things that philosophy of science should be focused on, given his dismissal of the demarcation problem. The last couple pages are available in google books, but I’ll quote him here again:

It remains as important as it ever was to ask questions like: When is a claim well confirmed? When can we regard a theory as well tested? What characterizes cognitive progress?

He is, at least in part, making a semantic argument that the class of things appropriately labeled ‘science’ (and its counterparts in the classes of things labeled ‘non-science’ and ‘pseudoscience’) just isn’t particularly (philosophically) interesting. One last quote, from the concluding paragraph:

…we have managed to conflate two quite distinct questions: What makes a belief well founded (or heuristically fertile)? And what makes a belief scientific? The first set of questions is philosophically interesting and possibly even tractable; the second question is both uninteresting and, judging by its checkered past, intractable.

It’s interesting to note, given his conclusions here, that Laudan has more recently been focusing on legal epistemology, which is to say that he’s been pursuing his ‘first set of questions’ in a non-scientific area. It was also interesting to note that the essay immediately following the Demise essay in the book I have (a critique of a judicial decision about teaching creationism in Arkansas schools) kind of foreshadows this move. But I digress.

Next up: summaries and discussion of the first few essays in TPoP.

Posted in philosophy of science | Comments Off

More on (moron!) bad pop science

Blogging clearly isn’t my priority lately (I have some great posts planned, though, I swear!), but nothing beats dumb pop science writing for generating a quick post. I’m sure I’ve missed some since my last post on the topic a month ago, but I just saw a new example (via the science subreddit) that I can’t resist responding to.

From the pop science article:

Although more research is necessary, the results suggest that spirituality or religion may protect against major depression by thickening the brain cortex and counteracting the cortical thinning that would normally occur with major depression.

And from the Results and Conclusions and Relevance sections of the publicly available journal page for the publication:

… We note that these findings are correlational and therefore do not prove a causal association between [religious/spiritual] importance and cortical thickness….

A thicker cortex associated with a high importance of religion or spirituality may confer resilience to the development of depressive illness in individuals at high familial risk for major depression, possibly by expanding a cortical reserve that counters to some extent the vulnerability that cortical thinning poses for developing familial depressive illness.

It’s a subtle issue, I know, but there is a logical distinction between, on the one hand, spirituality protecting against depression by making brains thick and, on the other, thick brains protecting against depression and simultaneously being statistically associated with (self-reported importance of) spirituality.

Posted in SCIENCE! | Comments Off

On bad pop science

I just love this kind of writing about abstruse, abstract physics for a lay audience:

A team of physicists has provided some of the clearest evidence yet that our Universe could be just one big projection.

In 1997, theoretical physicist Juan Maldacena proposed that an audacious model of the Universe in which gravity arises from infinitesimally thin, vibrating strings could be reinterpreted in terms of well-established physics. The mathematically intricate world of strings, which exist in nine dimensions of space plus one of time, would be merely a hologram: the real action would play out in a simpler, flatter cosmos where there is no gravity.

[snip]

In two papers posted on the arXiv repository, Yoshifumi Hyakutake of Ibaraki University in Japan and his colleagues now provide, if not an actual proof, at least compelling evidence that Maldacena’s conjecture is true.

[snip]

Neither of the model universes explored by the Japanese team resembles our own, Maldacena notes.

Excellent.

Posted in SCIENCE! | Comments Off

A quick pfisking

The first link in this week’s “This week in stats” (by Matt Asher) post leads to a fairly silly rant (by a Wesley) about p-values. I feel like it deserves a quick (but partial, because I don’t disagree with everythingfisking, in addition to reiterating the point made by Mr. Asher that, whatever problems p-values have, no solutions are on offer here (though I know at least a dozen or so people who would argue against his claim that no one has come up with a satisfactory substitute to p-values). Anyway:

Wesley: P-values … can also be used as a data reduction tool but ultimately it reduces the world into a binary system: yes/no, accept/reject.

Noah: Given that p-values are but one part of a statistical analysis in the frequentist hypothesis testing tradition, I have a hard time seeing why this is so problematic. A calculated test statistic either exceeds a criterion or it doesn’t. This doesn’t tell the whole story of an data set, but it’s not meant to.

W: Below is a simple graph that shows how p-values don’t tell the whole story.  Sometimes, data is reduced so much that solid decisions are difficult to make. The graph on the left shows a situation where there are identical p-values but very different effects.

N: I don’t understand what Wesley means when he links data reduction and decision-making difficulty, so I’ll leave that one alone. I’ll also not go into depth about why I think these graphs kind of stink (to mention maybe the worst thing about the graphs: they’re mostly just white space, with the actual numbers of interest huddled up against the [unnecessary] box outline).

Anyway, it’s not at all clear how the two “effects” in the left panel could be producing the same p-value (and the code from the post isn’t working when I try to run it – the variable effect.match is empty, since the simulation with the minimum CI difference isn’t in the set of p-values that match, i.e., logical indexing fails to produce a usable index – so I can’t reproduce the plot). Contra my intuition when first seeing the graph, it is not illustrating a paired t-test, but, rather, two single-sample t-tests. I gather that each of these red dots is illustrating a mean, and each vertical line is illustrating an associated confidence interval, and that the means are being compared to zero. Given that one (CI) line covers zero and the other does not, the p-values shouldn’t be the same.

W: The graph on the right shows where the p-values are very different, and one is quite low, but the effects are the same.

N: I disagree that the effects are the same. Sure, the means are the same (by design), but the data illustrated on the right is much more variable than the data illustrated on the left.

W: P-values and confidence intervals have quite a bit in common and when interpreted incorrectly can be misleading.

N: I agree, but this is a pretty anodyne statement. Now back to the fisking.

W: Simply put a p-value is the probability of the observed data, or more extreme data, given the null hypothesis is true.

N: Close, but nope. A p-value is the probability of an observed or more extreme test statistic, not the data. It’s an important distinction, and it’s related to the conflation of “effects” with “means” and the different p-values for identical means with different variability around the means in the figure above.

So, anyway, none of this is meant to imply that p-values don’t have limitations. Of course they do. And understanding these limitations is worthwhile. But posts like Wesley’s don’t, in my opinion, do much to foster such understanding.

Posted in statistical description, statistical graphics, statistical modeling | Comments Off

Something old, something new, two somethings under review

I’ve recently revised a manuscript that had been posted on my CV page for a while. It’s a rather technical paper on optimal response selection and model mimicry in Gaussian general recognition theory. In its previous state, it was all and only technical information on these topics. Now, thanks to the urging (and encouraging) of my co-author, it has an introduction and conclusion that actually relate the technical information to a (slightly) wider body of work. It’s here, if you’re interested.

I’ve also recently revised a manuscript that, until today, had never been exposed to public scrutiny. It’s from work I did while at CASL on individual differences in non-native perceptual abilities and how these abilities relate to second language learning of (slightly) higher-level linguistic structure. It feels very nice to finally have it written up and ready for public consumption. It’s here, if you’re interested.

Both papers are under review at, I hope, suitable journals. I mean, the first is under review at pretty much the only journal I can even begin to imagine it being published in. I’m reasonably confident that it will, eventually, be published there. On the other hand, I could see the second fitting in okay in a number of different journals. The one I submitted it to is fairly high-profile (and high impact factor!), though, so it would be nice to get it published there.

That’s all for now.

Posted in SCIENCE! | Comments Off

Quotation of a day

Following (i.e., stealing the idea from) Don Boudreaux, who posts interesting and thought provoking “quotations of the day” (e.g., today’s post on evolution), here’s an amusing bit from page 72 of Hamming’s Digital Filters:

The relationship of formal mathematics to the real world is ambiguous. Apparently, in the early history of mathematics the mathematical abstractions of integers, fractions, points, lines, and planes were fairly directly based on experience in the physical world. However, much of modern mathematics seems to have its sources more in the internal needs of mathematics and in esthetics, rather than in the needs of the physical world. Since we are interested mainly in using mathematics, we are obliged in our turn to be ambiguous with respect to mathematical rigor. Those who believe that mathematical rigor justifies the use of mathematics in applications are referred to Lighthill and Papoulis for rigor; those who believe that it is the usefulness in practice that justifies the mathematics are referred to the rest of this book…. Furthermore, since we are interested in the anatomy of the mathematics, we shall ignore many of the mathematically pathological cases. The fact that we are dealing with samples of a physical function implies that we are trying to understand a reasonable situation.

It makes on feel downright Newtonian. If it works, use it, foundations be damned.

Posted in mildly informative filler | Comments Off

More than who in the what now?

Kirk Goldsberry explains an interesting new basketball shooting statistic he and a colleague have developed today. I’ll be discussing two strange statements that Mr. Goldsberry made. One is strange for logical reasons, and the other is strange for syntactic reasons.

In the introduction of the article, Goldsberry is expressing an amazing fact about LeBron James:

…However, consider the following ridiculous statistical couplet:

No player scored more points close to the basket than LeBron James last season.

No player converted a higher percentage of his shots near the basket than LeBron James last year.

Think about that. Not only did he outscore every player in the entire league within the NBA’s most sacred real estate, he converted his shots at the highest rate, too.

Okay, I’ve thought about it, and I don’t find it ridiculous at all. Unless James took substantially fewer shots close to the basket than did other NBA players (which even someone as NBA ignorant as me knows just can’t be the case), the fact that he was more accurate makes it essentially a mathematical necessity that he would outscore everyone else. This seems like an oddly innumerate bit in an otherwise relatively statistically sophisticated article.

The syntactically strange sentence is more fun. Goldsberry writes:

No player accumulated more points than expected than James.

It’s clear what he means – no one exceeded the number of expected points to a greater degree than did James – but, as written, this sentence is meaningless. I mean, I can garner some meaning from it, but it seems ill-formed for expressing that meaning (or any other).

It reminds me of the plausible Angloid gibberish sentences “More people have been to Russian than I have” and “In Michigan and Minnesota, more people found Mr Bush’s ads negative than they did Mr Kerry’s.

Posted in language | 1 Comment

Picking some more nits

Geoffrey Pullum has an interesting recent essay on the difficulties of pinning down exactly how old the cognitive revolution is. Naturally, I won’t be focusing directly on this topic.

Rather, I’d like to take this opportunity to bemoan the fact that Pullum is uncritically invoking Kuhnian philosophy of science:

Maybe revolution is not quite the right metaphor. I know Thomas Kuhn taught us that science develops through revolutions, the detailed work being done under the assumptions of the last one during periods of “normal science.” And it’s an exciting thought, the idea of an annus mirabilis when the whole conceptual world turns upside down, and what was formerly nonsense becomes accepted science (and vice versa), and old guys who don’t get with the program are left to face an embittered retirement. But I’m inclined to think it isn’t quite like that in this case.

I’d give him credit for challenging the idea that “it isn’t quite like that in this case” if I wasn’t already convinced that it’s never quite like that. As Larry Laudan wrote in 1986 (for the record, that’s 27 years ago) in Science and Values (p. xii, in the preface; emphasis mine):

In sum, this is a book about the role of cognitive values in the shaping of scientific rationality. Among recent writers, no one has done more to direct our attention to the role of cognitive standards and values in science than Thomas Kuhn. Indeed, for more than two decades, the views of Thomas Kuhn – and reactions to them – have occupied center stage in accounts of scientific change and scientific rationality. That is as it should be, for Kuhn’s Structure of Scientific Revolutions caused us all to rethink our image of what science is and how it works. There can be no on active in philosophy, history, or sociology of science whose approach to the problem of scientific rationality has not been shaped by the Gestalt switch Kuhn wrought on our perspective on science. This debt is so broadly recognized that there is no need to document it here. Less frequently admitted is the fact that, in the twenty-two years since the appearance of The Structure of Scientific Revolutions, a great deal of historical scholarship and analytic spadework has moved our understanding of the processes of scientific rationality and scientific change considerably beyond the point where Kuhn left it.

Indeed, we are now in a position to state pretty unequivocally that Kuhn’s model of scientific change, as developed in Structure and elaborated in The Essential Tension, is deeply flawed, not only in its specifics but in its central framework assumptions. It is thus time to acknowledge that, for all its pioneering virtue, Kuhn’s Structure ought no longer be regarded as the locus classicus, the origin and fount, for treatments of these questions. It is time to say so publicly and openly, lest that larger community of scientists and interested laymen, who have neither the time nor the inclination to follow the esoteric technical literature of these fields, continues to imagine that Kuhn’s writings represent the last (or at least the latest) word on these matters.

Some simple math puts the origins of Kuhn’s ideas right around the time the so-called cognitive revolution began (though, as argued by Pullum, it’s not clear exactly when the cognitive revolution started, or even if it has a discernible beginning). It seems that Laudan’s nearly thirty year old call to move past Kuhn’s Structure either wasn’t heard or wasn’t heeded.

Posted in philosophy of science | Comments Off

Mischaracterizing Chomsky

Norbert ponders the first year of Faculty of Language, and in doing so links to the first post. I started reading FoL maybe six months ago, and I didn’t go back and peruse the archives, so I hadn’t previously seen this post.

Anyway, two things amused me about this post.

First, there’s an amusing case of structural ambiguity in the second paragraph (“It” refers to the blog and its stated purpose of rectifying ignorance about the generative enterprise):

It will partly be a labor of hate; aimed squarely at the myriad distortions and misunderstandings about the generative enterprise initiated by Chomsky in the mid 1950s.

Given the rest of the post, even if you didn’t know anything about Chomsky, the generative enterprise, or Norbert’s position with respect to either, you would know that the intention here was not to say that Chomsky initiated myriad distortions and misunderstandings about same. Nonetheless, I was amused by the a priori very reasonable parse that gives exactly this interpretation.

Second, I hadn’t realized that Norbert’s contentious relationship with Christina Behme started with the first post on FoL. She seems to have essentially infinite time on her hands to respond quickly, and often voluminously, to Norbert’s posts, but it surprised me to learn that she was there from the very beginning. I rarely read comments on most of the blogs I follow, but the comments on FoL are often interesting and worthwhile, and the Behme-Hornstein non-interactions provide amusing drama.

Posted in fun with structural ambiguity | Comments Off

Populations and individuals

Maggie Koerth-Baker posted a link at Boing Boing to a blog post purporting to explain why any particular finding in experimental psychology may or may not apply to any particular individual.

Actually, the author (Jamil Zaki) goes a lot farther than that:

Psychological studies are not about you.  They make few if any predictions about how you should live your life, how to tell if you’re an introvert, or anything else about you as an individual.

So, it’s not that it’s uncertain if such studies apply to you, it’s that they make “few if any predictions” about you.

The justification for this assertion is problematic, though. Zaki writes:

A typical study might include 200 people, dividing them into groups (say, people told to act generously versus those told to act selfishly), and demonstrate a statistically significant edge in happiness for one over the other.  Like a batting average, though, even strong differences across groups tell us virtually nothing about how generosity or selfishness would affect the happiness of any one person….

…psychological studies, without telling us about any one person, can tell us about how changes in behavior (again, think generosity) might affect the well-being of whole populations.

A strong difference across groups is, in the vast majority of cases, a difference in the means of some measured variable. Depending on how the individuals making up the groups are distributed, a group (mean) difference can tell us quite a bit about how generosity or selfishness (or whatever experimental manipulation we’re interested in) affects the happiness of any one person.

Of course, an experimenter has already observed how their manipulation affected the individual people in their study. And if we’ve got a reasonable model, we could, in principle, generate (probabilistic) predictions about more or less likely effects on a thus far unobserved individual. The predictions will be noisy, and they will be uncertain, but it seems too strong to say that measured group differences tell us “virtually nothing” about individuals.

If changes in behavior affect whole populations, then by necessity, they affect the individual members of those populations.

Posted in SCIENCE!, statistical description, statistical modeling | 2 Comments