The first thing I did was wonder why there was a graph at all. You have two numbers which are already discussed in the text: “That means that the actual default rates for CDOs were more than two hundred times higher than S&P had predicted.” (emphasis in original)

So, what purpose does a figure serve here? At best, no purpose at all.

I think it’s actually worse than no figure at all, though. In thinking about the graph, I wondered what the outer circles represent. I assumed they represent 100%, but neither of the shaded areas really look like the corresponding percentages of the circles’ areas. At first glance, they both look too big to me, but then I’m not at all certain that my (or anyone’s) intuitions about areas like these are reasonable.

So, I measured the circles. In the hard copy of the book that I have, the diameter of each of the big circles is 43mm, while the diameter of the small shaded circle is 2mm, and the diameter of the big shaded circle is 23mm.

The area of a circle is , so the area of the small shaded circle is just . The area of the big shaded circle is , and the area of each of the 100% circles is .

The ratio of the areas of the big shaded circle and the 100% circle is about right: . The ratio of the areas of the small shaded and 100% circle is pretty close, too: (it should be ).

But the ratio of the actual to the observed default rate is , while the ratio of the areas of the big and the small shaded circles is . That’s a big difference.

So, my intuition that the shaded areas were inaccurate was wrong, but the depicted relationship between the data points of interest is way off. Given that the information provided by the two data points is communicated effectively in the text, an inaccurate and nonintuitive figure is worse than simply pointless.

Of course, this is to say nothing of the rest of the book (though a quick glance through it reveals a mix of reasonable statistical graphics and silly, similarly pointless illustrations scattered throughout). You can read a brief review of the content of the book at Normal Deviate (with lots of interesting discussion in the comments) as well as a discussion of two other reviews at Andrew Gelman’s blog.

In the SCM, the probability of giving response r to stimulus s is (where is the bias to give response and is the similarity between stimulus and stimulus , and is the number of stimuli):

(1)  

You might want to use this model because it has convenient, closed-form solutions for the similarity and bias parameters:

(2)  

Whence the values? You could just divide your confusion counts by the row totals, or you could, since you care about statistical rigor, first obtain maximum likelihood estimates of your confusion counts, only then normalizing by the row sums to obtain MLEs of the confusion probabilities.

And you can obtain MLEs of your confusion counts by using iterative proportional fitting. I learned about iterative proportional fitting from Rob Nosofsky in one of the best classes I took as a graduate student (or any other kind of student, come to think of it). And just today, I found myself needing to use iterative proportional fitting once again.

So I wrote an R function:

iter.prop.fit <- function(M,delta = .001){

  nr <- nrow(M)
  nc <- ncol(M)
  M.h <- matrix(1,nrow=nr,ncol=nc)
  dimnames(M.h) <- dimnames(M)
  d.t <- 1
  nz <- .001
  while(d.t >= delta){
    M.h.a <- M.h
    for(ri in 1:nr){
      for(ci in 1:nc){
        M.h[ri,ci] <- M.h[ri,ci]*sum(M[ri,])/sum(M.h[ri,]+nz)
        M.h[ri,ci] <- M.h[ri,ci]*sum(M[,ci])/sum(M.h[,ci]+nz)
        M.h[ri,ci] <- M.h[ri,ci]*(M[ri,ci]+M[ci,ri])/(M.h[ri,ci]+M.h[ci,ri]+nz)
      }
    }
    d.t <- max(abs(M.h.a-M.h),na.rm=T)
  }
  return(M.h)

}

M is your confusion matrix, delta is a criterion that determines when the algorithm stops (smaller delta means less change from one step of the algorithm to the next), and M.h (h = hat, so M.h = ) is the matrix of maximum likelihood confusion counts.

The basic idea is to take a matrix full of ones and gradually adjust it to match, as closely as possible, the counts in the observed confusion matrix.

Because we’re using iterative proportion fitting to get SCM MLEs, there are three steps for each iteration of the algorithm. First, M.h is adjusted by row (i.e., stimuli). Second, M.h is adjusted by column (i.e., responses). Third, and finally, M.h is adjusted for each unique pair of stimuli (i.e., symmetric similarity).

When the largest change from one step to the next is smaller in magnitude than delta, the algorithm stops and returns M.h. Once you have M.h, you can normalize by row to get maximum likelihood confusion probability estimates (i.e., values) which you can plug into the equations above to obtain maximum likelihood estimates of the similarity and bias parameters.

What you do with the these is, of course, up to you.

I find this match of algorithm and model interesting, primarily because it seems kind of backwards. When you want to find MLEs for the parameters of, say, GRT, you can run a Newton-Raphson algorithm that navigates parameter space to find the MLEs for a given data set.

This makes intuitive sense to me – you have data, you make an educated guess about initial values, and you have rules for moving from your initial guess to your best guess.

With iterative proportional fitting and the SCM, on the other hand, you find MLEs of the data first, and only after you have this do you calculate parameter values. You don’t make an initial guess about your parameters, and you don’t explicitly navigate parameter space in order to find your best guess. Of course, because you have closed form solutions for the SCM parameters, for each step in the algorithm, you could calculate parameter values if you wanted to, so, even though iterative proportional fitting is expressed entirely in terms of the data, the algorithm is effectively following a simple set of rules for implicitly navigating parameter space.

“[N]ews is to the mind what sugar is to the body”. I’ve long since given up ingesting news. Sugar not so much.

“Over 41 days, the researchers collected more than 2.4 billion readings and documented 9.4 million interactions between the workers.” I assume they’ve upgraded to R 3.0.0.

We should keep giving Diederik Stapel plenty of attention.

“No one is suggesting that there is anything fraudulent about the results, but the charges that some of Dijksterhuis’s key papers may report false positives is a particular embarrassment for the Netherlands.” See also this, this, this, and this.

Criticism 1 of Null Hypothesis Significant Testing. Criticism 2. Criticism 3. Criticism 4. Some other thoughts on NHST that are possibly inconsistent with some other other thoughts on NHST. I’ve been planning to write about some of these posts for a while. I’d still like to.

The updated draft frames everything in terms of mimicry of decisional separability rather than the presence/absence of decisional separability proper, since the optimal model makes use of discriminant functions between each pair of perceptual distributions, which is distinct from the decision bounds discussed in, e.g., my other recent paper with Robin. There are also some additional figures and a new section on how linear transformations of the modeled perceptual space determine a broad class of models that implicitly mimic decisional separability (in contrast to the narrow class of models that explicitly mimic decisional separability).

Any mistakes in the paper are mine, since Robin hasn’t had a chance to dig in and review it yet.

This dendrogram shows a (fake, er… simulated) hierarchical clustering solution to data indicating the perceptual distances between pairs of some Arabic consonants. To be clear, I am working with real data along these lines (and real hierarchical clustering solutions from this data), but I am not at liberty to share the data or any associated analyses right now, but I wanted to post about this font stuff, so I made some data up. Anyway, fake or not, clusters that are lower down in the dendrogram consist of more similar items (or clusters) than clusters that are higher up.

I was dissatisfied with the orientation and size of the labels in the dendrogram, but I couldn’t figure out how to do anything about either issue. I found some hilariously (and all too typically) awful responses in various R “help” forums online (e.g., here), and I tried every strategy I could muster on my own by digging around in the relevant help files. The labels in the dendrogram persisted in their smallness and sideways orientation.

The solution I devised is inelegant, but it works: give the plot function blank labels and add text after the fact. It’s inelegant because you have to fiddle with the height of each label (or each pair of labels) to get it to look okay. (Depending on your personal degree of obsessiveness, this may be a feature rather than a bug.) The inelegance and extra work are worth it (to me, anyway), since this method automatically reorients the characters and it makes it easy to change the font size.

Here’s the better version of the figure above, with example code following:

# unicode for various IPA symbols
ipa.lab =  c("\U0263","x","\U0295","\U0127","\U0294","h")
n.lab = length(ipa.lab)

# example distance matrix, cluster fitting
D.s = matrix(c(0.00,1.55,2.04,2.26,2.45,2.14,
               1.55,0.00,2.62,1.62,2.43,1.91,
               2.04,2.62,0.00,2.12,1.40,2.06,
               2.26,1.62,2.12,0.00,2.11,0.78,
               2.45,2.43,1.40,2.11,0.00,2.10,
               2.14,1.91,2.06,0.78,2.10,0.00),
             nrow=6,byrow=T)
dimnames(D.s) = list(ipa.lab,ipa.lab)
D.t = as.dist(D.s)
fit = hclust(D.t,method="average")

# put the labels in the right order
lab.reord = ipa.lab[fit$order]

# write a pretty figure to a png file
png(file="dendrogram_example.png",width=750,height=750,res=100)
par(mai=c(.5,.5,.5,.5))
plot(fit,labels=rep("",n.lab),axes=F,
     xlab="",sub="",ylab="",cex=1.5,main="",lwd=2)
heights = c(1.19,1.19,.58,.58,1.34,1.34)

# cex = character expansion, the key argument here
text(1:n.lab,heights,labels=lab.reord,cex=1.75)
dev.off()

As noted in that last link, “There’s a huge literature on IQ and g, going back almost 100 years. This stuff is not based on brain imaging, but just on IQ test scores, and it’s a complex topic.”

As it happens, I’ve read a teeny-tiny bit of this literature. In particular, a couple years ago (after reading this blog post) I read van der Maas et al.’s 2006 paper A Dynamical Model of General Intelligence: The Positive Manifold of Intelligence by Mutualism [pdf], which presents a pretty compelling argument against g being anything other than a statistical artifact that can be readily explained by positively interacting cognitive subsystems.

When I looked at the ‘Fractionating’ paper, I was surprised to see that the van der Maas et al. paper isn’t cited. The take-home message seems to be more or less the same in both papers (even if it’s not identical, they’re both at least roughly consistent with one another), so it’s a natural predecessor. So why isn’t it cited?

I’ve often felt that it’s impossible, or very nearly so, to keep up to date with all the literature relevant to my research interests. I imagine a lot of people feel this way. Or that a lot of people don’t actually keep fully up to date, however accurate they’re feelings on the matter are.

And I wonder how much noise this adds to the scientific system. If there’s enough redundancy in the system, then missing this particular paper or that particular paper maybe doesn’t matter so much, since you can pick up the slack with a different particular paper that covers the same basic issue.

But what if you miss a paper (or a set of papers) and thereby miss the fact that someone’s done something crucial to your work? This seems likely to create redundancy, which seems like a waste of resources.

I’m not sure how big a problem this is, nor do I have any clever solutions in mind. Maybe a certain amount of noise and redundancy is inherent to modern science simply by virtue of the number of people conducting science.

Or maybe I’m just fishing for an excuse to not have to read every last (all too often extremely mediocre) study that’s slightly greater than tangentially related to my research.

This is not surprising because, while Jedi mind tricks are an obvious part of the Star Wars canon (e.g., “You don’t need to see his identification… These aren’t the droids you’re looking for…”), the “mind meld“ is from Star Trek, not Star Wars.

Now, I don’t really have a horse in this race. I like science fiction, and I’ve enjoyed some of both franchises, but I don’t strongly prefer one to the other, and I don’t particularly care how well Obama knows either one.

So, Obama mixed up his Wars and Trek references? I’m more concerned about secrecy and the drone program and the drug war.

That said, I can still engage in some nerdy frivolity, particularly when language comes into play. When someone asserts that “[Obama] was more correct than any of his critics could possibly imagine,” as a certified language geek, I can’t let it pass without pointing out a serious (linguistic) flaw in the “evidence” provided for the assertion.

The assertion is based on the fact that there is, as it turns out, something called the “Jedi meld” in the Star Wars canon. Furthermore, this non-Trek meld is arguably conceptually more appropriate to the point Obama was making. Whereas the Vulcan Mind Meld is all about sharing memories and emotions (and discovering secrets), the Jedi Meld “permits a group of Jedi to connect their minds so closely as to act as a single person.” Obama was lamenting the fact that he and Congress can’t make collaborative decisions with respect to the budget. A Jedi Meld would be more useful here than a Vulcan Mind Meld, to be sure.

Hence, Petey argues that the Jedi Meld “was the right reference for Obama to make.”

Maybe so, conceptually speaking, but Obama didn’t make that reference. Obama referred to the “Jedi mind meld.” As far as I can tell, it’s only called a “mind meld” in the Trek universe. The only time “mind” comes immediately before “meld” in the Star Wars wiki article on the Jedi Meld is in the section discussing Obama’s reference, whereas virtually every reference to a “meld” in the Star Trek wikipedia article is preceded by “mind.”

I don’t find it very plausible that Obama’s knowledge of the Star Wars canon is so deep that he both knows what a Jedi Meld is and refers to it using the apparently non-standard, and very Star-Trek-ian, “mind” modifier. It seems much more likely that he has relatively shallow knowledge of Wars and Trek and that he simply crossed the (conceptual) streams, so to speak.

I’m not sure why Obama should have had any nerd street cred in the first place, but if he’s lost some as a result of this, this seems appropriate to me.

In null-hypothesis significance testing (NHST), the false alarm rate (or, if you’re a glutton for confusing terminology, the Type I error rate) is the rate at which a true null model is incorrectly rejected. Power, on the other hand, is the probability of correctly rejecting a false null model (or 1 – the Type II error rate).

Even if we’re doing a very simple test (e.g., comparing the means of two samples), the “null” model can, in principle, be pretty much whatever you want. Of course, there are conventional choices for various tests (e.g., for comparing two means, ). In a Bayesian framework, the model is more elaborate, since you have priors, at least, and maybe also a “region of practical equivalence,” or ROPE, which is an interval rather than a point null (see, e.g., Kruschke’s BEST).

In both NHST and Bayesian frameworks, you also have a decision procedure. In NHST, for example, you would reject a null hypothesis if an observed test statistic exceeds a certain criterion. The criterion is supposed to be set, a priori, based on some determination of acceptably low false alarm rates (α). In a Bayesian framework, the decision procedure might be to reject a model (or a particular assumption about a model) if, say, the 95% highest density interval (HDI) for the posterior distribution of a parameter of interest is entirely outside the ROPE associated with that model (assumption).

Okay, so, for calculating power, whether you’re working as a Bayesian or not, you also have an alternative (to the null) model. Using any of a number of methods (e.g., analytic math, simulations), you figure out how likely you are to reject a particular null model using your decision procedure and assuming that your alternative model is true. Fairly straightforward, and pretty uncontroversial (at least if it’s prospective power).

For calculating false alarm rates, you do something very much analogous to calculating power. In NHST, you assume that your null model is true, and then you take your decision procedure and calculate how likely you are to incorrectly reject that null model. For a given false alarm rate (α), you can figure out associated criterion value(s) of your test statistic, and if your observed test statistic exceeds the criterion, you reject the null.

In a Bayesian framework, you can do more or less the same thing, I would think. You assume your (more elaborate) null(ish) model is true – priors, ROPE, and all – and you use your decision procedure to estimate the proportion of tests that would incorrectly reject the null(ish) model.

For Kruschke, the crucial bit that makes power okay is that, unlike false alarm rates, “power is not being used to make decisions about data that are actually obtained, nor is power being used to set decision criteria for data that are actually obtained.”

So, here’s what I don’t get. If you’ve done the work to carry out a Bayesian t-test, then you’ve already got everything you need to calculate a false alarm rate: a null(ish) model with priors, a ROPE, and a HDI-based decision procedure. Which is to say that for every model sufficient to the task of testing a statistical hypothesis with a particular data set, there is a false alarm rate. It may not be the false alarm rate for Bayesian t-test’s in general, but it is the false alarm rate for the model and decision procedure you’ve chosen to use.

It seems to me that this false alarm rate provides useful information. If your model has a 0.50 false alarm rate, for example, you might want to know that, figure out why it’s so high, and then change your model to lower that number. Which is to say, in part, that it is appropriate to take the false alarm rate for a given model and decision procedure into account when making decisions about obtained data.

For a given model (or parameter therein) for which you want to make a decision, the utility of knowing the relevant false alarm rate is much like the utility of knowing the relevant power. Whether you’re a Bayesian or a devotee of NHST, if you’re carrying out statistical tests, power and false alarm rates are both relevant.

Full disclosure: I took multiple statistics and cognitive modeling courses from John Krushcke, and I was the grader for his Bayesian Data Analysis class multiple times as a doctoral student at IU. When I was his grader, he was developing his BDA textbook, and I’m even acknowledged in the book (next to one Thomas Wisdom, who I don’t know, but who has a much better last name than me). So maybe this blog post is kind of like me becoming Darth Vader to John’s Obi Wan Kenobi. Or maybe that’s overstating the magnitude of this disagreement.

Solomon, on the other hand, responded with agitation: “They’re not leftover – Mama just made them!”

To which Zoë, irritated by Solly’s flawed parse, replied: “The oatmeal was leftover, not the muffins!”

Silas merely screeched, either in joy or anger. It’s hard to tell sometimes.

The lesson: structural ambiguity and simmering sibling annoyance are a moderately volatile combination.

Update: And now, a day later, the muffins are [leftover [leftover oatmeal [muffins]]].

I believe that this statement is important in at least three ways. First, it asserts that at least one synonym of ‘moose’ exists. Second, it asserts that there is exactly one synonym of ‘moose’. Third, it asserts that Solomon generated this lexical knowledge on his own.

I may be reading too much into it, but I’m impressed that my six year old has apparently worked out existence and uniqueness proofs in the domain of lexical semantics.

–

1. When my 11 year old daughter Zoë was born, I decided that I wanted my kid(s) to call me ‘Pops.’ When Zoë was 2 or so, when I would arrive home from work, she would see me and exclaim “Pops!” to which I would respond “Zo!” After a couple days of this, she took the initiative and did both parts of our dialogue on her own, exclaiming “Popso!” and it’s been my name every since.

2. The derivation of Popso is somewhat ambiguous. It’s either Pops + o, i.e., the ‘z’ from ‘Zo’ is dropped, or it’s Pop + Zo, i.e., the ‘s’ from Pops is dropped and the ‘z’ from ‘Zo’ is devoiced.