I’ve blogged about GRT (General Recognition Theory) at least a few times before (e.g., here, here, here). The third of those links goes to an introductory GRT post (#1 of an unknown number of planned posts). Here’s the super-abbreviated version: GRT is a multidimensional extension of signal detection theory (SDT). The simplest GRT model consists of four bivariate Gaussian densities (perceptual distributions) and two decision bounds. These are used to model identification confusion patterns for stimuli consisting of a factorial combination of two levels on each of two dimensions (e.g., square or octagon on a shape dimension, red or purple on a hue dimension).
Robin Thomas and I have done some theoretical work with this model recently. More specifically, we wrote a paper in which we described how failures of decisional separability (i.e., decision bounds that are not parallel to the coordinate axes) are not identifiable in the model. The basic issue is that, with linear bounds, affine transformations of the modeled perceptual space can always map a model with failure of DS onto a model in which DS holds. (This is illustrated in the second ‘here’ link above.)
More recently, Soto, et al. developed a multilevel GRT model (called GRT wIND – GRT with Individual Differences) and claimed to have solved the identifiability problem that Robin and I detailed (pdf). Still more recently, Soto and Ashby wrote a second paper with GRT wIND (pdf), which I reviewed (twice) for the journal Cognition. Yesterday, I got an email (as a member of the Society for Mathematical Psychology) announcing an R package for fitting GRT wIND (and carrying out various other GRT-related analyses).
The paper I reviewed has since been accepted for publication, and I confess some measure of frustration with the fact that Soto and Ashby (and the editor who handled the paper for Cognition) seem to have completely ignored the comments and suggestions from my second (and signed) review.
One key issue I raised was a concern that the GRT wIND model is over-parameterized, i.e., that given a data set, the parameters are not uniquely identifiable. In part motivated by my frustration, and in (larger) part because I actually want to know if my concern is valid or not, and in part because they made the model fitting software available, I decided to ry to validate (or not) my concern about over-parameterization in GRT wIND.
Some GRT wIND background first: The group level model consists of four bivariate Gaussian densities. Each has a mean vector and a covariance matrix. The individual subject level for subject consists of four bivariate Gaussian densities with the same means and transformed covariance matrices as well as two decision bounds, the latter possibly not parallel with the coordinate axes.
If the group level covariance matrix corresponding to stimulus (i.e., the level of the dimension and the level of the dimension) is given by:
then the corresponding individual level matrix for subject is given by:
The two additional parameters and scale the subject’s space overall () and scale the two dimensions relative to one another (). As increases, the marginal variances decrease, thereby increasing the salience of the stimuli and reducing the number of (predicted) errors. As increases, the marginal variance on the axis decreases and the marginal variance on the axis increases, shifting the relative salience of the two dimensions. As Soto, et al., note, equivalent changes in response probabilities could be obtained by leaving the covariance matrices alone and shifting the means of the group level densities, instead.
As a brief aside, it’s interesting (to me) to note that you can represent the individual level matrix as a linearly-transformed group level matrix:
The point here being that the Silbert & Thomas results concerning linear transformations and identifiability of decisional separability apply to the individual subject level models in GRT wIND. Soto, et al., say that the individual subjects share a perceptual space, but in reality they share some, but not all, properties of the group level perceptual space. In GRT wIND, expansion/contraction of the space and stretching/shrinking the group space along one or another coordinate axis are acceptable transformations, but rotations, shears, etc…, are not. So, GRT wIND solves the decisional separability identifiability issue if this distinction is valid for a given stimulus set and experimental subject group, but it doesn’t solve the problem in general.
But I digress.
A little more background about GRT: In some of the oldest papers on GRT, there are mentions of the fact that the means and marginal variances of the perceptual distributions are not both uniquely identifiable (this is closely related to the note above about scaling the variances vs the means in GRT wIND). I long assumed that this was true, but to the best of my knowledge, no one had proven it. I figured out that an affine transformation can be applied to each perceptual distribution (with arbitrary means and covariance matrices) in a GRT model to force it to have unit variances and appropriately shifted means (where by “appropriately” I mean that the shift in the means exactly balances the scaled variances, producing the same predicted response probabilities for a given set of decision bounds, assuming decisional separability holds, which see above with regard to identifiability). More specifically, if you have decision criteria and , the following transformation gives you unit variances and appropriately shifted means:
Application of this transformation produces a new covariance matrix and new mean vector . The transformed covariance matrix is the correlation matrix:
And the transformed mean vector is shifted by the signed distance (in standard deviation units) between the mean and the decision criterion (on each dimension):
Note that this transformation is invertible. The upshot of this is that a standard GRT model can be scaled arbitrarily. Or, to repeat the point made above, the means and marginal variances are not both identifiable.
So, what does this all have to do with GRT wIND? My intuition was that this identifiability issue applies to GRT wIND, too. I didn’t feel like working through all the algebra to prove that this is the case, so I did the next best thing. Specifically, I simulated individual subject data using two GRT wIND models. In one, the group level perceptual means describe a square centered on the origin of the space with sides of length four, and the group level covariance matrices have randomly generated variances and correlations. In the second model, I applied the “unit-variance” transformation to the model (with the coordinate axes as the ‘decision criteria’).
I also randomly generated a set of 50 , , , and values, where and are the GRT wIND parameters described above, and specifies the angle between and the axis, and specifies the angle between and :
The following figure illustrates the mean (red), median (blue), and maximum (green) absolute error between the predicted identification confusion probabilities for the two models:
And here are the and values:
And here is the R script that runs the simulations and generates these plots.
I’ve run the simulation a number of times, and, while sometimes the maximum errors are larger (~.15 or thereabouts), the two models consistently produce extremely similar, in many cases essentially identical, confusion matrices.
I take this as strong support for my intuition that GRT wIND is over-parameterized. A mathematical proof would be better, of course, but simulation results provide relevant evidence. Now, I’m not totally sure why there is any deviation between the models, though there is no correlation between any of the randomly generated parameters and the error. But because there is some error, the simulations don’t provide the kind of airtight case a proof would. Nonetheless, it is all but unimaginable to me that the true parameters of the two models could be recovered by fitting sets of confusion matrices that are this similar to one another.
As it happens, I would like to fit GRT wIND models to simulated data to test how accurately known parameters are recovered and to come at the problem from the opposite direction (i.e., instead of seeing if different parameter values can generate essentially identical data, see if different parameter values can provide essentially identical model fits to identical data). Alas, in the R package, the function
grt_wind_fit crashes R, and the function
grt_wind_fit_parallel returns an error. I’ve opened up an issue on the GitHub repository, so maybe I’ll try again when/if I can get the code to work on my computer(s). And, I suppose, depending on some of my colleagues and collaborators react, I may also get to work on a more rigorous mathematical treatment of the matter.