SCS Reads 2010
From Wiki1
The chosen text for this year is Lance, C.E., & Vandenberg, R.J. (Eds.) (2009). Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences. New York, NY: Routledge.
archived list of books considered
Week 1:
- Friday, September 24, 2010 at 1:00 pm
We voted on the book
Brief Tutorial on subscribing to the RSS feed for the SCS Wiki
Week 2: Lance & Vandenberg, Introduction and Chapter 1
- Friday, October 8, 2010 at 1:00 pm
- File:Lance Vandenberg intro ch1 p1.pdf First 10 pages
- File:Lance and Vandenberg intro ch1 p2.pdf Last 10 pages
Materials on selection models
Material related to Mike Ornstein's message about selection models:
Also see Chapter 20 (particularly section 20.5) in Fox, J. (2008). Applied Regression Analysis and Generalized Linear Models, Second Edition.
Note that these models are presented as methods for dealing with non-ignorable (MNAR) missingness.
An interesting article
An article by Nancy L. Allen, Paul W. Holland (1989) 'Exposing Our Ignorance: The Only "Solution" to Selection Bias', Journal of Educational Statistics, Vol. 14, No. 2, pp. 121-140, discusses Rubin's mixture, Heckman's selection and Tukey's simplified selection models. The last few sentences of the article are particularly interesting. In reference to the lead article in the same issue, an article by Howard Wainer entitled "Eelworms, bullet holes and Geraldine Ferraro: Some problems with statistical adjustments and some solutions", Allen and Holland say:
If we have any complaint with Wainer, it is with his title. He may have left someone somewhere clinging to the false hope that selection bias has a "solution," that is, a magic bullet that makes the problem of nonignorable nonresponse go away. To this poor soul we have only one fact to relate. You must be prepared to think as hard about your nonrespondents as you do about your substantive research and to incorporate this into a sensitivity analysis. Otherwise, you have not handled selection bias but have only ignored it.
Week 3
- Friday, October 22, 2010 at 1:00 pm
- Baraldi, A. N., & Enders, C. K. (2010). An introduction to modern missing data analyses. Journal of School Psychology, 48, 5-37
- Visualizing missing data: Page devoted to graphical methods for missing data
Week 4 Lance & Vandenberg, Chapters 4 and 5
- Friday, November 5, 2010 at 1:00 pm
Week 5 Lance & Vandenberg, Chapter 6
- Friday, November 19, 2010 at 1:00 pm
also see short paper by F. Lord
Week 6 Lance & Vandenberg, Chapter 3
- Friday, December 3, 2010 at 1:00 pm
also see
MacCallum, R.C. (2009). Factor analysis. In Millsap, R.E., & Maydeu-Olivares, A. (Eds.) The Sage Handbook of Quantitative Methods in Psychology.
Week 7 Lance & Vandenberg, Chapter 11
- Friday, January 14, 2010 at 1:00 pm
- First meeting of 2011
- Discussion on power analysis. Feel free to post any other relevant readings here!
Week 8 Special topic
- Friday, January 28, 2010 at 1:00 pm
- Michael Friendly to talk on "Advances in Visualizing Categorical Data, Using the vcd, gnm and vcdExtra packages in R" (with Heather Turner, David Firth, Achim Zeileis)
Some background reading:
- Turner H & Firth D (2010). Generalized nonlinear models in R: An overview of the gnm package.
- Meyer D, Zeileis A, & Hornik, K The Strucplot Framework: Visualizing Multi-way Contingency Tables with vcd
Week 9 Lance & Vandenberg, Chapter 9
- Friday, February 11, 2010 at 1:00 pm
Week 10 Lance & Vandenberg, Chapters 7 and 8
- Friday, March 4, 2010 at 1:00 pm
Week 11 Lance & Vandenberg, Chapters 12 and 15
- Friday, March 18, 2010 at 1:00 pm
Week 12 Lance & Vandenberg, Chapter 2 and Thissen and Steinberg (2009) chapter
- Friday, April 1, 2010 at 1:00 pm
Thissen, D., & Steinberg, L. (2009) Item response theory. In R.E. Millsap & A. Maydeu-Olivares (Eds.), The Sage handbook of quantitative methods in psychology.
Fitting the 2PL in R with the ltm package
Commands:
NoDIF <- read.fwf(file.choose(), widths=rep(1, 13))
library(ltm)
descript(NoDIF)
system.time(mod2pl <- ltm(NoDIF ~ z1)) # fits 2PL model
summary(mod2pl)
plot(mod2pl, type="ICC", legend=TRUE, lty=rep(1:3)) # item response curves
plot(mod2pl, type="IIC", legend=TRUE, lty=rep(1:3)) # item info curves
plot(mod2pl, type="IIC", items=0) # test info function
item.fit(mod2pl) # tests of fit
unidimTest(mod2pl) # test if unidimensionality
factor.scores(mod2pl)
par(mfrow=c(4, 4)) # plotting individual item response curves
dif <- coef(mod2pl)[, 1]
for (i in 1:13){
plot(mod2pl, type="ICC", item=i, main=paste("ICC for item", i))
abline(v=dif[i], h=0.5, lty=2)
}
Get the data file: Media:noDIF-IRT-data.txt
A parenthesis on Allan Birnbaum
It is of interest that Allan Birnbaum, who proposed logistic IRT, is the author of a paper that many consider to be one of the most important in the 20th century on the foundations of statistical inference. It is appropriately named:
- Birnbaum, Allan (1962). "On the foundations of statistical inference". J. Amer. Statist. Assoc. (American Statistical Association) 57 (298): 269–326. doi:10.2307/2281640. (With discussion.)
The paper shows that the joint acceptance of the 'sufficiency principle' and of the 'conditionality principle' implies acceptance of the 'likelihood principle', i.e. two experiments that yield the same likelihood function (defined up to an arbitrary multiplicative constant) should yield exactly the same inferences.
The principle says nothing about the manner in which inferences ought to be characterized. It only asserts an equivalence class of equivalent results.
Now, the likelihood function is just a dead function defined on the parameter space. It does not imply a distribution over the parameter space nor is its own distribution over repeated sampling relevant in Birnbaum's result, i.e. two experiments with the same likelihood function yield the same inference even though the distributions of those likelihood functions are different. This means that inference leaves us with a function over the parameter space without a guide to its interpretation and with nothing random, thus no way of addressing the accuracy of inferences.
To bring the likelihood to life, we need to embed it in something random. The two principal ways are to use sampling variation or to view the parameters as random through a Bayesian prior. To many, sampling variation seems unreasonable because it interprets the observed data in the context of other observations that might have been observed but were not -- and violates the likelihood principle. To others, a distribution on the parameter space seems unreasonable because, in general, it has no 'objective' interpretation in terms of relative frequencies. Thus, restoring randomness to the likelihood function seems to require a choice between two alternatives neither of which is entirely satisfactory.
Some approaches, in particular Fraser's structural inference and, less successfully, Fisher's fiducial inference, attempt to parry this problem by using the randomness of the error term, which has a frequentist interpretation, to induce a distribution on the parameter space. Fraser's approach probably come closest to succeeding but it only clearly works with models that are generated by a group (in the algebraic sense) of transformations.
Thus, statistics, at its core, faces a yet unresolved dilemma. Must we accept the relevance of data that was not observed or must we abandon an 'objective' interpretation of probability.
You are well cautioned not to worry too much about this. Many who descended to the essential core of mathematics and statistics to discover the apparent fragility of their foundations did not come back in very good shape: Cantor, Godel and Birnbaum all had tragic lives. Russell is a happy exception.
As a footnote: there is, but only asymptotically, a fascinating relationship between the shape of the likelihood function and its distribution. This is the basis of Fisher Information, Wald tests using the Hessian to estimate the variance of the MLE, etc. It would be interesting to animate this in 3d using a Poisson model for example.
A footnote on Alan Birnbaum, Lord & Novick and IRT
Lord & Novick's Statistical Theories of Mental Test Scores was published in 1968, listing the authors as "Lord and Novick, with contributions by Alan Birnbaum". The ambitious goal of this book was to take test theory from its classical, Thurstonian roots, as embodied in Harold Gulliksen's (1950) Theory of Mental Tests into the modern era. Below are a few comments about the genesis of this book and Birnbaum's contribution.
In 1966, my first year as a psychometric fellow at Princeton, I took the Test Theory course offered by Lord and Novick. Teaching was clearly not their priority, because the class was offered from 8:00-10:00, so as not to interfere with their real work, which was writing the book. Each week or so, we would get a few draft chapters, and Fred or Mel would "discuss" them over the following weeks. But as I recall, discussion consisted largely of writing THEOREM 1: ... on the board and proceeding to prove Theorem 1. Next, came THEOREM 2, ... After some weeks of this we students (Howard Wainer, Charles Lewis, James Arbuckle, Jim Ramsay and others) formed a Flaws and Counterexamples Club. It was everyone's job to try to find a flaw in a proof, or --- better yet, a counterexample. So, after the proof on the board, someone could say "Excuse me, sir. I think I have a counterexample to THEOREM 15." Charlie Lewis turned out to be the best at this, earning a mention in the preface that "Mr. Charles Lewis offered many penetrating comments that impelled us to sharpen our arguments..." I was better on the flaws side, earning a mention for having "checked a number of chapters for typographical errors".
All this certainly livened up the class, but at the same time, there was an ongoing seminar series in the Psychometric Research Group at ETS, with Karl Joreskog, Michael Browne, Walter Kristoff and Alan Birnbaum and others attending. I think this served as an impetus for Birnbaum to develop the materials on latent trait models based on the logistic distribution that now appear in Part 5 of the book. I recall that the material in the early draft on item characteristic curves based on the normal ogive model was extremely complex and peppered with many long equations of multiple integrals. Sometime toward the end of the course Birnbaum's chapters began to arrive and our spirits brightened. The material was difficult, but it extended the ideas of modern test theory greatly. For me, this was a key development in item response theory.
