MCMC in Educational Research

• Published in: Handbook of Markov Chain Monte Carlo pp. 531 - 546
• Main Authors: Levy, Roy, Mislevy, Robert J., Behrens, John T.
• Format: Book Chapter
• Language: English
• Published: United Kingdom Chapman and Hall/CRC 2011; CRC Press LLC
• Subjects: BIOLOGY, LIFE SCIENCES; Probability & statistics; Posterior Distribution; Random Effect Variance; Acceptance Probability; Count Data; Covariate Dependence; Survival Probabilities; Demographic Parameters; Random Effect Terms; Posterior Conditional Distribution; RJMCMC Algorithm; Proposal Distribution; True Population Sizes; Joint Posterior Distribution; Mixed Effects Model; Adult Survival Probabilities; Prior Sensitivity Analysis; Auxiliary Variables; Recovery Probabilities; Φ1 Φa; Posterior Model Probabilities; Bayes Factor; Ring Recovery Data; RJ; Regression Parameters; Posterior Probability
• ISBN: 1420079417; 9781420079418
• DOI: 10.1201/b10905-23

Quantitative educational research has traditionally relied on a broad range of statistical models that have evolved in relative isolation to address different facets of its subject matter. Experiments on instructional interventions employ Fisherian designs and analyses of variance; observational studies use regression techniques; and longitudinal studies use growth models in the manner of economists. The social organization of schooling-of students within classrooms, sometimes nested within teachers, of classrooms within schools, schools within districts, districts within states, and states within nations-necessitates hierarchical analyses. Large-scale assessments employ the complex sampling methodologies of survey research. Missing data abound across levels. And most characteristically, measurement error and latent variable models from psychometrics address the fundamental fact that what is ultimately of most interest, namely what students know and can do, cannot be directly observed: a student's performance on an assessment may be an indicator of proficiency but, no matter how well the assessment is constructed, it is not the same thing as proficiency. This measurement complexity exacerbates computational complexity when researchers attempt to combine models for measurement error with models addressing the aforementioned structures. Further difficulties arise from an extreme reliance on frequentist interpretations of statistical methods that limit the computational and interpretive machinery available (Behrens and Smith, 1996). In sum, most applied educational research has been marked by interpretive limitations inherent in the frequentist approach to testing, estimation, andmodel building, a plethora of independently created and applied conceptualmodels, and computational limitations in estimatingmodels thatwould capture the complexity of this applied domain. This chapter discusses how a Markov chain Monte Carlo (MCMC) approach to modelestimation and associated Bayesian underpinnings address these issues in threeways. First, the Bayesian conceptualization and the form of results avoid a number of interpretive problems in the frequentist approachwhile providing probabilistic information of great value to applied researchers. Second, the flexibility of the MCMC models allows a conceptual unification of previously disparate modeling approaches. Third, the MCMC approach allows for the estimation of the more complex and complete models mentioned above, thereby providing conceptual and computational unification. Because MCMC estimation is a method for obtaining empirical approximations of pos-terior distributions, its impact as calculation per se is joint with an emerging Bayesian revolution in reasoning about uncertainty-a statistical mindset quite different from that ofhas characterized educational research.A fertile groundwork was laid in this field from the 1960s through the 1980s by Melvin Novick. Two lines of Novick's work are particularly relevant to the subject of this chapter. First is the subjectivist Bayesian approach to modelbased reasoning about real-world problems-building models in terms what one knows and does not know, from experience and theory, and what is important to the inferential problem at hand (see, e.g. Lindley and Novick, 1981, on exchangeability). His application of these ideas to prediction acrossmultiple groups (Novick and Jackson, 1974) foreshadows the modular model-building to suit the complexities of real-world problems that MCMC enables. In particular, the ability to "borrow" information across groups to a degree determined by the data, rather than pooling the observations or estimating groups separately, was amajor breakthrough of the time-natural from a Bayesian perspective, but difficult to frame and interpret under the classical paradigm. Second is the realization that broad use of the approach would require computing frameworks to handle the mathematics, so the analyst could concentrate on the substance of the problem. His Computer-Assisted Data Analysis (CADA; Libby et al., 1981) pioneered Bayesian reasoning about posteriors inways that are today reflected in the output of MCMC programs such asWinBUGS (Spiegelhalter et al., 2007).