Cognitive diagnostic models are an area of psychometric research that has seen substantial growth in the past decade, though the mathematics behind them, dating back to MacReady and Dayton (1977). The reason that they have been receiving more attention is that in many assessment situations, a simple overall score does not serve our purposes and […]
Guttman errors are a concept derived from the Guttman Scaling approach to evaluating assessments. There are a number of ways that they can be used. Meijer (1994) suggests an evaluation of Guttman errors as a way to flag aberrant response data, such as cheating or low motivation. He quantified this with two different indices, G […]
What is a rubric? It’s a rule for converting unstructured responses on an assessment into structured data that we can use psychometrically. Why do we need rubrics? Measurement is a quantitative endeavor. In psychometrics, we are trying to measure things like knowledge, achievement, aptitude, or skills. So we need a way to convert qualitative data […]
Test security is an increasingly important topic. There are several causes, including globalization, technological enhancements, and the move to a gig-based economy driven by credentials. Any organization that sponsors assessments that have any stakes tied to them must be concerned with security, as the greater the stakes, the greater the incentive to cheat. And threats […]
Today I read an article in The Industrial-Organizational Psychologist (the colloquial journal published by the Society for Industrial Organizational Psychology) that really resonated with me. Has Industrial-Organizational Psychology Lost Its Way? -Deniz S. Ones, Robert B. Kaiser, Tomas Chamorro-Premuzic, Cicek Svensson Why? Because I think a lot of the points they are making are also true […]
A modified-Angoff method study is one of the most common ways to set a defensible cutscore on an exam. If you have a criterion-referenced interpretation, it is not legally defensible to just conveniently pick a round number like 70%; you need a formal process. There are a number of acceptable methodologies in the psychometric literature […]
I often hear this question about scaling, especially regarding the scaled scoring functionality found in software like FastTest and Xcalibre. The following is adapted from lecture notes I wrote while teaching a course in Measurement and Assessment at the University of Cincinnati. Test Scaling: Sort of a Tale of Two Cities Scaling at the test […]
Item response theory (IRT) is a family of mathematical models in the field of psychometrics, which are used to design, analyze, and score exams. It is a very powerful psychometric paradigm that allows researchers to build stronger assessments. This post will provide an introduction to the theory, discuss benefits, and explain how item response theory […]
ASC attended the 2016 Conference on Test Security (COTS), held October 18-20 in Cedar Rapids IA, graciously hosted by Pearson. The conference brings together thought leaders on all aspects of test security, including statistical detection of test fraud, management of test centers, candidate agreements, investigations, and legal implications. ASC was lucky enough to win three […]
If you are delivering high-stakes tests in linear forms – or piloting a bank for CAT/LOFT – you are faced with the issue of how to equate the forms together. That is, how can we defensibly translate a score on Form A to a score on Form B? While the concept is simple, the methodology […]
