What is a standard setting study?
A standard setting study is a formal process for establishing a performance standard. In the assessment world, there are actually two uses of the word standard – the other one refers to…
Nathan Thompson earned his PhD in Psychometrics from the University of Minnesota, with a focus on computerized adaptive testing. His undergraduate degree was from Luther College with a triple major of Mathematics, Psychology, and Latin. He is primarily interested in the use of AI and software automation to augment and replace the work done by psychometricians, which has provided extensive experience in software design and programming. Dr. Thompson has published over 100 journal articles and conference presentations, but his favorite remains https://pareonline.net/getvn.asp?v=16&n=1.
A standard setting study is a formal process for establishing a performance standard. In the assessment world, there are actually two uses of the word standard – the other one refers to…
If you are dealing with data science, which psychometrics most definitely is, you’ve probably come across R. It is an environment that allows you to implement packages for many different types of analysis, which are built by a massive community of data scientists around the world. R has become one of the two main languages […]
Psychometric forensics is a surprisingly deep and complex field. Many of the indices are incredibly sophisticated, but a good high-level and simple analysis to start with is overall time vs. scores, which I call Time-Score Analysis. This approach uses simple flagging on two easily interpretable metrics (total test time in minutes and number correct raw score) […]
Subject matter experts are an important part of the process in developing a defensible exam. There are several ways that their input is required. Here is a list from highest involvement/responsibility to lowest: Serving on the Certification Committee (if relevant) to decide important things like eligibility pathways Serving on panels for psychometric steps like Job […]
One of the most cliche phrases associated with assessment is “teaching to the test.” I’ve always hated this phrase, because it is only used in a derogatory matter, almost always by people who do not understand the basics of assessment and psychometrics. I recently saw it mentioned in this article on PISA, and that was […]
ASC has been empowering organizations to develop better assessments since 1979. Curious as to how things were back then? Below is a copy of our newsletter from 1988, long before the days of sharing news via email and social media! Our platform at the time was named MICROCAT. This later became modernized to FastTest PC […]
Item response theory is the predominant psychometric paradigm for mid or large scale assessment. As noted in my introductory blog post, it is actually a family of models. In this post, we discuss the two parameter IRT model (IRT 2PL). Consider the following 3PL equation (simplified from Hambleton & Swaminathan, 1985, Eq. 3.3). The IRT […]
Item response theory (IRT) is an extremely powerful psychometric paradigm that addresses many of the inadequacies of classical test theory (CTT). If you are new to the topic, there is a broad intro here, where you will learn that IRT is actually a family of mathematical models rather than one specific one. Today, I’m talking […]
Classical test theory is a century-old paradigm for psychometrics – using quantitative and scientific processes to develop and analyze assessments to improve their quality. (Nobody likes unfair tests!) The most basic and frequently used item statistic from classical test theory is the P-value. It is usually called item difficulty but is sometimes called item facility, […]
Cutscores set with classical test theory, such as the modified-Angoff method, Nedelsky, or Ebel methods, are easy to implement when the test is scored classically. The Angoff cutscore approach is legally defensible and meets international standards such as AERA/APA/NCME, ISO 17024, and NCCA. It also has the benefit that it does not require the test […]