Psychometric Analysis of Item and Test Performance

An absolutely essential step in developing a defensible test is statistical analysis of response data.  This step provides detailed feedback on what is working well in the test, what is not, and why.  This drives continual improvement, but also provides documentation of validity evidence.

Our psychometricians will need response data from examinees as well as metadata on each item.  Once you provide these, we clean the data then run it through our specially designed software.  This software drafts the report for you in seconds, which means we can provide this service at a fraction of the cost that large vendors will charge you.  Though of course, we first go through the results with a fine-toothed comb and write up our recommendations.

Well, suppose that we analyze a test question whose correct answer is A, but find that only 40% of your candidates got it correct (item difficulty). Meanwhile, just as many examinees answered B – and it happened to be smarter examinees. This item would be flagged and discussed with your experts, who might point out something like “Only candidates who have completed a special training course would know how to answer this question.” Even though the answer is technically correct, the statistics have helped flag an issue with the item, and it might be retired or rewritten.

Have psychometric talent in-house but want to make them more effective in conducting these analyses? Iteman and Xcalibre provide user-friendly interfaces and automatically create Microsoft Word reports with all the tables and graphs already embedded for you. Or, start using our best-in-class platform Assess.ai and have the reports ready-made for you!