Item analysis in assessment: plain words, not jargon
Item analysis in assessment, in plain words: what difficulty, discrimination and distractor analysis tell a QA office, and how each figure traces to the work.
Item analysis in assessment is the review of how each question performed once candidates have answered it: how hard it was, whether it separated stronger candidates from weaker ones, and how its wrong options were chosen. It tells a QA office whether an item did its job. On NUADU, item quality statistics per item, distractor analysis and patterns by rubric criterion, by standard and by item are reported in the language of academic quality, not psychometric jargon. This post explains each statistic in plain words, and what happens to an item that falls short.
What is item analysis in assessment?
Item analysis in assessment is the statistical review of individual items after a sitting, used to decide whether each one should be kept, revised or retired. It rests on two ideas, difficulty and discrimination, plus distractor analysis for multiple-choice items. Item analysis in education uses the same method for a classroom test and a high-stakes paper.
The questions behind it are plain. Was the item too hard for this cohort, or too easy to tell anyone apart? Did it separate the candidates who knew the material from those who did not? Did a wrong option attract the strongest candidates? Does the paper as a whole measure what the module says it measures?
Those are academic-quality questions, and they deserve answers in the same language. Item statistics too often arrive as a column of decimals with no sentence attached, and the head of assessment ends up translating for the QA office. In NUADU's assessment analytics, the numbers are still there; they arrive with their meaning attached, and programme decisions stay with educators.
Item quality is evidenced, not assumed.
What is the difference between item difficulty and item discrimination?
Item difficulty is the share of candidates who answered an item correctly. Item discrimination is whether candidates who did well on the paper overall were more likely to get the item right than candidates who did not. Difficulty describes how hard the item was for this cohort; discrimination describes whether it pulled in the same direction as the rest of the paper.
Difficulty needs context. An item nearly everyone answers correctly tells you little about differences within a cohort. An item nearly nobody answers correctly may be testing something that was not taught, or may be badly written. Neither is automatically wrong — an opening item is allowed to be easy — but each should be a choice the author makes knowingly.
Discrimination needs attention whenever it is weak or reversed. If stronger candidates tended to miss an item that weaker candidates got right, the item is working against the paper. The usual causes are an ambiguous stem, a key that is arguably wrong, or a wrong option that is more defensible than the intended answer.
So what is a good discrimination index? A positive one: candidates who scored well overall were more likely to answer correctly. An index near zero means the item barely separates candidates; a negative index means it separates them the wrong way and should be reviewed first. Some reports compare top and bottom scorers, others correlate the item with the total score — both answer the same question. Where the line between acceptable and needs review falls is a matter for your assessment policy, not for the statistic.
What is distractor analysis in item analysis?
Distractor analysis is the part of item analysis that looks at the wrong options in a multiple-choice item: how many candidates chose each one, and which candidates chose it. It shows whether every distractor is doing work, and whether one of them is trapping candidates who otherwise know the material.
Two patterns matter most. A distractor that almost nobody chooses is doing no work and can be replaced; the item is easier than it looks. A distractor that attracts the strongest candidates is a warning about the item, not about them — often that option is defensible, or the stem points towards it.
On NUADU, objective formats such as MCQ, matching, ordering and code are scored deterministically by the rules you configure, and distractor analysis is available for item review. Because objective scoring is deterministic, what the analysis shows is about the item. Where objective formats end and rubric-marked ones begin is set out in objective vs subjective questions.
How do you interpret an item analysis report?
Read it from the item outwards. Start with items whose discrimination is weak or reversed, then look at their distractors, then at the patterns by criterion and standard — and before acting on any figure, open the responses behind it. A report a QA office can defend is one where every number leads back to the work that produced it.
On NUADU that route is built in. Where a number is shown, it can be traced to the rubric criterion, the item, the marker and the candidate's response. Pattern analysis runs by rubric criterion, by standard and by item, so one weakness can be read three ways: as a teaching question (which criterion), a curriculum question (which standard) and an assessment-design question (which item). Trend analysis adds a fourth reading: whether changes in teaching are being reflected in evidence from one sitting to the next.
Constructed-response items need one more column, because their scores pass through a marker. Inter-rater agreement is visible to QA, and every override is logged; how AI fits an existing second-marking policy is a separate question for the same office.
Reports also have to leave the platform. Saved reports from the drag-and-drop builder can be sent on a schedule to programme leads or QA, exported to PDF or to CSV and Excel, or fed through the REST API. A figure in the academic board's PDF is the same figure a QA officer can trace inside the platform.
What happens to a weak item?
It goes back to its author, not into a bin. A weak distractor or a reversed item returns through the review gate of the item bank: a named approver decides whether to edit or retire it, and the change enters the item's change history under that approver's name. No statistic retires an item on its own.
That loop is what turns item analysis from a report into a better paper. The next sitting uses the revised item, and the record shows what changed and who approved it. Item quality statistics describe items; they never label an individual learner or predict a candidate's outcome.
The loop matters beyond the next sitting. Where stakes require parallel test forms, Assign draws them from item banking with calibration statistics. Two forms can only be called comparable once the quality of each item is known — and that knowledge is the evidence a QA office holds.