Comparative Judgment

Share / assign

Why it works

Scoring complex work (essays, reasoning, open problems, portfolios) against rubrics is often unreliable, different scorers disagree, and some qualities resist being broken into rubric criteria. Comparative judgment takes a different approach rooted in a robust finding: humans are much more reliable at comparative decisions (which of these two is better?) than absolute ones (what score is this?). By having judges make many pairwise comparisons of student work, the results can be combined statistically into a reliable rank order and scores, capturing holistic quality that rubrics miss. For teachers, comparative judgment can be a powerful way to assess complex work reliably, moderate judgments across a team, and build shared understanding of quality, though it is a different model than rubric scoring and suits some purposes better than others.

The research: Research on comparative judgment in assessment.

The run-of-show

Choose your slot. The agenda, timings, and length update to match.

The core activity: assess complex work by comparing

Teachers leave understanding how to assess complex work by comparison.

Pick complex work that rubrics struggle to score reliably (writing, reasoning).
Have judges compare pairs (which is better?) across many comparisons.
Combine the comparisons into a rank order and shared understanding of quality.
Use it to assess reliably and build shared standards.

Judge comparisons, not absolute scoresPeople compare more reliably than they score. For complex, holistic work, comparative judgment, many pairwise comparisons, assesses quality rubrics struggle to capture, and builds shared standards.

Facilitator notes

Use for complex, holistic work, and compare pairs rather than score each.
Combine comparisons into a rank.
Build shared quality understanding.

Forcing holistic quality into rubric boxes can be unreliableSome complex qualities resist rubric criteria and score unreliably. Comparative judgment, comparing pairs, can assess them more reliably where it fits.

Adapt it

Elementary and secondary: compare work, build shared standards.
Team use: moderation, shared understanding.
All settings: compare pairs for complex work.

Participant handout

One page for every teacher. Print it, or save it as a PDF.

K12 Academics · Professional Learning

Comparative Judgment: compare, don't just score

Reliable holistic assessment: compare pairs of work, combine into a rank, and build shared understanding of quality.

The complex work that rubrics struggle to score:
How judges compare pairs (which is better?):
How comparisons combine into a rank and shared understanding:
When comparative judgment fits better than rubrics:
A use to try:

Make it stick

Compare, don't just score:

Assess complex work by comparing.
Build shared standards.
Bring a comparative-judgment use to the next PLC.

Common questions

What is comparative judgment?

Assessing by repeatedly comparing pairs of student work (which is better?) rather than scoring each against a rubric. Many comparisons are combined statistically into a reliable rank and scores. It leverages that people judge comparisons more reliably than absolute scores.

Why use it instead of rubrics?

For complex, holistic qualities (writing, reasoning) that rubrics struggle to capture and where rubric scoring is unreliable, comparative judgment can produce more reliable results and shared understanding of quality.

When is it worth it?

For assessing complex work reliably, moderating across a team, or building shared quality standards. It is a different model than rubric scoring and suits some purposes (holistic quality) better than others.