Skip to content

Paired comparison

Also called pairwise comparison, head-to-head comparison, Bradley-Terry

Asking which of two options someone prefers, over many pairs, and turning the wins into a score for every option.

Choosing between two things is the easiest judgment there is, and it never asks anyone to hold a whole list in mind. Over enough pairs, the wins and losses place every option on one scale. A Bradley-Terry model does this properly: it gives each option a strength such that the chance of one beating another depends on their two strengths, and the strengths can be read as shares of preference that sum to 100.

The number of pairs grows fast: 6 options make 15 pairs, 10 make 45. Past about thirty, people stop reading. Showing each respondent a random subset keeps the task short while every pair is still answered across the sample.

Allowing a tie (about the same) is honest when two options really are close, and a tie counts half a win to each.

Where it goes wrong

Raw win rates when respondents saw different pairs. An option that happened to meet the strongest rivals looks weaker than it is; a model that accounts for the opponents does not have that problem.

See also

Ordinal data
Answers with a known order but unknown spacing, which is what almost every rating question produces.
Random assignment
Allocating participants to conditions by chance, which is what lets a difference between groups be read as an effect.
Sample size
The number of responses in your analysis. It affects the precision of your estimates.

All 35 terms

Start

Put it into practice in your next survey.