Skip to content

Selection bias

Error introduced by how people came to be in your sample, when that process is related to what you are measuring.

Selection bias is upstream of everything else. A link posted to your own newsletter reaches people who already like you. A survey advertised as being about cycling gets answered by cyclists. A comparison between people who used a feature and people who did not measures the choosing as much as the feature.

It is not solved by weighting, sample size or a careful questionnaire, because the problem is in who had the chance to answer. It is reduced by choosing a sampling frame that covers the population you want to describe, and by assigning people to groups at random when you are comparing anything.

Describe recruitment in your report, for example: “Respondents were recruited from our mailing list.” This helps readers assess who the findings apply to.

Where it goes wrong

Comparing self-selected groups and reporting the difference as an effect. Randomise the assignment, or describe the result as an association.

See also

Nonresponse bias
The error that arrives when the people who did not answer differ from the people who did, in a way that matters to your question.
Sampling frame
The actual list your sample is drawn from, which is rarely the same thing as the population you want to describe.
Random assignment
Allocating participants to conditions by chance, which is what lets a difference between groups be read as an effect.

All 35 terms

Start

Put it into practice in your next survey.