7 min read
Randomize your question order.
The order of questions and answer options can affect responses. Randomizing that order spreads the effect across different items. This guide explains what to shuffle, what should stay fixed, and how to report the setup.
What order does to answers
Two common effects are Position bias: in a visual list, the first options get disproportionate attention and disproportionate picks, an effect that grows with list length and shrinks with participant motivation. Question order effects: an early question changes the frame for later ones. Ask people to rate their satisfaction with life after asking about their dating life and the two answers correlate far more than when the order flips. The general problem is that every question is also context for the next one.
You cannot remove these effects for an individual participant. What you can do is stop them from accumulating in one direction: show different participants different orders, and the position advantage distributes evenly instead of always subsidizing option A.
What to randomize
Option order, whenever the options have no natural order. Brand lists, feature lists, candidate names, anything where “first” is an accident of authoring:
Which of these tools have you used in the past year?
- Typeform
- Google Forms
- SurveyMonkey
- Qualtrics
- None of these (pinned last)
Question order within a block, when several questions measure the same construct and none depends on another. Rotating them spreads fatigue evenly: the tenth item in a battery always gets sloppier answers than the second, so make sure a different item is tenth for every participant.
Block order, when whole sections are independent. Showing half your sample the pricing block before the brand block, and half the reverse, is also the cleanest way to measure whether order matters at all: compare the halves.
What to leave alone
Do
- Shuffle unordered choice options, per participant.
- Pin escape options: Other, None of the above, Prefer not to say stay last.
- Rotate items inside a same-construct battery.
- Keep the order stable for one participant across re-renders and back-navigation.
- Note what you randomized in your write-up. One sentence is enough.
Don't
- Shuffle ordinal scales. Strongly disagree to Strongly agree is an order, not a list.
- Randomize across a dependency: a question that references an earlier answer must come after it.
- Shuffle screening questions away from the front. They gate everything behind them.
- Re-shuffle on every page refresh. Same person, same order, always.
- Randomize so aggressively that the survey loses narrative flow. Confused participants quit.
Doing it, and reporting it
A repeatable shuffle uses seeding. The shuffle must be random between participants and frozen within one: if someone navigates back a page, the options must not rearrange under them, both because it is disorienting and because their first read of the list is the one that shaped their answer.
In Plumeform, option shuffle is a per-question toggle with keep-last pinning, and question order within a page has its own toggle. The order is seeded per participant and holds when they go back or reload the page, and the same controls sit next to the attention check and quota settings on the research toolkit.
In your methods section, state what you shuffled, whether the order changed per participant, and which options stayed fixed.
Quick answers
Does randomization remove bias?
Should I randomize a Likert scale's options?
What about 'Other' and 'None of the above'?
How do I report randomization in a paper?
Keep reading: How to write attention check questions · Random assignment in survey experiments · Academic research · Likert scales
Start
Put it into practice.
Start with 100 responses a month free. Pro includes 10,000 responses and the advanced research controls described in these guides.