GUIDES7 MIN READ
Randomization, explained.
The first option in a list gets picked more. Questions asked early shape the answers to questions asked late. None of this is a flaw in your participants; it is how reading works, and it quietly skews any survey that shows everyone the same order. Randomization is the cheap, boring, extremely effective fix. This guide covers what to shuffle, what to leave alone, and how to write it up.
01What order does to answers
Two effects do most of the damage. Position bias: in a visual list, the first options get disproportionate attention and disproportionate picks, an effect that grows with list length and shrinks with participant motivation. Question order effects: an early question changes the frame for later ones. Ask people to rate their satisfaction with life after asking about their dating life and the two answers correlate far more than when the order flips. The general problem is that every question is also context for the next one.
You cannot remove these effects for an individual participant. What you can do is stop them from accumulating in one direction: show different participants different orders, and the position advantage distributes evenly instead of always subsidizing option A.
02What to randomize
Option order, whenever the options have no natural order. Brand lists, feature lists, candidate names, anything where “first” is an accident of authoring:
Which of these tools have you used in the past year?
- Typeform
- Google Forms
- SurveyMonkey
- Qualtrics
- None of these (pinned last)
Question order within a block, when several questions measure the same construct and none depends on another. Rotating them spreads fatigue evenly: the tenth item in a battery always gets sloppier answers than the second, so make sure a different item is tenth for every participant.
Block order, when whole sections are independent. Showing half your sample the pricing block before the brand block, and half the reverse, is also the cleanest way to measure whether order matters at all: compare the halves.
03What to leave alone
DO
- Shuffle unordered choice options, per participant.
- Pin escape options: Other, None of the above, Prefer not to say stay last.
- Rotate items inside a same-construct battery.
- Keep the order stable for one participant across re-renders and back-navigation.
- Note what you randomized in your write-up. One sentence is enough.
DON'T
- Shuffle ordinal scales. Strongly disagree to Strongly agree is an order, not a list.
- Randomize across a dependency: a question that references an earlier answer must come after it.
- Shuffle screening questions away from the front. They gate everything behind them.
- Re-shuffle on every page refresh. Same person, same order, always.
- Randomize so aggressively that the survey loses narrative flow. Confused participants quit.
04Doing it, and reporting it
The implementation detail that separates correct randomization from decorative randomization is seeding. The shuffle must be random between participants and frozen within one: if someone navigates back a page, the options must not rearrange under them, both because it is disorienting and because their first read of the list is the one that shaped their answer.
In Plumeform, option shuffle is a per-question toggle with keep-last pinning, and question order within a page has its own toggle. The order is seeded per participant, stable for their whole session, and the same controls sit next to the attention check and quota settings on the research toolkit.
When you write up results, one method sentence covers it: what was shuffled, at what level, and what was pinned. If a reviewer asks whether your effect is an order artifact, the answer you want to give is “order was randomized per participant,” not a paragraph of apology.
Quick answers
Does randomization remove bias?+
No, it spreads it evenly. Any single participant still experiences order effects; randomizing means those effects average out across the sample instead of piling onto whichever option you happened to list first. Your aggregate numbers get fairer; individual responses do not.
Should I randomize a Likert scale's options?+
Never. Scales from Strongly disagree to Strongly agree are ordinal; shuffling them destroys the meaning of the answers and infuriates participants. Randomize which ITEMS appear in what order, not the scale points themselves.
What about 'Other' and 'None of the above'?+
Pin them. Participants expect escape options at the end of a list, and shuffling them into the middle measurably changes how often they get picked. Every serious tool, Plumeform included, has a keep-last option for exactly this.
How do I report randomization in a paper?+
One sentence in the method section, stating what was randomized at what level. For example: option order was randomized per participant with the None option fixed last, and question order within each block was randomized. If you tested for order effects, report that too.
Keep reading: How to write attention check questions · Likert scale best practices · Plumeform for researchers
Put it into practice.
250 responses a month free. Every control in these guides is built in.