A/B experiments inside a survey.

Random assignment is doing one job: making your groups exchangeable, meaning that differences other than the manipulation are balanced in expectation. Preserve each participant's assignment throughout the survey.

The common DIY approaches leak. Assigning by day (“Monday participants get version A”) confounds condition with time and whoever is online on Mondays. Publishing two survey links confounds it with however the links were distributed — and lets one link get shared more than the other. Letting the browser pick with Math.random() on page load re-rolls on every refresh, so a participant who reloads can land in a second condition, and a curious one can farm versions.

Use one link, assignment made server-side at the moment the survey loads, sticky for that participant's session, recorded with the response. The participant can refresh, go back, or resume; the condition stays put and the row in your data says which arm they saw.

Assignment is only half the mechanism; the survey has to change based on it. The standard tool is the survey's own branching logic with the condition as a variable: if condition is “treatment”, show the treatment vignette page; if “control”, show the control page; everyone converges on the same outcome questions afterward.

Keep the arms similar outside the manipulation, including page count, outcome questions and length. Extra time in one arm, for example, can introduce fatigue that affects the comparison.

In Plumeform this is one settings block: name 2–6 arms, and each new respondent is assigned server-side to the arm with the fewest completed responses so far, ties broken at random, so dropout cannot unbalance the groups. The arm is available to logic rules as Condition arm (branch pages and questions on it like any answer). It survives refreshes and return visits in the same browser, and a spoofed ?condition= on the link is ignored: the server keeps the assignment it made.

At analysis time you need, for every response, which condition it was assigned — not inferred from which questions have answers, but recorded as its own column. Inferring breaks the moment two arms share most questions, and it silently miscodes partial responses. Insist on a condition column in results and in every export, and check the split early: with 2 arms and a few hundred participants, anything wildly far from 50/50 means the plumbing is broken, not that chance is being quirky.

Watch differential dropout as data comes in. If the treatment arm abandons at twice the control rate, your completed sample is no longer exchangeable — the treatment filtered who finished. Report completion rates per arm alongside the result; a large gap is a finding about the manipulation, and a threat to the rest.

When it works, the write-up is one sentence: “Participants were assigned server-side at survey load to one of two conditions (treatment n = 103, control n = 97), each to the condition with fewer completed responses at that moment, ties broken at random; assignment was recorded with each response and kept on reload; completion rates did not differ by arm.” Every clause in that sentence is something a reviewer checks, and every one of them was decided by the plumbing you chose before launch.