How to write unbiased survey questions

A biased question rarely looks biased to the person who wrote it. It reads naturally because it asks the question the way its author already thinks about the answer. The respondent does not share that starting point, gets nudged toward it anyway, and the result looks like a finding.

More responses do not fix this. A leading question asked of a thousand people gives you a precise measurement of your wording. The only fix is in the draft, before launch, and most of it comes down to knowing what to look for. Here is what to look for, with a rewrite for each.

Start with the decision, not the question

Many biased questions begin as unclear goals. When you cannot say what you will do differently depending on the answer, the wording tends to drift toward the answer you are hoping for. Before writing a question, write down the decision it informs: keep the feature or cut it, change the onboarding email or leave it, run the event again next year or not.

Then write the question that could change your mind. If you are deciding whether to keep a feature, you need to hear from the people who would not miss it, and the wording has to make that answer as easy to give as the other one.

Ask one thing at a time

A double-barrelled question asks about two things and accepts one answer. “How satisfied are you with the price and quality of your order?” leaves someone who loved the quality but found it overpriced with no honest option, and whichever they pick, you cannot tell which half they meant.

The giveaway is usually an “and” or an “or” joining two things being rated. Split them:

• “How satisfied are you with the price of your order?”
• “How satisfied are you with the quality of your order?”

The same problem hides in answer options (“Affordable and easy to use”) and in the rows of a rating grid, where a shared scale makes a compound statement easy to miss.

Take the answer out of the question

A leading question points at the answer it expects. It takes a few common forms:

• Loaded words. “How helpful was our award-winning support team?” asks people to weigh a reputation before they weigh their own experience. Ask “How helpful or unhelpful was the support you received?”
• Agreement bait. “Don't you agree that meetings run too long?” makes agreeing the polite answer. Ask “How do you feel about the length of your team's meetings?” with options from “Much too short” to “Much too long”.
• Assumed behavior. “How often do you use the export feature?” assumes people use it at all. Ask whether they have used it first, or make “I have not used it” the first option.
• Social pressure. “Our team worked hard on the new dashboard. How useful is it?” tells people which answer would be kind. Leave the preamble out.

The test for all of these is simple: could someone who holds the opposite view answer comfortably, in words the question offers them?

Balance the answer scale

Scales lead too. If the options run “Extremely satisfied”, “Very satisfied”, “Satisfied”, “Somewhat satisfied” and “Not satisfied”, four of the five answers are some kind of satisfied, and the scale has taken a side before anyone answers.

Give the scale the same number of steps on each side of the middle, label both sides in matching words (satisfied and dissatisfied, not satisfied and poor), and name both ends in the question itself: “How satisfied or dissatisfied were you with…”. Whether to offer a middle option is a separate decision, covered in the Likert scale guide. The Likert scale generator gives you balanced labels for agreement, satisfaction, frequency, importance and other common scales.

Make the options fit everyone, once

Answer options bias results when some people cannot find themselves in the list. Two rules cover most cases:

• Options should not overlap. Age bands of 18–25 and 25–34 give every 25-year-old two answers. Use 18–24 and 25–34.
• Options should cover everyone. Add “Other”, “None of these” or “I have not done this” wherever a real person might need it, and “Prefer not to say” to anything personal.

A missing option rarely makes people skip a question. They pick the nearest thing, and your data now says something it should not. In Plumeform, a multiple-choice question can include an Other option with a text box, so the answers you did not anticipate arrive in the respondent's own words instead of as a forced choice.

Be wary of agree and disagree statements

Batteries of statements rated from “Strongly disagree” to “Strongly agree” are quick to write and quick to answer, which is part of the problem. Some respondents lean toward agreeing with whatever is put to them, especially when they are tired or unsure. This tendency is called acquiescence, and it inflates agreement for reasons that have nothing to do with your topic.

Where you can, ask about the thing directly. Instead of asking for agreement with “The onboarding was clear”, ask “How clear or unclear was the onboarding?” on a scale that names both ends. You learn the same thing without first asking anyone to agree with you.

Lower the cost of an honest answer

Some honest answers are awkward: how often someone exercises, whether they read the policy, whether they finished the training. Social desirability bias rounds those answers toward the respectable one. Three habits help:

• Ask about a specific, recent period. “In the past week, on how many days did you…” is easier to answer truthfully than “How often do you usually…”.
• Normalize the less flattering answer. A short preface such as “Plenty of people do not get to this every week” makes it easier to say so.
• Make anonymity real. Do not collect a name or email address you do not need, and never ask a sensitive question straight after one.

Plumeform respondents never sign in, and a survey only collects an email address if you add a question asking for one, so what you know about each respondent is decided by your questionnaire.

Replace vague words with time frames

Words like “often”, “regularly” and “recently” mean different things to different people, so identical answers carry different amounts. Ask for a count within a time frame (“How many times in the last month…”) or offer ranges people can place themselves in. Absolutes cause the opposite problem. Few people can honestly say they “always” or “never” do something, so a statement built on one measures how literally people read it as much as what they do.

Mind the order

Earlier questions set the frame for later ones. Ask for an overall rating before the questions about specific problems, not after you have reminded people of everything that went wrong. Put general questions before specific ones, and sensitive ones after respondents have had a chance to settle in.

When the order of options is arbitrary, such as a list of brands, the ones near the top can get picked more often simply for being first. Shuffling them for each respondent spreads that effect evenly; the survey randomization guide explains what to shuffle and what to leave alone. In Plumeform, option and question shuffling are part of the research tools on Pro.

Test the wording before it goes out

You are the person least able to spot bias in your own questions, because you already know what they mean. Four checks help:

1. 1Read each question aloud. If you stumble or reread, so will respondents.
2. 2Ask two or three people outside the project to say, in their own words, what each question asks. Where their answers differ, the question is ambiguous.
3. 3Paste questions into the free question readability checker, which flags reading level, long sentences, double-barrelled asks, negations and vague quantifiers.
4. 4Read the pre-flight notes in the Plumeform builder. As you edit, they point at questions that may lead, may ask two things at once or use absolutes, and they give the reading grade of your wording. They are prompts to look again, not verdicts. On Pro, an AI review of the wording suggests a rewrite for each question it flags.

Wording is one part of getting a survey ready. The pre-launch checklist covers the rest, from routes and phones to the data itself.