How to write and analyze open-ended survey questions

Open-ended questions give you what closed questions cannot: reasons you did not think to list, the words people actually use, and the occasional answer that changes how you read the whole survey. They also cost more on both sides. Respondents have to compose an answer, and you have to read and make sense of every one.

Used well, a few open questions make a survey far more useful. Used everywhere, they make it longer, lose you respondents, and leave a pile of text nobody gets around to analyzing. Here is how to use them well, from the wording through to the report.

When an open question earns its place

Use one when you genuinely do not know the range of answers, or when the reason matters more than the count:

• The reason behind a score. After a rating or a Net Promoter Score question, “What is the main reason for your score?” turns a number into something you can act on.
• Exploring before you write a list. If you are not sure what options a multiple-choice question needs, ask it open in a pilot, then build the options from what people said.
• Language you want to reuse. How customers describe a problem is often clearer than how the team describes it.
• Room for what you missed. One question near the end, such as “Is there anything else we should know?”, catches issues the questionnaire did not anticipate.

Do not use an open question to count something you could list. “Which tools do you use?” as free text gives you twenty spellings of the same five products to clean up, where a checkbox question would have counted them for you. The guide to types of survey questions covers which type fits which job.

Word it so the answer is usable

Vague prompts get vague answers. “Any comments?” invites anything at all; “What is one thing we could have done better during your visit?” gets something specific. A few habits help:

• Ask about one topic, and about a concrete, recent experience rather than opinions in general.
• Say what kind of answer you want: “one thing”, “the main reason”, “in a sentence or two”.
• Match the box to the answer. A single-line field says a few words will do; a paragraph box says you are ready to read more.
• Keep open questions optional unless you truly need every answer. A required essay box is where a tired respondent either quits or types filler to get past it.

Keep the number small

Each open question takes longer to answer than a choice question, and each is a point where someone who was already flagging may give up. One or two well-placed open questions do more than five scattered through the survey. If you have several, put the most important early and make the rest optional. The article on how long a survey should be covers budgeting that time.

Placement matters as much as number. Put an open question straight after the closed question it explains, while the reasoning is still fresh, and keep the general “anything else?” question for the very end, where it catches what the rest of the survey missed.

Read before you count

Start the analysis by reading. Take a sample of responses, or all of them if there are only a few dozen, and read without coding anything. Note the themes you keep seeing and anything that surprises you. This first pass stops you from forcing answers into categories you decided on before you saw them.

Build a simple codebook

A codebook is a short list of themes, each with a one-line definition and an example answer. For answers to “What is the main reason for your score?” after a delivery, the first draft might look like this:

• Speed: how long delivery or replies took. “Arrived a day later than promised.”
• Condition: the state the order arrived in. “Box was crushed but the item was fine.”
• Communication: updates, tracking and replies. “No tracking link until it was already here.”
• Price: the cost of the product or of delivery. “Shipping cost more than I expected.”
• Unclear: answers you cannot interpret. “See above.”

Write the codebook from your first read, then apply it:

1. 1Give each response every code that fits. One answer can mention both price and speed.
2. 2Keep an “Other” code, and an “Unclear” code for answers you cannot interpret. They stop weak matches from inflating real themes.
3. 3Revise as you go. When a code keeps needing exceptions, split it; when two codes always appear together, merge them. Then recode the early responses with the final list.
4. 4Stop adding codes when new responses stop producing new themes.

A spreadsheet is enough: one row per response, one column per code. Plumeform exports results to CSV on every plan, with one column per question, so you can code each answer next to the rest of that person's responses and later compare themes between groups, such as new and long-standing customers.

Count carefully, quote honestly

Report a theme as the number or share of respondents who mentioned it, with the base beside it: for example, “Delivery times came up in 14 of 52 answers.” Keep two limits in mind. A theme raised by a handful of people is not a percentage of everyone. And an open question measures what came to mind, not everything people think: someone who did not mention price may still care about it.

Themes become more useful when you compare groups. The share of detractors who mention delivery speed, set beside the share of promoters who do, says more than either number alone, as long as both groups are large enough for the difference to mean something. The significance test calculator compares two shares like these from their counts.

Choose quotes that represent a theme rather than the most dramatic line, and say how common the view is. Remove names, places and details that could identify someone, especially in small groups. De-identification covers what counts as identifying, which is more than names and email addresses.

Check your own reading

Coding is interpretation, and interpretation drifts toward what you expected to find. Two checks keep it honest. Ask a colleague to code a sample independently and compare the results; where you disagree, sharpen the code's definition. And before you write the summary, look deliberately for answers that contradict it.

Where software helps, and where it does not

In Plumeform's results, each open question shows its most common words and its latest answers, and the Responses view lets you read every answer alongside the rest of that response. The guide to reading results explains what each view counts.

On Pro, Summarize with AI proposes a short summary and a handful of themes, each with an approximate number of mentions and an example quote, drawn from up to the 400 most recent answers kept in your results. The AI only sees those answers when you ask for a summary.

Treat an AI summary like a colleague's first pass: a quick way to see the shape of the answers, not a finished analysis. The mention counts are estimates, and a theme stated confidently can rest on a few answers. Read enough responses to confirm each theme before it goes in a report, and take your quotes from the responses themselves.