How to spot AI-generated survey responses and what to do about them

Some open-text answers read well and still tell you nothing. They are fluent, balanced and correctly spelled, but they could have been written by anyone about anything. Some respondents produce answers like these by pasting your question into an AI tool and pasting the reply back into your survey.

You cannot prove that from the text alone, and you do not need to. Your real question is narrower: is this answer usable evidence for what you are studying? Good question design makes generic text easy to recognize as unusable. Exclusion rules written down before you collect let you act on it fairly.

The signs that an open answer may be AI-written

No sign here is proof on its own. Each one is a reason to look more closely.

It is polished but does not answer your question

Take this question from a study about grocery delivery:

And this answer:

It is well written, but it never mentions a last order. It describes online grocery shopping in general. A real respondent who had no problems would more likely write "Nothing, it came on time." The generic answer covers every possibility so that it cannot be wrong, and that makes it useless to you.

It has the shape of an essay

Watch for answers that restate the question in the first sentence, list three balanced points, and close with "Overall" or "In summary." Watch too for hedges like "it depends on individual preferences" in reply to a question about the person's own preferences. Most people answering a survey on a phone write fragments, skip punctuation and jump straight to the point.

Many responses share the same skeleton

One tidy answer proves little. Twenty answers that open the same way, cover the same three points in the same order and use the same phrases are a pattern. When you read a batch of open answers together, notice whether they sound like different people or like variations on one template.

It does not fit the rest of the response

Compare the open answer with the person's other answers. A respondent who rated delivery 2 out of 10 and then wrote a neutral overview has given you two answers that do not agree. So has someone who wrote 120 careful words in less time than it takes to type them. Timing and consistency often tell you more than the wording does.

Why no single signal proves anything

Fluent writing is not suspicious in itself. Some respondents are simply good writers, some use grammar or translation tools because English is not their first language, and some draft answers in a notes app and paste them in. An exact duplicate can be innocent too, because "N/A", "No" and "Nothing to add" are common real answers.

The cost of getting this wrong is real. On a paid panel, a wrongful rejection takes money from someone who did the work and can count against their standing on the platform. In an employee or customer survey, an accusation can damage trust and future response rates. That is why the decision should rest on a combination of signals, applied the same way to everyone, and never on a hunch about tone.

It also helps to stop asking "did AI write this?" and start asking "does this answer tell me anything about this person's experience?" A vague, generic answer fails that test whoever wrote it. You can set it aside on those grounds without claiming to know how it was produced.

Ask questions that generic text cannot answer

The best defense is a question that only the respondent can answer. AI tools are good at general knowledge and bad at knowing what happened to one person last Tuesday. Ask about specific, recent, personal experience.

Before and after

Before:

After:

The first version invites an essay that any tool can write. The second asks for a memory, and a generic answer to it is obviously off target.

Ways to make questions more personal

• Anchor to an event. "The last time you…", "On your most recent visit…", "In the past week…"
• Ask for concrete details. What they did, what happened next, what they tried instead.
• Refer back to their own answers. "You said you stopped using the feature. What were you doing when you decided to stop?" On Pro, answer piping can repeat an earlier answer inside a later question, so each follow-up is tailored to the person.
• Ask for short answers. "One or two sentences is plenty" gives people permission to be brief. A short, plain answer is easier to give than to fake, and it leaves less room for padding.
• Ask fewer open questions. A survey with eight essay boxes invites shortcuts. Two well-aimed ones usually give you more. There is more on this in how to write and analyze open-ended survey questions.

You can also say plainly what you want: "We want your own words. There are no right answers, and spelling does not matter." That will not stop everyone, but it tells honest respondents that a rough answer is welcome.

Write your exclusion rules before you collect

Decide what will remove a response from your analysis before you see any data. Rules written afterward tend to drift toward whatever makes the results look cleaner. Rules written in advance can be applied by anyone on the team, and you can report them.

A workable rule set for open text might look like this:

1. 1An open answer that does not address the question asked, judged against a written definition such as "does not refer to the respondent's own experience when the question asks for it," counts as one flag.
2. 2An exact duplicate of another respondent's substantive answer counts as one flag. Short stock replies like "N/A" are exempt.
3. 3A completion time below a threshold you set in advance counts as one flag.
4. 4A failed attention check counts as one flag.
5. 5A response with two or more flags is excluded from analysis. A response with one flag is kept and noted.

Here is a worked example. Say 400 people complete your study. Eighteen have one flag and 9 have two or more. You analyze 391 responses and report the 9 exclusions with the rule that removed each. You keep the 18 single-flag responses and can check whether including or dropping them changes your conclusions.

Keep two decisions apart: excluding a response from analysis and refusing to pay for it. You can drop an unusable answer from your dataset and still pay the person. Reserve non-payment for clear cases defined in your rules, such as failing several checks, and follow the panel's own policies when you apply it. The wider set of quality checks is covered in the guide to low-quality responses.

What Plumeform flags, and what stays your decision

Plumeform marks responses for review and leaves the judgment to you. On every plan, exact duplicate open-text answers are flagged, and the results also flag responses that look rushed or careless, such as speeders and straight-liners. On Pro, bot protection also flags open answers that were mostly pasted rather than typed and browsers that report automation. Nothing is rejected on those flags alone.

That design fits the approach above. A paste flag is one signal among several, since some people do draft elsewhere and paste. Combine it with the answer's content, the timing and the attention checks according to your written rules. On Pro, you can save a clean-dataset rule so the same criteria apply every time you look at the data. Reading results and reviewing response quality covers where the flags appear.

For panel studies, participant IDs from the survey link are saved with each response and included in CSV exports, so your exclusion list maps directly to participants. On Pro, bonus and reject lists help you prepare payouts once you have applied your rules. The steps are in launching a panel study and preparing payouts.

Handling a flagged response

Open the whole response, not just the open answer. Read it alongside the closed questions and the timing, count the flags against your rules, and record the outcome and the reason. Responses you have already collected are always kept, so excluding one from analysis does not destroy it.

If you need to contact a participant, stay neutral and specific. Write "Your answer to question 6 did not describe your own experience, which our study required" rather than "You used AI." The first statement is about your criteria and can be checked. The second is a guess about how someone wrote, and you cannot back it up.

Then look at your design. If a large share of answers to one question come back generic, the question may be too abstract, too demanding or placed too late in a long survey. Rewrite it for the next wave so that only a real, recent experience can answer it.