GUIDES9 MIN READ
The responses that quietly wreck your data.
Every open survey collects some junk: people racing for an incentive, bots filling fields, one enthusiast submitting five times, a respondent clicking straight down the middle column. None of it announces itself, and all of it drags your averages toward the noise. The fix is not suspicion, it is instrumentation: measure the few things that separate a careful response from a careless one, flag rather than guess, and decide with a rule you wrote in advance. This guide covers the four signatures and how to design them out.
01Speeders: finished before they read
The single most useful quality signal is time. A person who genuinely reads and answers a ten-minute survey cannot finish it in ninety seconds; someone chasing a payout can. Record the duration of every response and you can see the racers as a cluster far below everyone else.
Fixed thresholds are fragile because they depend on your survey, so compute the cutoff from your own data: take the median completion time of real responses and flag anything below roughly a third of it. Plumeform does this automatically once a handful of timed responses are in, and marks the speeders for you.
Median completion time: 6m 40s
- Most responses: 4 to 11 minutes — normal spread
- A tight cluster at 40 to 70 seconds — flag these
- One response at 38 minutes — a distracted tab, not junk
02Straight-liners: the same answer, all the way down
Give someone a matrix of ten statements on a five-point scale and a careless responder will pick one column and ride it to the bottom. It is fast, it looks complete, and it is meaningless. This is straight-lining, and it is invisible in a summary chart because the answers are perfectly valid values.
You catch it by looking at the variance within a single response: a real person disagrees with some items and agrees with others, so their answers vary. Zero variance across a long battery, especially a battery that mixes positively and negatively worded items, is the tell. Reverse-wording a couple of items in each matrix makes straight-lining obvious, because a genuine reader has to switch sides.
03Bots and duplicates: the same response, twice
Automated and repeat submissions cluster. The same option pattern, the same suspiciously round timing, often the same referrer, arriving in a burst. Open-text answers are the giveaway: generic, off-topic, or copied verbatim across “different” people.
DO
- Use a one-response-per-browser limit for casual, uncompensated sharing.
- Pass a unique participant ID from your panel and reject repeat IDs.
- Cap total responses so a leaked link can't run up an unlimited bill.
- Add a short open-text question; junk answers are easy to spot by eye.
DON'T
- Don't rely on IP alone; shared networks and VPNs make it noisy.
- Don't put the incentive amount in the invite; it recruits the wrong people.
- Don't assume a completed response is a real one. Completeness is not quality.
04Attention checks: was anyone reading?
The most direct quality signal is to simply ask. An instructed-response item (“to show you are reading, select Disagree”) has a verifiably correct answer for anyone processing the words, and none for someone skimming. One or two per survey, written fairly, will separate the readers from the clickers without annoying honest participants.
Attention checks pair with everything above: a response that speeds, straight-lines, and fails a check is not a borderline call. We keep a whole guide on writing checks that people fail honestly.
05Flag, don't delete on a hunch
The cardinal rule of data cleaning: decide your exclusion criteria before you see the results. Deleting responses after you have looked at how they move your numbers is how honest researchers fool themselves. Keep everything, mark the flagged rows, and set them aside by a rule, not a vibe.
In Plumeform every response carries its quality flags, and a single filter hides the flagged ones so you can see your results clean and dirty side by side. If the story is the same either way, you can report it with confidence; if it changes, that is worth knowing before you present it.
Quick answers
How do I know if survey responses are fake or careless?+
Look for four signatures. Speeders finish far faster than any careful reader could. Straight-liners pick the same column down a whole matrix. Failed attention checks mean the words were not being read. And duplicate or bot submissions cluster: the same answers, the same timing, often the same source. No single flag is proof, but two or more on one response is a strong signal to set it aside.
What is a speeder, and how fast is too fast?+
A speeder is someone who completes the survey too quickly to have genuinely read it. A common rule is anything under roughly one third to one half of the median completion time. Because that threshold depends on your own survey, it is better computed from your data than fixed in advance: take the median duration of real responses and flag the ones far below it.
Should I delete low-quality responses?+
Flag first, decide later, and write the rule down before you look at results. Deleting on a hunch after you have seen the answers invites bias. The defensible pattern is to keep every response, mark the flagged ones, and report your data both with and without them so a reader can see the cleaning didn't change your conclusion.
How do I stop one person submitting a survey many times?+
For casual sharing, a one-response-per-browser setting stops the honest double-submit. For anything with an incentive, add a unique participant ID from your panel and reject duplicates on it, cap total responses, and use screening plus quotas so a filled segment stops accepting. Nothing is bulletproof against a determined bad actor, but layered friction removes the easy abuse.
Keep reading: Writing attention checks · How many responses you need
Put it into practice.
250 responses a month free. Every control in these guides is built in.