How to write attention checks people actually fail honestly.

An attention check is a question with a verifiably correct answer for anyone who is reading. It does not measure opinion, memory, or intelligence. It measures exactly one thing: was this participant processing the words on the screen. That narrow job description is the test for every check you write. If a careful, honest, average reader could plausibly get it wrong, it is not an attention check, it is a trap, and traps generate noise of their own.

Checks work best as one signal among several. Response time, answer patterns like straight-lining down a matrix, and open-text quality all triangulate the same question. A response that fails an attention check and finished in a third of the median time is an easy call. A response that failed one check but looks normal everywhere else deserves more benefit of the doubt.

1. Instructed response. The workhorse, and the one to reach for first. You tell the participant exactly what to pick, inside a question that looks like its neighbors:

It is unambiguous, format-matched, and takes a reader two seconds. There is no honest way to fail it, which is what makes a failure meaningful.

2. Infrequency items.Statements almost nobody can truthfully endorse, dropped into an agreement scale: “I have never used the internet” in an online survey, or “I was born on the planet Neptune.” Useful in long matrix batteries where an instructed item would stick out. Keep them literal; humor invites playful agreement.

3. Bogus knowledge items. A factual question with one obviously correct answer, like which of four words is a color. Fine as a light-touch check, but weaker than instructed response because a distracted person can still answer on autopilot.

4. Open-ended checks.Ask for a short written answer to something concrete from the study itself, such as “in one sentence, what was the scenario about?” The most informative and the most work to grade. Best reserved for high-stakes studies where you are already reading the text answers.

Flag, do not delete. Keep every response and mark the ones that failed, then apply your pre-written exclusion rule at analysis time. Reviewers increasingly expect exactly this: the rule stated in advance, the number of exclusions reported, and ideally the results shown with and without the excluded rows.

On paid panels, be slower to reject than to exclude. Excluding a row from analysis costs the participant nothing; rejecting their submission costs them money and you reputation. Prolific's rules only permit rejection over checks that are unambiguous and fair, and generally expect repeated failure. When in doubt, pay, and filter the row out of the data anyway.

In Plumeform, any multiple choice question becomes an instructed response check with one toggle: mark the correct option, and wrong picks flag the response in results and in every export. Speeders and straight-liners are flagged automatically alongside it, and a one-click filter shows the clean subset. The mechanics are on the researcher tour.