Write attention checks that test attention.

An attention check is a question with a verifiably correct answer for anyone who is reading. It does not measure opinion, memory, or intelligence. It measures exactly one thing: was this participant processing the words on the screen. Review the wording if a careful reader could reasonably misunderstand it.

Checks work best as one signal among several. Response time, answer patterns like straight-lining down a matrix, and open-text quality all triangulate the same question. A response that fails an attention check and finished in a third of the median time warrants closer review. A single failed check with otherwise consistent answers is weaker evidence.

1. Instructed response. The workhorse, and the one to reach for first. You tell the participant exactly what to pick, inside a question that looks like its neighbors:

The instruction names one answer and matches the surrounding format. A failure is useful evidence to review alongside other quality signals; it can still be a mistake.

2. Infrequency items. Statements almost nobody can truthfully endorse, dropped into an agreement scale: “I have never used the internet” in an online survey, or “I was born on the planet Neptune.” Useful in long matrix batteries where an instructed item would stick out. Keep them literal; humor invites playful agreement.

3. Bogus knowledge items. A factual question with one obviously correct answer, like which of four words is a color. Fine as a light-touch check, but weaker than instructed response because a distracted person can still answer on autopilot.

4. Open-ended checks. Ask for a short written answer to something concrete from the study itself, such as “in one sentence, what was the scenario about?” The most informative and the most work to grade. Best reserved for high-stakes studies where you are already reading the text answers.

Keep the original responses and mark failed checks. Apply the exclusion rule you wrote before collection, report how many responses it removed, and compare results with and without those exclusions.

On paid panels, be slower to reject than to exclude. Excluding a row from analysis costs the participant nothing; rejecting their submission costs them money and you reputation. Prolific's rules only permit rejection over checks that are unambiguous and fair, and generally expect repeated failure. When in doubt, pay, and filter the row out of the data anyway.

In Plumeform, any multiple choice question becomes an instructed response check with one toggle: mark the correct option, and wrong picks flag the response in results and in every export. Speeders and straight-liners are flagged automatically alongside it, and a one-click filter shows the clean subset. The mechanics are on the academic research page.