Confirmation Bias: Why You Test the Cases That Prove You Right

Confirmation bias is three mistakes at different points. The first is in the search: given a belief, people look for the cases that would fit it rather than the cases that would break it. The second is in the reading: when evidence arrives, the pieces that fit are taken at face value and the pieces that do not are examined for flaws, discounted or explained away. The third is in memory: the fitting cases are easier to recall later, so the belief seems to have more support than it ever had. Raymond Nickerson's 1998 review collected these strands under one name and called the result ubiquitous.

The search error is the one the demo above shows. A guess about the rule produces triples that the guess predicts, each gets a yes, and the yeses pile up without the guess ever being at risk. Klayman and Ha showed in 1987 that this positive test strategy is a reasonable default in many situations, because when the truth is narrower than your guess a positive test can still turn up a no. It goes wrong when the truth is broader than the guess, as it was in Wason's task, and when the people answering have a reason to say yes, as they do when you ask leading questions.

The reading error is harder to see because it feels like careful thinking. Lord, Ross and Lepper gave supporters and opponents of the death penalty two studies in 1979, one finding that capital punishment deterred murder and one finding that it did not. Each side rated the study that agreed with them as better designed and more convincing, found methodological holes in the other, and came away more sure of their original position than before. Both groups had read the same two studies. Asking people to be objective does not reliably fix this.

The bias has a motivational side and a cognitive side. Sometimes people protect a belief because they want it to be true, because it is tied to their identity, their group or a decision they have already made. But Wason's participants had nothing riding on the rule and still tested one-sidedly. Part of the bias is simply that a hypothesis tells you where to look, and looking there is easier than imagining where else to look.

Peter Wason published "On the failure to eliminate hypotheses in a conceptual task" in the Quarterly Journal of Experimental Psychology in 1960. Twenty-nine undergraduates at University College London were told that the experimenter had a rule in mind that applied to sets of three numbers, that 2, 4, 6 conformed to it, and that their task was to discover the rule by writing down triples, each of which the experimenter would mark as conforming or not. They were told to announce the rule only when highly confident, and if the announced rule was wrong they could keep going.

The rule was simply three numbers in increasing order. Six of the 29 participants announced the correct rule without first announcing a wrong one. Thirteen announced one incorrect rule along the way, nine announced two or more, and one reached no conclusion. Wason reported that those who announced two or more wrong rules had been unable or unwilling to test their hypotheses, and his analysis of the records showed the pattern this page is about: participants tended to generate triples that conformed to their own hypothesis, received a string of yeses, and then announced the hypothesis with confidence. Few proposed a triple designed to get a no. Some who were told their rule was wrong responded by restating it in different words rather than testing it differently.

Wason read the result as a failure to seek disconfirmation, in the spirit of Karl Popper's argument that a hypothesis is tested only by attempts to refute it. He followed the paper with the four-card selection task in 1968, which found the same one-sidedness in logical reasoning. Klayman and Ha's 1987 reanalysis kept the finding but changed the interpretation: the participants were not failing to reason, they were applying a positive test strategy that usually works and happens to fail when the true rule is more general than the hypothesis. Both readings agree on the practical point. If every test you run is one your belief predicts, a run of confirmations tells you less than it feels like it does.

The bias shows up wherever a person already has a view and then meets new information.

The counter is not to believe less but to test differently, and most of the effective methods change the procedure rather than the attitude.

• Propose the triple that should fail. Before accepting a conclusion, name one observation that your belief says you will not see, and go look for it. If you cannot name one, the belief is not being tested.
• Consider the opposite. Lord, Lepper and Preston found in 1984 that asking people to imagine how they would have evaluated a study if it had reached the opposite result reduced biased reading, where simply asking them to be fair did not.
• Write the prediction down first. A written forecast made before the evidence arrives cannot be quietly revised to fit it afterward.
• Count the misses. For any pattern you believe in, tally the cases where the cause was present and the effect absent, and where the effect appeared without the cause. A belief built on hits alone has not been checked.
• Separate the search from the decision. Assign one person, or one hour, to build the strongest case against the preferred option, and give it the same effort the case for it received.

Confirmation bias is a broad term, and several narrower patterns are often filed under it.

• It is not the backfire effect, the claim that correcting a false belief makes people hold it more strongly. That claim has not held up well in replication; confirmation bias only says the correction is weighed less, not that it reverses.
• It is not belief bias, in which an argument is judged valid because its conclusion is believable. Belief bias is about logic; confirmation bias is about which evidence gets sought and how it is read.
• It is not selective perception, which concerns what is noticed in the first place. Confirmation bias can run on evidence that was fully noticed and then discounted.
• It is not congruence bias, the narrower habit of testing only the hypothesis in hand rather than the alternatives. The 2-4-6 task shows that habit; confirmation bias also covers the reading and recall of evidence.