The Prosecutor's Fallacy in the Case of Sally Clark

I watched a video about the tragic case of Sally Clark (1964-2007). Her family experienced two cases of Sudden Infant Death Syndrome (SIDS) in the 1990s. She was put on trial and convicted of murder. Her case is now recognized as an extreme miscarriage of justice.

SIDS is a horrible event in which a baby simply passes away in the night. People in the UK call SIDS cot-death. People in the United States use the term crib-death.

It is very difficult to determine the cause of SIDS. It might be something bizarre like the nervous system forgetting how to breathe.

I asked Night Cafe to generate an image of a spirit rising from a baby in the night for this post. I hope readers find it respectful.

SIDS is extremely sad. Fortunately, the problem is rare.

During the trial a pædiatrician named Roy Meadow stated that 1 in 8,500 babies died of SIDS. Assuming that the odds of a mother suffering two cases of SIDS would be 8,500 * 8,500 which rounds up to a 1 in 73 million chance.

The jury was aghast at this number 1 in 73 million. The conviction was influence by the scary statistics provided by Dr Meadow.

Courts overturned the conviction in 2001. The Royal Statistical Society released a letter condemning the misuse of statistics in trials.

Here is the problem I see: The RSS statement and most of the articles focus on Meadow's act of multiply independent variables.

The problem was not the multiplication. The problem was that Dr. Meadow conflated the probability of a double SIDS case with the probability that the deaths resulted from murder. This is a form of the prosecutor's fallacy or the base-rate neglect.

Dr Meadow's thinking was not entirely wrong.

Imagine that homicidal maniac killed a baby and police passed it off as SIDS.

The homicidal maniac, by definition, has mania for committing homicide and is likely to kill another baby.

For that matter, authorities routinely investigate clusters of unexpected deaths in care facilities. When they do so, they often find things like neglect, safety hazards or even caregivers killing the people in their charge.

A case of double SIDS warrants investigation. It is not sufficient to warrant a conviction.

Dr Meadow conflated the chance that a case of double SIDS occurred with the likelihood that Sally was a murderer.

NOTE: the number of serial killers is actually quite small; so one is comparing two low probability events.

The best way to describe where this trial went wrong is with base rate bias.

The Prosecutor's Fallacy and Base Rate Bias

It is common for people to focus on an individual case and fail to consider the whole population.

Dr Meadow presented the scary number 1 in 73 million.

Well, we live in a world with billions of people. Even if there was just 1 in 73 million people suffering natural cases of double SIDS; it is likely that there hundreds of natural cases of double SIDS. I did some web searches which seem to confirm this number.

Dr Meadow's approach to jurisprudence would convict every single person who experienced natural cases of double-SIDS with murder. Dr Meadow's process of reasoning would convict hundreds of innocent people.

Interestingly the problem of the prosecutor's fallacy is rearing its ugly head in our age of mass surveillance.

We can see this in fingerprint analysis.

Lets imagine that the police find a bartender slumped over bar stool with a knife protruding from his back. The police find a fingerprint on the knife. The fingerprint matches one of the drunks that frequents the bar.

Well, this fingerprint is usually enough information to convict the drunk as there were only a few hundred people who frequented this particular bar.

Today, the world has databases with hundreds of millions of fingerprints. Investigators are receiving more false positive.

Prosecutors have found that when they run fingerprints through massive databases that they often get false positives.

Investigators have experienced similar problems with genetic searches and facial recognition software. Computers run so many facial scans through databases that they routinely encounter false positives.

I have to inject. Before I get my morning cup of coffee, I look like Charlie Manson and you better get out of my way.

While the fingerprint on the knife was sufficient to get a conviction in the bar murder, a fingerprint run through INTERPOL is likely to pick up false positives. The set of people in the bar was small.

A fingerprint run through an international database has a huge base and false positives become a problem.

Lets Write the Problem in Set Theory

I will try to write the argument using symbols from Set Theory.

Let U be the set of Unexplained Deaths. Let M be the subset of these deaths that are actually Murders. To complete things, lets say N is the set of Natural deaths. M and N are mutually exclusive; so

M ∩ N = ∅;

M ∪ N = U.

Lets define m as the count of M and u as the count of U.

m:u is the ratio of murders to unexplained deaths.

Now lets say that U₂ is the set of unexplained death clusters and M₂ is the subset of murders.

Assuming that deranged killers are apt to kill again; one would expect the ratio of m₂:u₂ to be higher than m:u.

m₂:u₂ > m:u

The appearance of an unexpected death cluster is sufficient to warrant an investigation.

The ratio of m₂:u₂ is not sufficient for a conviction because the set of natural clusters N₂ exists. In the case of SIDS, I suspect that n₂:u₂ > m₂:u₂.

To get a conviction in the US, the prosecutor must prove beyond reasonable doubt that a particular case is a murder.

While the ratio m₂:u₂ might be sufficient to warrant an investigation, it is not sufficient for a conviction.

Lets denote Sally Clark as s. The court would have to prove s ∈ M when it is more likely that s ∈ N.

Conclusion

The case of Sally Clark presents a situation in which an expert witness conflated the rarity of double SIDS cases with the possibility that Sally was a baby killer.

This egregious miscarriage of justice was not an example of an expert confusing dependent and independent variables. The case was an example of base rate bias and conflation of two unrelated ratios.

The best way to avoid base rate bias is for the expert to consider the whole base rather than just focusing on an individual instance.