In 1973, the graduate program at the University of Berkeley, in the United States, had 12,763 applications, of which 8,442 were men and 4,321 were women. 3,799 men and 1,512 women were accepted. The numbers drew attention because the acceptance rate of men (45%) was much higher than that of women (35%). An urban legend has it that Berkeley was sued for discriminating against women, but it didn’t go that far.
Concerned, the rectory had the admissions process audited, and was in for a big surprise: in almost every department the acceptance rate for women was higher than for men! The audit even concluded that “there is a bias, small but statistically significant, in favor of women.” What was happening?
This is one of the strangest (and most frequent) phenomena in statistics: groups of data individually show the same trend, but it disappears, or can even be inverted, when we put the data together. See this simple example.
Dr. Alice and Dr. Benedict are experienced surgeons. Bento has already had 350 surgeries, of which 289 (83%) were successful. Alice also made 350, but only 273 (78%) were successful. He’s clearly better than she is, right?
But there are two groups of patients: moderate and severe. In the first, Alice performed 87 operations, 81 of which were successful: success rate of 93%. Bento scored 270, of which 234 were successful: a rate of 87%. In this group, Alice who has the upper hand!
In the second group, she performed 263 surgeries, 192 of which were successful: a rate of 78%. Bento did 80, of which 55 were successful: a rate of 69%. In this group, it is also Alice who has the best performance!
How to explain this? Most of Bento’s patients are in the moderate group, where success rates (for both) are better. Alice, on the other hand, deals mainly with serious cases, whose success rates are naturally worse. That’s why, although she is better than him in both groups, on the whole he seems to perform better.
This phenomenon is called “Simpson’s paradox”, in honor of the British statistician Edward Simpson (1922–2019) who, in 1951, published a work on a related topic. But the paradox had already been discovered in 1899, by his compatriot Karl Pearson (1857–1936), and in 1903, by the also British Udny Yule (1871–1951).
