How Statistics Can Validate Our Beliefs… or Trick Us

The truly genuine problem… does not consist of proving something false but in proving that the authentic object is authentic.
-Umberto Eco
*

You know the old story about storks delivering babies? It’s true. I can prove it with statistics. Take a look at the estimated population of storks in each country, and then at the number of babies born each year. Across Europe, there’s a remarkably strong relationship. More storks, more babies; fewer storks, fewer babies.

The pattern is easily strong enough to pass a traditional hurdle for publication in an academic journal. In fact, a scientific paper has been published with the title “Storks Deliver Babies (p = 0.008).” Without getting too technical, all those zeros tell us that this is not a coincidence.

Perhaps you have already guessed the trick. Large European countries such as Germany, Poland, and Turkey are home to many babies and many storks. Small countries such as Albania and Denmark have few babies and few storks. While there’s a clear pattern in the data, that pattern does not mean that storks cause babies to appear.

You can “prove” anything with statistics, it seems—even that storks deliver babies.

You’d certainly have gotten that impression from reading How to Lie with Statistics. Published in 1954 by a little-known American freelance journalist named Darrell Huff, this wisecracking, cynical little book immediately received a rave review from the New York Times and went on to become perhaps the most popular book on statistics ever published, selling well over a million copies.

The book deserves the popularity and the praise. It’s a marvel of statistical communication. It also made Darrell Huff a nerd legend. Ben Goldacre, an epidemiologist and bestselling author of Bad Science, has written admiringly of how “The Huff” had written a “ripper.” The American writer Charles Wheelan describes his book Naked Statistics as “an homage” to Huff’s “classic.” The respected journal Statistical Science organized a Huff retrospective 50 years after its publication.

I used to feel the same way. As a teenager, I loved reading How to Lie with Statistics. Bright, sharp, and illustrated throughout with playful cartoons, the book gave me a peek behind the curtain of statistical manipulation, showing me how the swindling was done so that I would not be fooled again.

Huff is full of examples. He begins by pondering how much money Yale graduates make. According to a 1950 survey, the class of 1924 had an average income of close to $500,000 a year in today’s terms. That is just plausible enough to believe—this is Yale, after all—but half a million dollars a year is a lot of money. Is that really the average?

No. Huff explains that this “improbably salubrious” figure comes from self-reported data, which means we can expect people to exaggerate their income for the sake of vanity. Furthermore, the survey is only of people who bothered to respond—and only those alumni Yale could find. And who are easily found? The rich and famous. “Who are the little lost sheep down in the Yale rolls as ‘address unknown’?” asks Huff. Yale will keep track of the millionaire alumni, but some of the also-ran graduates might easily have slipped through the net. All this means that the survey will present a grossly inflated view.

You can “prove” anything with statistics, it seems—even that storks deliver babies.

Huff briskly moves on through a vast range of statistical crimes, from toothpaste advertisements based on cherry-picked research to maps that change their meaning depending on how you color them in. As Huff wrote, “The crooks already know these tricks; honest men must learn them in self-defense.”

If you read How to Lie with Statistics, you will come away more skeptical about the ways numbers can deceive you. It’s a clever and instructive book.

But I’ve spent more than a decade trying to communicate statistical ideas and fact-check numerical claims—and over the years, I’ve become more and more uneasy about How to Lie with Statistics and what that little book represents. What does it say statistics—and about us—that the most successful book on the subject is, from cover to cover, a warning about misinformation?

Darrell Huff published How to Lie with Statistics in 1954. But something else happened that very same year: two British researchers, Richard Doll and Austin Bradford Hill, produced one of the first convincing studies to demonstrate that smoking cigarettes causes lung cancer.

Doll and Hill could not have figured this out without statistics. Lung cancer rates had increased sixfold in the UK in just 15 years; by 1950 the UK had the highest in the world, and deaths from lung cancer exceeded deaths from tuberculosis for the first time. Even to realize that this was happening required a statistical perspective. No single doctor would have formed more than an anecdotal impression.

As for showing that cigarettes were to blame—again, statistics were essential. A lot of people thought that motorcars were the cause of the rise in lung cancer. That made perfect sense. In the first half of the 20th century, motorcars became commonplace, with their exhaust fumes and the strangely compelling vapor from the tar in new roads. Lung cancer increased at the same time. Figuring out the truth—that it was cigarettes rather than cars that caused lung cancer—required more than simply looking around. It required researchers to start counting, and comparing, with care. More concisely, it required statistics.

The cigarette hypothesis was viewed with skepticism by many, although it was not entirely new. For example, there had been a big research effort in Nazi Germany to produce evidence that cigarettes were dangerous; Adolf Hitler despised smoking. The Führer was no doubt pleased when German doctors discovered that cigarettes caused cancer. For obvious reasons, though, “hated by Nazis” was no impediment to the popularity of tobacco.

So Doll and Hill decided to conduct their own statistical investigations. Richard Doll was a handsome, quiet, and unfailingly polite young man. He had returned from the Second World War with a head full of ideas about how statistics could revolutionize medicine. His mentor, Austin Bradford Hill, had been a pilot in the First World War before nearly dying of tuberculosis. Hill was a charismatic man, had a sharp wit, and was said to be the finest medical statistician of the 20th century. Their work together as data detectives was to prove lifesaving.

The pair’s first smoking-and-cancer study began on New Year’s Day 1948. It was centered around 20 hospitals in northwest London, and Richard Doll was in charge. Every time a patient arrived in a hospital with cancer, nurses would—at random—find someone else in the same hospital of the same sex and about the same age. Both the cancer patients and their counterparts would be quizzed in depth about where they lived and worked, their lifestyle and diet, and their history of smoking. Week after week, month after month, the results trickled in.

In October 1949, less than two years after the trial began, Doll stopped smoking. He was 37, and had been a smoker his entire adult life. He and Hill had discovered that heavy smoking of cigarettes didn’t just double the risk of lung cancer, or triple the risk, or even quadruple the risk. It made you 16 times more likely to get lung cancer.

Hill and Doll published their results in September 1950, and promptly embarked on a bigger, longer-term, and more ambitious trial. Hill wrote to every doctor in the UK—all 59,600 of them—asking them to complete a “questionary” about their health and smoking habits. Doll and Hill figured that the doctors would be capable of keeping track of what they smoked. They would stay on the medical register, so they’d always be easy to find. And when a doctor dies, you can expect a good diagnosis as to the cause of death. All Hill and Doll had to do was wait.

More than 40,000 doctors responded to Hill’s request, but not all of them were delighted. You have to understand that smoking was extremely common at the time, and it was no surprise to find that 85 percent of the male doctors in Doll and Hill’s initial sample were smokers. Nobody likes to be told that they might be slowly killing themselves, especially if the suicide method is highly addictive.

One doctor buttonholed Hill at a London party. “You’re the chap who wants us to stop smoking,” he pointedly declared.

“Not at all,” replied Hill, who was still a pipe smoker himself. “I’m interested if you go on smoking to see how you die. I’m interested if you stop because I want to see how you die. So you choose for yourself, stop or go on. It’s a matter of indifference to me. I shall score up your death anyway.”

Did I mention that Hill originally trained as an economist? It’s where he learned his charm.

The study of doctors rolled on for decades, but it wasn’t long before Doll and Hill had enough data to publish a clear conclusion: Smoking causes lung cancer, and the more you smoke the higher the risk. What’s more—and this was new—smoking causes heart attacks, too.

Doctors aren’t fools. In 1954, when the research was published in the doctors’ own professional magazine, the British Medical Journal, they could draw their own conclusions. Hill quit smoking that year, and many of his fellow doctors quit, too. Doctors became the first identifiable social group in the UK to give up smoking in large numbers.

In 1954, then, two visions of statistics had emerged at the same time. To the many readers of Darrell Huff’s How to Lie with Statistics, statistics were a game, full of swindlers and cheats—and it could be amusing to catch the scoundrels at their tricks. But for Austin Bradford Hill and Richard Doll, statistics were no laughing matter. Their game had the highest imaginable stakes, and if it was played honestly and well, it would save lives.

By the spring of 2020—as I was putting the finishing touches to this book—the high stakes involved in rigorous, timely, and honest statistics had suddenly become all too clear. A new coronavirus was sweeping the world. Politicians had to make the most consequential decisions in decades, and fast. Many of those decisions depended on data detective work that epidemiologists, medical statisticians, and economists were scrambling to conduct. Tens of millions of lives were potentially at risk. So were billions of people’s livelihoods.

As I write these words, it is early April 2020: countries around the world are a couple of weeks into lockdowns, global deaths have just passed 60,000, and it is far from clear how the story will unfold. Perhaps, by the time this book is in your hands, we will be mired in the deepest economic depression since the 1930s and the death toll will have mushroomed. Perhaps, by human ingenuity or good fortune, such apocalyptic fears will have faded into memory. Many scenarios seem plausible. And that’s the problem.

An epidemiologist, John Ioannidis, wrote in mid-March that COVID-19 may be “a once-in-a-century evidence fiasco.” The data detectives are doing their best—but they’re having to work with data that are patchy, inconsistent, and woefully inadequate for making life-and-death decisions with the confidence we’d like.

Details of the fiasco will, no doubt, be studied for years to come. But some things already seem clear. At the beginning of the crisis, for example, politics seem to have impeded the free flow of honest statistics—a problem we’ll return to in the eighth chapter. Taiwan has complained that in late December 2019 it had given important clues about human-to-human transmission to the World Health Organization—but as late as mid-January, the WHO was reassuringly tweeting that China had found no evidence of human-to-human transmission. (Taiwan is not a member of the WHO, because China claims sovereignty over the territory and demands that it should not be treated as an independent state. It’s possible that this geopolitical obstacle led to the alleged delay.)

Did this matter? Almost certainly; with cases doubling every two or three days, we will never know what might have been different with an extra couple of weeks of warning. It’s clear that many leaders took their time before appreciating the potential gravity of the threat. President Trump, for instance, announced in late February, “It’s going to disappear. One day, it’s like a miracle, it will disappear.” Four weeks later, with 1,300 Americans dead and more confirmed cases in the United States than any other country, Mr. Trump was still talking hopefully about getting everybody to church at Easter.

As I write, debates are raging. Can rapid testing, isolation, and contact tracing contain outbreaks indefinitely, or only delay their spread? Should we worry more about small indoor gatherings or large outdoor gatherings? Does closing schools help prevent the spread of the virus, or do more harm as children go to stay with vulnerable grandparents? How much does wearing masks help? These and many other questions can be answered only by good data on who has been infected, and when.

But a vast number of infections were not being registered in official statistics, due to a lack of tests—and the tests that were being conducted were giving a skewed picture, being focused on medical staff, critically ill patients, and—let’s face it—rich, famous people. At the time of writing, the data simply can’t yet tell us how many mild or asymptomatic cases there are—and hence how deadly the virus really is.

As the death toll rose exponentially in March—doubling every two days—there was no time to wait and see. Leaders put economies into an induced coma—more than three million Americans filed jobless claims in a single week in late March, five times the previous record. The following week was even worse: another six and a half million claims were filed. Were the potential health consequences really catastrophic enough to justify sweeping away so many people’s incomes? It seemed so—but epidemiologists could only make their best guesses with very limited information.

It’s hard to imagine a more extraordinary illustration of how much we usually take accurate, systematically gathered numbers for granted. The statistics for a huge range of important issues that predate the coronavirus have been painstakingly assembled over the years by diligent statisticians, and often made available to download, free of charge, anywhere in the world. Yet we are spoiled by such luxury, casually dismissing “lies, damned lies, and statistics.” The case of COVID-19 reminds us how desperate the situation can become when the statistics simply aren’t there.

Click here for the new season of Cautionary Tales, a podcast by Tim Harford.

__________________________________

Excerpted from The Data Detective: Ten Easy Rules to Make Sense of Statistics. Used with the permission of the publisher, Riverhead Books, an imprint of Penguin Random House LLC. Copyright © 2021 by Tim Harford.