On Living in Manipulative Systems (and Why We Shouldn’t Blame Others For Falling Into The Trap)

Jacob Ward Considers Our Free Will (Or Lack Thereof)

Featured image: SciTechDaily

When we were kids, my parents took me and my sister to Disneyland in California a few times, usually as an adjunct to a visit with my grandparents, and I remember each occasion vividly. I loved the obscure narratives, and being jerked this way and that by the overlooked Mr. Toad’s Wild Ride, and while I would never have admitted it then, I found a real bliss in the placid predictability and intricate miniature landscapes of It’s a Small World. But my favorite experience by far, the one that stuck with me in my dreams and my drawings when we returned home to Connecticut, was Autopia, a winding set of roadways where anyone over thirty-two inches tall was allowed to drive a car.

I remember the volcano of anticipation as we waited in line, and when I got the signal I sprinted out onto the track to choose among the various beach-buggy-like numbers, their early-1980s bright green and purple and yellow metal-flake paint jobs glittering in the Anaheim sun. We’d all climb in and I’d grip the steering wheel, anticipating power and freedom as the attendant fired up the incredibly loud and undoubtedly terribly polluting little car. And I can remember in my body, as I write this, the half-second delay between pressing down on the accelerator and feeling and hearing the two-stroke engine roar to life and pull my family and their nine-year-old driver (me!) wherever I decided to go.

But of course I wasn’t actually deciding anything. I always misjudged the first corner—it turns out there’s a reason we don’t let kids drive—and subjected everyone on board to the horrific thump of striking the hard guidance spine that ran down the center of the roadway, a sort of solitary curb that coursed the length of the track and violently forced the front wheels left or right if the person at the wheel didn’t keep the car centered in the lane. Because of the spine, I wasn’t driving at all, in the strict sense of the word. I was, in effect, only steering, and even that part wasn’t necessary, strictly speaking.

If I’d taken my hands off the wheel, that guidance spine would have just forced the car in the proper direction. My efforts had more to do with trying to keep the front wheels parallel to the spine to avoid the thumped reminder of its existence than with actually choosing where to go. But all of that didn’t matter. I felt like I was driving. I felt powerful. I felt free. And that experience hit my brain so hard it created for me a lifetime’s fascination with cars and planes and boats and the mechanized pursuit of speed and freedom, even though I was really only in charge of the accelerator.

Over and over again, we see that our behavior, which feels to us like free will and clear choices, is actually the result of guidance systems we’re helpless to obey, whether it’s our own brains or something external like a guidance spine. And yet the factors I’ve tried to describe in the earlier chapters allow our brains not only to make us believe we’re making our own choices, but also to fill in a narrative of power and independence in the same way that Autopia infused me with the feeling that I was on the open road.

Our brains are disabled by our unmet needs.

Let’s talk first about what I view as the largest obstacle standing in the way of our seeing this dynamic clearly: Not only don’t we have the control of ourselves we think we do, for some reason our brains also make us resentful of other people wriggling in the grip of forces beyond their control. It’s an ongoing reason we’re so easily manipulated by each other and the systems we build. Maya Bar-Hillel, who studied alongside Amos Tversky, did pioneering work on inaccuracies in human reasoning, and is now a professor emeritus of psychology at the Hebrew University of Jerusalem, asked me, “Why is it that when we encounter perceptual illusions, we get all smiling and excited?” Fun-house mirrors, hollow heads—all of that is somehow enjoyable. “But when we hear about cognitive tendencies, we get all tense and defensive!”

“All these things that we do,” she told me, “it’s not because we’re stupid. It’s because we’re human.”

Sendhil Mullainathan, a professor of computational science and behavioral science at the University of Chicago, who has worked on everything from behavioral psychology to putting artificial intelligence toward a whole-body understanding of cancer, has done groundbreaking work on the cognitive burden of poverty. And not only has he revealed that poverty is vastly more crippling than we knew, he’s also shown that we unfairly misjudge people suffering under its weight.

Mullainathan was drawn to the subject in part by reading about the infamous Minnesota Starvation Study, conducted in 1944, at the height of America’s involvement in World War II. Huge swaths of the world were starving at that time, but little formal research had ever been done on the biological and physical effects of going without enough food. Researchers selected thirty-six males from a group of more than two hundred volunteers, many of them conscientious objectors who had agreed to serve the war effort but wouldn’t engage in violence. They were chosen for their good physical and mental health, as well as their collegiality under pressure, because they were about to be placed under tremendous amounts of it.

The plan was that every man—they were all men—would lose 25 percent of his body weight. For three months researchers fed them a standard diet, 3,200 calories a day, mostly potatoes, pasta, bread, and other foods widely available in Europe. Then they suffered through six months at only 1,570 calories per day. The before and after photos are a nightmare: men with the already-lean physiques of a more constrained era transformed into walking skeletons. And they weren’t allowed to just sit and suffer: they walked twenty-two miles a week, performed physical and written work, and were constantly interviewed. The negative consequences were clear: their resting heart rates slowed, their sex drives evaporated, and they reported irritability, depression, antipathy.

Something subtler leapt out at Mullainathan, though: starvation also transformed their unconscious lives. The researchers noted, in an offhand footnote, that many of the men took to reading cookbooks during the study—pretty uncommon for your average man in 1940, and a torturous choice for men consuming so little food. And a third of these men—drawn from all professions and walks of life—reported that when the study ended, they planned to open a restaurant. When the study ended, and the men resumed eating a healthy diet, that ambition evaporated. No one followed through.

Not only don’t we have the control of ourselves we think we do, for some reason our brains also make us resentful of other people wriggling in the grip of forces beyond their control.

Mullainathan went on to determine that poverty similarly reprograms the human mind. He and his colleagues spent five years investigating how poverty and hunger can affect our mental abilities as well as our bodies. And they found that just as being hungry makes people preoccupied with food, being poor will make them preoccupied with money. He and his researchers asked hundreds of people outside of grocery stores how much they’d paid for the items in their bags.

People with enough money to easily purchase groceries admitted they couldn’t remember. People who could barely afford what they needed turned out to know exactly what a tube of toothpaste cost. It shows, he says, that traditional assumptions about persistent poverty being a function of laziness, or disorganization, or some other personal failing are wrong. And it shows that someone experiencing poverty is intensely distracted at all times. They are literally impaired by it. He went on to write a book about this work with psychologist Eldar Shafir, Scarcity: Why Having Too Little Means So Much.

Other researchers have discovered similar hard, guiding dynamics at work in other parts of our lives. A study by Anuj Shah, an associate professor of behavioral science, also at the Booth School, found that when two groups of people—those with money and those without—were given the same instructions from a doctor for treating an illness, those with money tended to recall the instructions clearly. Those without money recalled only that the treatment called for medication and what the pills would cost.

Remember Potzl’s work with Obszut, and everything that came after it? Our brains assemble a secondhand version of reality from the raw feed? Well, the research also suggests that people with and without enough material resources are experiencing distinct realities—literally seeing the world differently. Our brains are disabled by our unmet needs, and yet political rhetoric, self-help authors, and the plots of countless popular movies try to convince us that not only can we muscle our way to an independent life, but also that any claim to the contrary is a betrayal of us all.

“You would never ask the starved guy in the Minnesota study to lift a heavy weight, right?” Mullainathan asks me. “And you’d never blame him for not being able to do it. So why do we blame people who are poor for having a hard time doing certain things?”

Even the most manipulative and harmful systems we build are typically allowed to run free for generations before we sort out whether or not they’re hurting us.

All of this makes it that much more miraculous that we, as a society, have been able to recognize the power certain invented systems hold over our behavior. Just as, according to Banaji, the human brain needs huge numbers of counterfactual examples to stop associating certain qualities with certain races or genders, our society requires enormous amounts of evidence to begin blaming our behavior on a guidance system. It tends to happen only once the science has clearly established the way those systems interact with our biology to grab hold of us, so clearly that it can be proved in court.

At the age of fourteen, Sean David imagined becoming a doctor. Growing up in the 1970s, watching medical dramas after school, “I wanted to be Marcus Welby MD,” he remembers. And so he took the first hospital job he could find after graduating from college. He became an orderly, scrubbing out surgical theaters after each procedure.

“You go in, you pick up all the biological waste and dispose of it. You put this sudsy soap on the floor and mop and hose it down, go room to room,” he remembers. “The orthopedic cases were really rough, they use hammers and saws, they leave a lot of stuff behind.” But he was building his mental capacity for observing, up close, the human body in distress, and the work was satisfying, in its way.

Then, in 1991, his supervisor found him as he was wheeling a bucket between rooms and took him aside. “Your father is in the catheter wing,” he told David.

Cigarettes had always been part of David’s memories of his father. “We’d go fishing and he’d be smoking cigarettes. I remember him putting bait on the line with a cigarette in his mouth. It wasn’t unusual in those days.” But now his dad was lying, ghostly pale, weak, frightened, in this hospital. He’d come in with chest pains. It was the first time David had ever seen him this way.

The cardiac surgeon told David and his mother that David’s father had a heart lesion, likely from his decades of smoking, that would require a bypass. And as the surgeon walked back into surgery to get started, David realized “I had just finished cleaning that room.

Luckily, David’s father survived the procedure. And the experience taught him what he wanted to do with his medical training. “It became clear that smoking is life and death,” he says. “So many people don’t survive that first heart attack. So emotionally, I’m very driven by this.” After medical school, David began a research career studying why it is that patients who show every outward sign of wanting to quit smoking simply cannot do it.

Over and over again, we see that our behavior, which feels to us like free will and clear choices, is actually the result of guidance systems we’re helpless to obey.

Ultimately, he says he believes there are a few basic facts of cigarette addiction that anyone hoping to fight it off must understand. The first is that it plays perfectly on our cognitive shortcomings. First of all, when we are first offered a cigarette outside a party or after a concert, we are terrible at understanding the odds. “People consistently underestimate the risk of smoking,” he says. “It’s a 50% chance of premature death. That’s one in two. A one-in-six chance of lung cancer. The risks are huge, but somehow risk has become attractive.” Human beings simply lack whatever it is we need to truly internalize the danger, he says, and to avoid the deadly gamble altogether.

So people need to be sold on avoiding tobacco and fighting off the grip of addiction, just as the cigarettes themselves are sold. “There has to be a key promise, something to get you out of smoking,” he argues. “We need formative research that finds out what motivates different audiences, because I haven’t seen a sustained public health campaign that resonates with every segment of the population. Maybe if we could talk about lung cancer—but the trouble is, there aren’t a lot of survivors, so there isn’t a good advocacy group out there.”

And there is no natural mechanism that will simply cause humanity to stop smoking. “People live to reproductive age while smoking, so evolution can’t touch it, can’t wipe it out,” he says. In creating cigarettes—a consumer technology that delivers a consistently satisfying experience in a portable package—we’ve invented a deadly vice that is immune to natural selection.

This is the difficulty of the modern world. We invent technologies upon which we build new businesses and, eventually, entire industries. And yet those technologies and the money we make from them usually outpace our understanding of their risks and rewards. They often prey on our psychological frailties, excite us when they should revolt us, and, cloaked in marketing and social acceptability, evade our natural ability to recognize that something is bad for us and should be avoided. We go ahead and form businesses without a clear understanding of just how completely they may control the customers they serve, and how that manipulation may affect everyone else.

In the case of cigarettes, which first entered industrialized mass production in 1845 under a French state tobacco monopoly, it was another 119 years before the US surgeon general Luther Terry issued a report that showed tobacco causes lung cancer, and another 55 years (2020!) before the US government finally required a national minimum age of twenty-one to purchase tobacco products. And all that time, secondhand smoke, which, it turns out, can slowly poison people physically separated from the smoker (in an adjacent apartment, for example), was affecting all of us, even if we managed to avoid picking up cigarettes ourselves.

We live in what Thomas Friedman calls the Age of Acceleration, when breakthroughs in fields like mathematics and computation are arriving every day, and entrepreneurs are just as quickly turning those breakthroughs into products. But this acceleration is dangerous. Like cigarettes, even the most manipulative and harmful systems we build are typically allowed to run free for generations before we sort out whether or not they’re hurting us.


Excerpted from THE LOOP: How Technology is Creating a World Without Choices and How to Fight Back by Jacob Ward. Copyright © 2022. Available from Hachette Books, an imprint of Hachette Book Group, Inc.

Jacob Ward
Jacob Ward
Jacob Ward is technology correspondent for NBC News, and previously worked as a science and technology correspondent for CNN, Al Jazeera, and PBS. The former editor-in-chief of Popular Science magazine, Ward writes for The New Yorker, Wired, and Men's Health. His ten-episode Audible podcast series, Complicated, discusses humanity's most difficult problems, and he's the host of a four-hour PBS documentary series, "Hacking Your Mind," that introduces a television audience to the fundamental scientific discoveries in human decision making and irrationality. In 2018, he was a Berggruen Fellow at Stanford University’s Center for Advanced Study in the Behavioral Sciences.

More Story
A Brief History of Mass-Manufactured Clothing Ready-to-wear garments have become ubiquitous, but before the late nineteenth and early twentieth centuries, ready-made clothing...