Adam Mastroianni is the author of Experimental History, a blog with science. He studies how people perceive and misperceive their social worlds. His work has been featured in Science, Nature, and The Tonight Show with Jimmy Fallon. He has a PhD in psychology from Harvard and a certificate of completion from 137 different escape rooms. He’s originally from Monroeville, Ohio (pop. 1,400) and currently lives in Washington, D.C. You can preorder his new book, The Strangest Idea in The World, here.
Years ago, I wrote a master’s thesis that was so bad I immediately threw it in the trash. But along the way I learned something important, which is how to get seventh-graders to smoke.
My lesson came from Hansen et al., 1989. They randomly assigned students at eight junior high schools in Los Angeles to complete one of two anti-drug programs:
A) A “Social” program that was all about resisting negative influences and fostering good friendships
B) An “Emotion” program that was all about managing bad feelings
or
C) No program at all
The researchers surveyed the 2,863 participating students three times: just before the programs started, and 12 and 24 months after the programs ended. (They also took saliva samples to keep students honest.) The programs themselves lasted for 12 weeks.
In the Emotion program, students did things like:
Compliment each other to improve their self-esteem
Set goals for themselves and think about how drugs interfere with those goals
Learn strategies for emotion regulation like deep breathing and muscle relaxation
Practice being assertive and saying no to drugs in role-plays
Make a videotaped commitment to stay drug-free
Some of this stuff seems a bit dated now, sure, and the video thing sounds a little creepy. But mostly it looks reasonable—couldn’t hurt, right?
Except it did hurt. Students who did the Emotion program ended up smoking more cigarettes and more weed and drinking more alcohol than students who received no program at all. Which is to say: researchers came into schools in Los Angeles and ran a drug prevention program that caused a bunch of seventh-graders to do more drugs
(Best of all, the name of this program was Project SMART.)
What about the kids in the Social program? Basically bupkis. The effects were way stronger in the Emotion program, and in the wrong direction.
(You should always be skeptical of any individual study, and especially one this old. But I’m at least willing to assume that these results weren’t p-hacked. If you were gonna cook your books, you would probably try to hide the fact that you got 12-year-olds to toke up.)
This story makes me go “GULP”. You can have a PhD and good intentions. You can have money and buy-in. You can do a bunch of reasonable things to prevent a problem that everyone agrees is bad. You can spend a lot of time and effort running your project and collecting your data. And after all that you can, according to your own best calculations, make people worse off.1
That’s just one random paper from the 80s, but the literature is littered with similar well-meaning interventions that failed or backfired:
“Scared Straight” programs, where at-risk youth go to prisons and get yelled at by prisoners, famously do not work and maybe make students more likely to commit crimes.
The Drug Abuse Resistance Education (D.A.R.E) program I had to do in elementary school, where a cop came to class and told us about all the drugs we weren’t supposed to take, also doesn’t do anything.
A big trial of mindfulness training in UK schools maybe made kids a bit more depressed, and two attempts to teach psychotherapy principles to students in Australia both failed.
Those programs all harmed kids or failed to help them, but don’t worry, we also harm adults sometimes, or fail to help them:
Social scientists have invented lots of programs that try to make people less prejudiced. Every review of these programs concludes that they either don’t work or they don’t work in the ways we want or we don’t know if they work.
Researchers came up with their best attempts at getting Democrats and Republicans to like each other more, and all of their ideas got outperformed by a Heineken commercial
You know those signs on the highway that say things like, “Drive safely! Over 20 people have died on this highway so far this year!”? They maybe cause more people to crash and die.2
So yes, it’s hard to change people. But that’s okay—making iPhones is also hard, and we seem to have nailed that one.
What really screws us is that it’s surprisingly hard to change people. We cook up schemes that seem like they should definitely work, then they don’t work, and this doesn’t chasten us or dim our enthusiasm for future schemes. Hansen et al., after accidentally causing seventh-graders to smoke, don’t end their paper with a long reverie on their hubris. They write a few self-exculpatory paragraphs and move on to the next project.
The problem is that our illusion of explanatory depth is so deep when it comes to human behavior that we never realize how little we understand, which prevents us from ever learning more. Nobody thinks they can whip up an iPhone in their garage over the weekend, but most people think they know how to save the children, fix the schools, reform the prisons, overhaul healthcare, repair politics, restore civility, and bring about world peace. Perhaps that’s why we have iPhones and we don’t have any of those other things.
SHOULD WE RUN PARTICIPANTS OVER WITH A FORKLIFT?
This myth of the easily-malleable human is so widespread and so deeply believed that it borders on delusion. Once you see it, though, it can make sense of some things that otherwise seem psychotic.
For example, in the US, anybody who does academic research on humans or animals has to undergo ethics training first. That sounds reasonable, until you realize the “training” is just PowerPoint slides that you click through as fast as you can, and then you get asked questions like:
Which of these represents responsible treatment of research subjects?
A) Poking them in the eye
B) Using the internet to discover their weaknesses
C) Running them over with a forklift
D) Treating them with respect
And then there are a few questions like:
Subsection 41b.46 of the 1972 Research Reform Act stipulates which of the following?
A) Paragraph 17 of the 1971 Research Reform Act is null and void
B) Principal Investigators are not allowed to be listed as Co-Principal Investigators
C) Conflicts of interest must be reported on Form 99-F
D) Any research expenditures greater than $100 must be done via certified check
If you fail the exam, you just retake it until you pass. So this “ethics” test is actually a test of whether you can read at a sixth-grade level.
Most workplace trainings look like this. Here are some questions from an actual sexual harassment training you can purchase for your organization at SexualHarassmentTraining.com:
Some hypocrisy is so profound that it distorts the space-time continuum. Like when a conservative Christian senator sponsors a bill that’s like “Let’s Make It Illegal to Be Horny” and then gets caught at a brothel wearing a diaper and nipple clamps. That’s the level of hypocrisy we’re dealing with here. If you claim to care about preventing sexual harassment and then you’re out here asking whether the Civil Rights Act of 1964 is a “county” law, you do not care about preventing sexual harassment.
I understand these trainings exist solely for the purposes of ass-covering. If you get sued because one of your employees, say, sticks a research participant with a dirty needle, you can go, “Hey, don’t blame us! We made them do an ethics training!”
But why does this ass-covering work? I don’t mean legally; I don’t care if this is all because of Subsection 31a of the Let’s Make Things Worse Act of 1986, or because of the Supreme Court decision in Buttmunch v. Fartmeister or whatever. I mean why do we allow it? Anybody who has to suffer through these trainings knows that they’re stupid and useless. But when those same people are sitting in the jury box, they nod and say, “Well, you can’t blame the company, they made their employees do an ethics training.” This is the hypocrisy that really makes no sense—it’s hypocrisy on top of hypocrisy, a nipple-clamp on a nipple-clamp.
I think we accept this ass-covering because we believe that it’s possible, and even easy, to mass-produce the improvement of humans. We think we can turn sex pests into good citizens via PowerPoint presentation. When we see those PowerPoints up close, of course, we go, “Oh no, not these PowerPoints. Some other PowerPoints, the ones that actually work.”
But are no PowerPoints that work. You cannot plonk someone in front of a computer screen for an hour and expect them to become a better person. Well-meaning researchers have tried way, way harder than that and gotten way, way less.
FUZZY, WET, AND UNFALSIFIABLE
There are lots of reasons why we are continually surprised by how hard it is to change people, but there are three particularly nefarious ones that deserve closer inspection.
1. OTHER PEOPLE ARE TOO FUZZY
We perceive ourselves in full 4000K HD with Dolby Atmos Surround Sound, but when we perceive each other, it’s like we’re watching a VHS tape that someone made by using a Motorola Razr phone c. 2007 to record the screen on the back of an airplane seat.
Psychologists call this “psychological distance”: the further something is from me, the less detail it has in my imagination. So we are profoundly aware of how hard it would be to change ourselves, but only dimly aware of everything it would take to change someone else. You’re already kinda fuzzy anyway, so can’t you just be kinda fuzzy in a different way instead?
Here’s an example. I’m a weakling, a real beta boy with limited upper body strength. Let’s say I want to get ripped, jacked, shredded, swoll, etc. If you made a workout plan for me, it might look something like this:
Go to gym
Work out
Get jacked
But if I made a workout plan for me, it would look more like this:
Decide which of my very important tasks to ignore so I can go to the gym instead.
Find workout clothes.
Workout clothes are all old and ratty.
Look for new workout clothes online.
Spend the rest of the day figuring out whether I’m more of a nylon poplin guy or a jacquard knit guy.
Wait for clothes to come in.
Go to gym.
Figure out which podcast I want to listen to while I work out.
Okay now I’m ready to work out.
Wait I’ve heard this episode before, this the one where Malcolm Gladwell proves that Toyotas are safe by driving one off the George Washington Bridge.
Okay now I’m ready to work out.
Oh no there’s someone on the machine I want to use.
Mill around a bit, get on another machine and pretend it’s the one I wanted to use in the first place.
Okay he’s gone. Get on the other machine.
Oh no I don’t know how to use this machine.
Watch a YouTube video where a frighteningly large man tells me how to use the machine.
Oh no the frighteningly large man is now telling me January 6th was a false flag operation.
Use machine.
It kinda hurts, but I can’t tell if it’s a “You’re getting stronger” kind of hurt, or a “If you keep doing this you won’t be able to walk when you’re 57” kind of hurt.
Repeat for another 45 minutes.
Go home.
Wait I have to do this every day until I die??
2. WE SPRAY PEOPLE WITH A HOSE AND EXPECT THEM TO STAY WET FOREVER
I’m going to talk about implicit bias for a second, so if you’re the kind of person who gets upset about that, please deactivate your Pedantry Module and just play along for the sake of the example.
Psychologists would very much like to reduce people’s implicit biases, especially their anti-Black biases. So far, they have failed to do that. And they’ve tried lots of stuff:
Helping people set a goal to not be biased
Lecturing them about how great multiculturalism is
Telling them a story that counters their stereotypes (“Imagine a White guy attacks you and throws you in his trunk, and then a Black guy shows up and saves the day!”)
Making people play a virtual dodgeball game where all of their teammates are courteous Black players and all of their opponents are mean nasty White players
Showing them a bunch of pictures of Black faces with the word “GOOD” written next to them
Some of these interventions changed people’s biases in the moment, but none of the effects lasted beyond 24 hours. And of course they didn’t! As soon as participants leave the lab, they go right back into the world that massaged those biases into the folds of their brains in the first place. You can’t permanently change implicit biases with 15 minutes of screentime, just like you can’t spray someone with a hose and expect them to stay wet forever.
This is part of why changing people is so surprisingly hard—no matter how much you focus on the person in front of you, you’ll never appreciate the million tiny influences that made them who they are and that keep them that way. If you really want to make someone different, you might have to change the TV they watch, the music they listen to, the things they learn in school, the friends they hang out with, the role models they look up to, etc., and if you do all that, congratulations, you’ve started what we call a cult. Which, unlike social scientists, do have a pretty good track record of changing people.
3. UNFALSIFIABILITY
Most people’s theories of human behavior are just never gonna be tested, and so their hypotheses can be both wrong and immortal. For example, if you think the solution to political polarization is to drug Congress with magic mushrooms, then a) nobody’s ever going to prove you wrong, and b) I think I know which anti-drug program you got in high school.
On top of this, we’re all running our own tiny-N, p-hacked studies and then slathering the results in confirmation bias:
“My kid does better when we feed him raw beef and read him Ulysses, so that’s what all schools should do, and the fact that this also happens to fit my personal and political aesthetics is merely a coincidence.”
“I gave my religion’s holy book to a person who was struggling and now they’re doing better, which is proof my religion is true.”
“Some famous CEOs were mean to their employees and got good results, so I should do that too, which is convenient because I am a jerk.”
These itsy-bitsy confounded little pseudo-studies are more convincing to us than any “official” experiments precisely because they unfold before our own eyes. On the one hand, that’s fair—“official” experiments are so poor on average that you should consider them guilty until proven innocent. But doing low-quality uncontrolled social science in your own backyard is a good way to increase your convictions without increasing your evidence.
Even when there is reasonably good data, our hypotheses about how humans work are often so vague that they can withstand any attempts at falsification. Okay, so this emotions-based anti-drug intervention didn’t succeed, but maybe that’s because the instructors weren’t motivated enough, or it didn’t go on long enough, or they used the wrong kind of deep breathing technique, or they should have targeted fifth-graders instead of seventh-graders, etc., on and on, forever. If you can only think of all these critiques and exceptions after the fact, however, you at least have to admit you didn’t really know what you were talking about in the first place.3
LET’S GET STUPIDER
The philosopher Michael Strevens argues that science requires humans to adopt an “alien mindset”—you have to ignore common sense, the wisdom of the ancients, the literal word of God, etc. Why would anyone toss out everything they know and instead try to learn things by putting rotten meat in a jar? Science took so long to develop, Strevens says, because it seemed stupid.
I say: we must become even stupider. If all of our intuitions, theories, and knowledge cause us to run programs that make tweens do more drugs—well, then, we oughta ditch our intuitions, theories, and knowledge!
Sufficiently stupid people would look at SexualHarassmentTraining.com and ask, “Wait, how does this work? You show people walls of text and then ask them multiple choice questions and then they don’t sexually harass each other anymore?” They would wonder whether an imaginary dodgeball game could really make people less racist. They might question whether a couple deep-breathing exercises are enough to stop seventh graders from doing shots at a party.
It’s important to cultivate that kind ignorance, because every single person on Earth is at least a part-time people-changer. We want to raise our kids to be kind. We want to earn our boss’s respect. We would like our spouse to stop playing the “road trip playlist” that’s just a 10-hour loop of “Dreams” by The Cranberries and “Dreams” by Fleetwood Mac. We’ll never have large-N, preregistered, multi-site RCTs that can tell us how to do these things. We have to make our best guesses, but we also have to treat those guesses with enough skepticism that we notice when we guess wrong, so we can guess differently next time.
Some people are supposed to be professional people-changers, and so they carry a greater burden of proof and a sacred set of responsibilities. When you have the power to compel people to do things, you should be able to prove that your compulsions make people better off. It’s not enough to say, “Hey, I’m just doing stuff that everybody would agree is intuitive and reasonable,” because, as we’ve seen, intuitive and reasonable stuff often hurts people.
We can’t expect every well-meaning program to work. We know so little about how to help people that sometimes we’re going to screw it up, in which case our responsibility is to figure out what we did wrong. What we cannot do is accidentally turn tweens into binge-drinkers and then keep right on going as if nothing happened.
This is a thorny problem, but I know exactly how to solve it. I’m just waiting for the IRB to approve my application to run people over with a forklift.
By the way, if you work for Philip Morris and you’re rubbing your greedy little hands together because you’ve just discovered a new way to make kids smoke, don’t get your hopes up. In a later paper where the researchers caught up with these same students again, the difference between the Social and Emotional programs had mysteriously disappeared. The authors don’t even say whether the overall effect of the program was still there, which I bet means it wasn’t. Instead, the researchers go digging for a bunch of subgroup analyses (“It works for girls but not for boys! It works for Asian kids but not for Black kids!”). I doubt they actually have enough participants to be running these analyses—they may have been cherry-picking to make it look like the program did something.
Jason Collins blog has a great post about why he doesn’t trust this paper, which has little to do with the paper itself, but the fact that it’s a surprising single result and therefore doesn’t count for much. “You need to move from a perspective of looking at papers to looking at literatures, and then looking at those literatures with a an understanding that the literature is not representative of the full body of research [...] I don’t yet see that supporting literature.” I think this is a wise approach, which is why I qualify this finding with “maybe.”
I run into this all the time when I talk to people about the universal pre-publication peer review system.
THEM: The problem is that no one gets paid to review papers.
ME: They tried paying people and it maybe made their reviews worse.
THEM: Well, it’s because nobody gets trained on how to review papers.
ME: They tried training people and it didn’t help.
THEM: Well, they need to pay people more and train them better.
And yes, maybe they do! But if disconfirming evidence doesn’t shake your beliefs at all, you don’t really have beliefs, you have an ideology.
Looks like we're totally screwed.