Maintenance Phase

School Lunches, P-Hacking and the Original "Pizzagate"

Share episode

Aubrey: Hi, everybody and welcome to Maintenance Phase, the podcast that will gladly trade you two regular milk tickets for one chocolate milk ticket. 


Mike: [laughs] That’s good. 


Aubrey: I knew we were talking about school lunches, and that was my one very vivid memory.


Mike: The podcast that serves you turkey tetrazzini.


Aubrey: [laughs] I'm Aubrey Gordon.


Mike: I'm Michael Hobbes. 


Aubrey: If you'd like to support the show, we are on Patreon at patreon.com/maintenancephase. We're now releasing bonus episodes so you can enjoy some bonus content. Support the show if you want to, and don't if you don’t.


Mike: Keep listening and never give us a dime. It's chill. 


Aubrey: That's right. And today, we are talking about school lunches, I think?


Mike: Aubrey, I'm so excited. 


Aubrey: Oh, I can't wait.


Mike: This is our first, I think, straightforward clickbait episode. 


Aubrey: [laughs] 


Mike: I am going to title this episode, School Lunches, P-Hacking and the Original Pizzagate. 


Aubrey: The Original Pizzagate.


Mike: That's like how we're drawing people in. But the actual story that we're going to talk about today is basically the rise and fall of a single food and nutrition researcher who was one of the most prominent people in this field for more than a decade. His name is Brian Wansink, and I think it's a really good story. It's like by far our most methodology, Quine-y episode. But also, if we call it Brian Wansink, nobody would listen to it. So, we've gotten you here with a catchy title. And now, we're going to pump statistics into you.


Aubrey: I can't wait. Also, my knowledge of this topic runs an eighth of an inch deep.


Mike: Excellent.


Aubrey: The only reason the name rings a bell is, listener sent in an email being like, “I think you should do an episode about this guy.” And I told you about it, when you were like, “I don't know what I'm going to do for my next episode.”


Mike: And then, my eyes got as big as the dinner plates in the research we're about to cover. 


Aubrey: Well, the great and hilarious thing is that I told you about it, I was like, “Apparently, there's this whole nutrition research scandal.” And you were like, “Oh, Brian Wansink?” 


Mike: Yeah. [laughs] 


Aubrey: Piqued Michael Hobbes response.


Mike: I've been following this for a while because, full disclosure, I was one of the people who totally fell for this guy. 


Aubrey: [gasps]


Mike: I'm not going to pretend to be above any of the biases that we're going to talk about or any of, really, I think the structural problems in media and in academic research that this episode is like an entry point into. He spent more than a decade being one of the most prominent brand-name researchers in this field. He wrote two best-selling books. He was on the TED Talk circuit. When this entire downfall happened, The New York Times writes an article about, this kind of perfunctory article, and at the end, they note that he had been quoted in 60 New York Times articles over the course of almost 20 years. 


Aubrey: Good lord. 


Mike: One of the frustrating things about this, honestly, is that for our main protagonists, Brian Wansink, there's actually very little information available about his early life and how he got into the field of food research. What we do know about Brian is that he starts as a marketing professor. We're already in the foreshadowing section. 


Aubrey: I was going to say, “If this does not bode well.” 


Mike: This is an excerpt from his book, Mindless Eating, which comes out in 2007. “I'm never sure what to say when someone asks how I first became interested in food, psychology, and marketing. I usually say I really liked Vance Packard's 1957 book, The Hidden Persuaders, because he tried to show how advertising unconsciously affects us. I think this also happens when we eat, except the hidden persuaders are the way we set up our tables, our kitchens and our routines.” 


Aubrey: I'm going to go out on a limb and also assume that Brian Wansink is not a fat dude. 


Mike: Oh, absolutely not. He's a skinny white guy. He's blond. He looks around 6’1”, something like that.


Aubrey: I'm looking up a picture of him. Just-- Yeah, there you go. He's got the Ed Begley, Jr. kind of look about him.


Mike: Yeah, like suburban dad. Like, do you guys want some nachos? Calling in from the kitchen, yes. 


Aubrey: The guy who owns a recumbent bike.


Mike: Absolutely. 


[laughter] 


Aubrey: It is the vibe with this guy. 


Mike: Yes. The only thing that I think that we can pull out of these origin stories is really that he's fascinated by the idea that people, especially consumers, make choices without really knowing why they're doing it. After he gets his PhD from Stanford, he's basically a normal marketing professor at various business schools. He works at Dartmouth. He goes to the Wharton Graduate School of Business at the University of Pennsylvania. Eventually, he lands at the University of Cornell in 2005.


Because there's no biographical details in his book, I had to piece together his career basically from his Google Scholar citations. I just organized all of his research in chronological order and just started looking at the kinds of studies that he was publishing. Over the course of his career, he publishes over 480 academic studies.


Aubrey: Man, oh, man. Where does that productivity come from? 


Mike: [gibberish] I'm going to look at that.


Aubrey: Fuck, okay. Shit.


Mike: I can't tell you without a huge spoiler.


Aubrey: Okay. [laughs] 


Mike: For the first 10, 15 years of his career, the research that he's producing is very straightforward marketing research. One thing that he's really obsessed with is this idea of unit size, that if people buy a large bag of chips, they'll eat the whole bag. And if they buy a small bag of chips, they'll eat the whole bag. But he's doing all of this research to give advice to companies on how big their unit sizes should be. It's very clear that what he's doing is he's helping companies sell more products. That's the way that all of his work is framed throughout the 1990s.


Aubrey: Yeah, this is the halo top approach, right? 


Mike: Yes. 


Aubrey: Don't stop till you hit the bottom, bop, bop, bop. Let's just assume that you're going to eat the whole package of whatever you buy.


Mike: Exactly. His early work shows that what you name food is actually really important for whether or not people buy it, and it even affects their taste. So, people will actually rate wine that is from California that they think is from California as more tasty than wine that they think is from North Dakota.


Aubrey: People rated Freedom fries as tastings superior to French fries. 


[laughter] 


Aubrey: This is one of those places where it's very tempting to believe that we're all much more sophisticated than we are, and our decision-making is really different than it is, but we're all on autopilot, and we're all way more predictable than we would like to think that we are.


Mike: Completely. We all are profoundly affected by marketing, and we all think that we're not. Like I bought a Casper mattress, man. 


Aubrey: Oh, did you? 


Mike: Oh, yeah. It's not because I like did a literature deep dive. It's like they know they mentioned it on five different podcasts. 


Aubrey: Truly, two days ago, I bought a Helix mattress for exactly the same reason. [laughs]


Mike: A fun fact, Aubrey, when your Helix mattress comes, you're more likely to eat more of it if it comes in a large package than a small package. 


Aubrey: [laughs] 


Mike: A lot of his early work is around these same kinds of ideas. It's basically trying to figure out what makes people purchase products. He finds that you eat more fat if you put olive oil on your bread than butter, but you also eat less bread. His studies get a little bit of play in the media, but he's not really a name in nutrition research.


That all changes in 2005. He publishes two studies that are explosive in the press. Like, you could not avoid these stories at this time. The first, I'm sure you've heard of this one, do you know the bottomless bowl study? 


Aubrey: I don't know the bottomless. Is this an all-you-can-eat kind of thing? 


Mike: Yeah. So, they did this thing. He has this lab now that has a hidden cameras and two-way mirrors. And it's all these ways of surveilling the way that people eat and why they eat differently. The study is exactly what it sounds like. They build a bowl that has a little tube underneath it, where it's actually feeding more soup into the bowl as you're eating it. So, he finds that people eat, I think it's 53% more of this bowl that is refilling itself. It's this idea that you just eat until the bowl is empty. None of us are eating based on any satiety cues. We're just, like, “Uh, there's more left in the bowl, so I better keep eating it.” 


Aubrey: This lab of his was just the Olive Garden?


Mike: [laughs] Yeah. There actually is a test restaurant. I don't know why they don't call it the testaurant.


Aubrey: [laughs] 


Mike: At this university, where people, diners can come and they know that they're participating in studies. It's like marketing studies. The menu will have French names one night and then English names the other night and they'll test like, “Does this affect your purchasing decisions?” He's also doing these lab studies in this testaurant constantly.


Aubrey: That's fascinating. Also, as you're just talking about the experimental designs, I feel myself going, like, “Ooh.” It feels like an interesting like dinner party conversation topics. There's a lot of curb appeal to these studies.


Mike: This is catnip for journalists. 


Aubrey: Sure. 


Mike: The other big study that comes out in 2005 is the-- do you know this one, the popcorn study? 


Aubrey: I don't know the popcorn study, at least not by name. 


Mike: This is one where people are going to a movie, and at the door, he tells them, like, “You've been entered into a drawing something, something. We're giving everybody free popcorn tonight.” They do this in a bunch of different conditions. In one night, they give everybody a medium-sized bucket of popcorn. And then the next night, they give them large buckets of popcorn. The twist of the study is that half of the people in all of these conditions get shitty popcorn. They said it would squeak when you eat it. It's old, it's stale and gross. Two people apparently asked for their money back and they're like, “But the popcorn is free. Just don’t eat it.” 


Aubrey: [laughs] 


Mike: This, of course, makes a big splash because it turns out, first of all people eat way more when they are given this large bucket of popcorn. Secondly, they eat more of even shitty food. This is what he says in Mindless Eating about the study. "Did people eat because they liked the popcorn? No. Did they eat because they were hungry? No. They ate because of all the cues around them. Not only the size of the popcorn bucket, but also the distracting movie, the sound of people eating popcorn around them, and the eating scripts we take to movie theaters with us. All of these were cues that signal it was okay to keep on eating and eating."


Aubrey: I understand that people are eating this mindlessly and without really any connection to the quality of the food or how fresh it was or anything like that. But how does he get from there to-- it's all these other-- I figured out what it is, I know it's not this thing, therefore, it's this other thing. 


Mike: Exactly.


Aubrey: How does he get from point A to point E?


Mike: These questions, Aubrey, are exactly the questions that nobody fucking asked at the time. These two studies, they're written up in The Atlantic. They show up in the New York Times. There's just a huge frenzy of media activity around these two stories. And thus begins Brian Wansink’s long career of just like blockbuster study after blockbuster study. This is an excerpt from a Vox article. "His experiments have found for example, that women who put cereal on their kitchen counters weigh more than those who don't, and that people will pour more wine if they're holding the glass than if it's sitting on the table."


Aubrey: I hate this shit so much, Mike. 


Mike: I know, dude.


Aubrey: I remember years ago, do you remember that shows The Doctors. It was a daytime TV show where they had like a few doctors on at the end of every show. They would read out shit like this that is completely decontextualized, completely, nonsense. I absolutely remember being at a nail place one time getting my nails done. The Doctors was on in the background, and they were like, “Here's an interesting finding. Women who have fresh-cut flowers at home report being happier than those that don't.” 


Mike: [blowing raspberries]


Aubrey: “So, if you want to make your wife happy, bring her some flowers.” I was like, “What the fuck is this?" You're not going to talk about who has 20 bucks to blow every week on fresh flowers? Flowers are expensive. It's a very broad statement, but because it comes from a sciency source, it feels more legit. When you're like, “It's still weird bullshit garbage.”


Mike: It's perfectly structured to be in that form of like, “Hey, did you know women who do this also have this?” as if it's some kind of rule.


Aubrey: It's essentially like a Snapple cap fact? 


Mike: Yes, exactly. 


Aubrey: Where you're like, “No, no. Listen, the great thing about this Snapple cap fact is that I find out that fish take naps.” It doesn't make me be a fish who takes a nap.


Mike: The best example of that is one of his studies that gets an amazing amount of play in the mainstream media is one that shows men eat 93% more pizza and 86% more salad in the presence of women. 


Aubrey: What?


Mike: It is one of the things that people try to translate this into a weight loss rule. They're like, “Don't eat with women.” I was like, “I think something else is going on.” 


[laughter] 


Mike: There's also an infamous study where Brian goes through old editions of The Joy of Cooking. You know that The Joy of Cooking has been around since 1936?


Aubrey: Uh-huh.


Mike: He publishes a study where he says there's only about 18 recipes that have endured from 1936 through all the additions to-- I think the most recent at the time was the 2006 edition. And it turns out the calories in those recipes have increased significantly over time. 


Aubrey: Dun-dun-dun.


Mike: One of his explanations for the obesity epidemic is this idea that portions, unit sizes, the food environment has completely changed over time, and this is a perfect little encapsulation of that, that a normal dinnertime recipe is just 30% larger now than it was in 1936.


Aubrey: That feels observably true. In my lifetime alone, the largest drink size that you can get has doubled, maybe tripled. 


Mike: Oh, yeah. Another thing that he mentions in the study is that muffin tins-- If you look at old muffin tins, the muffins were half as big as they are now. 


Aubrey: Yeah, there you go. 


Mike: Another infamous one is he measures the eye angle of cartoon characters on cereal boxes. 


Aubrey: What?


Mike: And he finds that brands aimed at children, the cartoon characters are looking downward at an angle of 9.67 degrees so that it looks to children in the grocery store, as if they are looking at them. This is another of his very famous studies. If you google it, you can still find like 50 references to it.


Aubrey: Man, when you started talking about cartoon eye angles, I was like, “Are we going to get into real racist territory?” Anyway, back to this Dr. Seuss book from the 40s, like, “Oh, no.”


Mike: For once, that isn't where that was going. 


Aubrey: Oh, thank God. [laughs] 


Mike: First time on the show. By 2007, he's basically translating all of these findings into weight loss advice. His book, Mindless Eating, comes out in 2007, becomes a massive bestseller. It's reviewed in The New York Times, in The New Republic. It's this huge deal. What his research implies is that there are all of these small forces on our behavior and if you can change those small forces, you can actually lose weight without really knowing it. One of the main pieces of advice that goes around from his book is, you can just switch to smaller plates in your house because people tend to eat less from smaller plates.


Aubrey: This explains why so many coworkers around that time were, like, “Hmm, get a smaller plate.” This is absolutely what people want to hear. You can become thin without having to really think about it. 


Mike: Exactly. Without anything else in society changing either. We are going to watch a clip. This is a presentation of his work from a Michael Pollan documentary in the early 2000s.


Aubrey: Oh. [laughs] Also, at some point, we're going to do some talking about Michael Pollan, man. 


[documentary begins]


Mike: I know, dude. I know.


Brian: Grab a plate up there. The pasta's right on the stove. Serve yourself up. 


Narrator: Brian Wansink is an expert on eating behavior. He's discovered, we're often not aware of why we eat as much as we do. Sometimes it's because of something we don't give the slightest thought to, like the size of our plate. 


Brian: We will bring people in, we've given them a large plate to serve themselves. But what they don't realize that pasta is cold. 


Narrator: Wansink concocts an excuse so that everyone has to get a different plate, which is slightly smaller. 


Brian: This thing isn't the right temperature. I’d like you to come back and just grab another plate out of the cupboard there. 


One of the things we find is that they'll serve themselves a second time, they won't believe they served an amount any different than they did the first time. 


Did you guys notice any difference between the first time you served yourself, and the second time you served yourself?


Participant: It feels a lot smaller, looks smaller. 


Brian: Here's one thing we found, the size of a plate tremendously biases us, in terms of how much we serve.


Narrator: The smaller the plate, the less food people take.


Brian: [crosstalk] -serving. You serve four ounces on a nice plate. But you go, “Holy cow, I'll never be able to eat that.” So, let's take a look what happened to you guys. Now, the big plate, 207 calories. Smaller plate, it dropped down to 162 calories. Woo. That's about 40 calories. If this happened three times a day over the course of a year, you use a smaller plate, you would weigh nine pounds less than you would if you had a bigger plate. 


It just really, really small things make this really huge difference. 


[documentary ends]


Aubrey: I hate this shit. 


Mike: I know we do. 


Aubrey: I hate it so hard. 


Mike: I knew you would do.


Aubrey: It really feels like it plays into this life hack approach to, “There are scientific reasons that are beyond your control. And if you just fix those scientific reasons, you will become thin.” Again, any fucking fat person can tell you, having less pasta on your plate does not make you nine pounds lighter at the end of the year. It's just it's such a weird facile logic, that I think because it's coming from a researcher and because it's coming from a researcher at an Ivy League University, that it feels fancier, it feels more legit. 


Mike: I also love the fact that it's this wildly artificial scenario. It's only what appears to be college undergrads. They are all white. They are in this weird situation where they serve themselves food. And then, he says, “No, no. We have to serve you again out of this microwaved bowl and new plates.” And then, they take the plates, but it's not clear if they're going to eat the first thing they served themselves. So, it kind of makes sense that like the second time, you would take less because you're like, “Well, do I have to eat all of this?” The whole scenario is just so fucking weird and artificial that it's not clear to me that you can actually extrapolate from this.


Aubrey: Also, I'll tell you what, that pasta looked overcooked before it went back in the microwave. 


Mike: I know. It looked bad. 


Aubrey: It looked mushy by the time it came out. So that part's also-- I would take less, it was already overcooked, and now it's pasta mush.


Mike: Do you remember this context? Do you remember the book, Nudge?


Aubrey: Yes. Also, this was around the time that Malcolm Gladwell burst onto the scene. We all started talking about 10,000 hours of whatever for mastery. It was a whole wave of, again, this life hacking kind of stuff. That was like, "You just need to know these little-- they seem small, but they're really important scientific findings that will impact every other thing about your life."


Mike: Yeah, I mean, this was one of the most important prominent ideas at the time, was this idea of behavioral economics, that the way that people behave is-- one of the books that came out at the time was called Predictably Irrational. So, the canonical example that showed up in every single article about this was organ donations. Some countries have 17% of people volunteer to be organ donors. And if you look at, I believe, it was the Netherlands, it's like 60% of people. It seems like, "Oh, my God, there's so much more virtuous. What's going on in the Netherlands, that they're also much nicer about their organs?” And it turns out that it's just the default on the form. In America, you have to tick a box to say, “Yes, I will donate my organs.” In the Netherlands, you have to tick a box that says, “No, I don't really want to donate my organs.”


Aubrey: It's the same thing with automatic voter registration. When people have to opt out of being a registered voter, more people are registered and more people turn out to vote.


Mike: Yes. Brian Wansink was actually a huge part of that. The guys that wrote the Nudge book, the book that basically began this entire trend in 2008, they wrote a really long review of Brian Wansink’s book in The New Republic. This idea of our eating behavior being a metaphor for all of these other behaviors, whether we pay our taxes, which schools we send our kids to.


Aubrey: This feels like an extremely dude way of approaching the world, which is like, we just need to listen to the data and do what the data says, that's it.


Mike: It was this very “rationalist approach” that life can be broken down into these inputs and outputs, these little flow charts. We know that people will do X, if we give them Y. We can predict the ways that people are going to behave, and all we need to do is follow the science and we'll be able to solve all of these social problems.


Before we get to the downfall, I just want to talk a little bit more about the kind of work that he was doing. He did a lot of workplace wellness consulting. 


Aubrey: Oh, no.


Mike: I know. This is the thing, I don't want to get into it, because we need to do a whole episode on the unbelievable trash fire that is the field but I just wanted you to read one very brief excerpt. This was the kind of advice that was going around to workplaces at the time. This is an excerpt from his book, Slim by Design, and he's talking about how he's doing a consultation with Google to prevent what's called The Google 15.


Aubrey: Fuck off.


Mike: When people start working at Google, because there's these canteens everywhere, and the food is really good, and it's free, everybody gains weight when they start working at Google. This is one of the ideas that he comes up with. I'm going to send this to you because I cannot get through this without tittering.


Aubrey: Okay. “To tackle the “I gained weight before I knew it” problem, Bob Evans, one of their software engineers, had an idea. Have you ever seen those iPhone or Android apps that let you upload a photo of yourself, and it shows you what you would look like if you were 20 or 40 pounds skinnier or fatter?” Oh, Mike, I hate this already. 


Mike: I know. It only gets worse.


Aubrey: “John figured out there might be a way to have a “food scanner” set up that could scan someone’s tray and a camera screen in front of them would take their photo and instantly display what they would look like in a year if they ate this much food every day for lunch. Way cool,” is the end of that quote.


Mike: [laughs] It's one of those demented fucking things I've ever heard. 


Aubrey: I hate every-- So, they have no concept of people with eating disorders or body dysmorphia. They have no concept of fat people and increasing bias. They have no concept of a lot of the things that I care the most about.


Mike: Imagine being a fat person at Google, and somebody gets their tray out and they hold it under this fucking miserable Minority Report scanner, and it shows them a body that looks like you? What are we fucking doing here, Brian?


Aubrey: It really feels like they're a hop, skip, and a jump away from just adding oinking sound effect or something. This is like school bully shit.


Mike: Oh, yeah. He also at one point in the same chapter suggests that employees should have to sign health declarations.


Aubrey: Fuck off.


Mike: Where they promise their employers that they're going to exercise two days a week. And if their BMI goes above 30, they have to be mandatory attendance at Weight Watchers. 


Aubrey: Fuck off. 


Mike: I know.


Aubrey: Truly, madly, deeply, fuck off. 


Mike: Again, we're going to save most of our fire emojis for our eventual workplace wellness episode, but that is a main thread of his work. The other main thread of his work is this Smarter Lunchrooms Program. Have you heard of this?


Aubrey: I remember this because, again, I have a mom who's an early childhood ed person who never stops yelling about this study, which is, the kids were offered-- I think it was fruit and candy. 


Mike: Yeah, it was apples and cookies. 


Aubrey: Apples and cookies. They studied essentially, if you just offer kids an apple or a cookie, which one do they pick? Unsurprisingly, a lot of kids picked cookies. And then, they put stickers of Elmo?


Mike: Yes, it was Elmo.


Aubrey: Is it Elmo? On apples, and then they were like, more kids chose the apples. This is the thing that my mom never stopped yelling about. She's like, “Look at the ages of the kids that they're putting fucking Elmo stickers on apples for.” They're like 10.


Mike: 8 to 11, yeah. 


Aubrey: You know who doesn't care about Elmo? Is like a fourth grader. [laughs] 


Mike: Man, you are in danger, grave danger of spoiling this episode-- [laughs] 


Aubrey: Oh, really?


Mike: We will come back to those.


Aubrey: Really, really? 


Mike: Yes. This is one of the central studies that becomes the basis of this entire program. So, this is by Nick Brown, a researcher who looks into this later. Here's how the study worked. Researchers recruited 208 students at seven elementary schools. As part of their regular lunch menu, these students were already allowed to take an apple, a cookie, or both in addition to their main dish. Before the study period, about 20% of the children chose an apple and 80% chose the cookie. But when researchers put an Elmo sticker on the apple, more than a third chose it. So, it's like perfect Brian bait. It's cheap. It's easy. It doesn't require taking away the cookie. It's just this little tiny thing and you've got more kids eating fruits and veg.


Aubrey: It does feel very odd to be like, "What if fruits and vegetables were branded? What if these were Star Wars grapes?" where you're like, “Well, sort of, but fine.”


Mike: A lot of this actually is trying to use traditional marketing techniques for relatively unsexy fruits and vegetables. Another one of the canonical studies that's part of this program is renaming vegetables. 


Aubrey: What?


Mike: Yeah. They tried to brand vegetables in cafeterias to make them cool. So, there's like x-ray vision carrots is one of them, because carrots have beta carotene and that helps your eyeballs. This is a list of the brands that they used for fruits and vegetables and lunch rooms. Orange squeezers, monkey phones, that's bananas, snappy apples, cool as a cucumber slices, sweetie pie sweet potatoes, and they renamed healthy bean burritos as big bad bean burritos. According to the studies, this actually increases consumption as much as 30%. So, this basically becomes a massive long-running program. 


In 2007, he's appointed to the USDA, and he starts helping them design this Smarter Lunchrooms Program, which is exactly what you would expect from his kind of work. There's a checklist of 15 different changes, and they're all along these lines. You add a salad bar, but you move the salad bar in the middle of the cafeteria so kids have to walk around it. It's not in a corner where they can ignore it. He suggests things like you put fruit in a bowl next to the cash register, rather than this special place where kids have to go search for it. He moves the chocolate milk to the back of the rack so you have to reach a little farther for it. It's all of these little tweaks.


Aubrey: It's the school lunch room equivalent of putting tabloids next to the cash register.


Mike: 100%. Yes, that's a very good metaphor. I did not know this when I started researching this, but this was used in 30,000 schools. 


Aubrey: Whoa. 


Mike: So, these are the three main threads of his work. There's the weight loss stuff, there's the workplace wellness stuff, and there's the school lunchroom stuff. He's giving talks on every continent, and he's quoted in the newspaper a billion times. And he's massively famous researcher, one of the few brand-name researchers in this field. On December 25th, 2016, the whole thing comes crashing down.


Aubrey: This is where we get to the scandal part?


Mike: Super-duper scandal part. Yes.


Aubrey: Excellent. Give me some scandal. 


Mike: I fucking love this. This is one of my favorite downfalls. 


Aubrey: Really?


Mike: I feel bad about celebrating this, but this is just one of the most delicious downfalls I've ever seen. Okay, the entire thing, the dominoes start to fall with a blog post.


Aubrey: Wait, Brain Wansink writes the blog post? 


Mike: Yeah. He has a blog at the time that's quite well known, and he talks about his research and their findings and just in the ways academics have blogs, so just have musings on various things. In late 2016, he writes a blog post that begins with three paragraphs, that I am going to make you read. 


Aubrey: Oh. Is it going to be better or worse than the fucking Google Scanner? 


Mike: Oh, way better.


Aubrey: Okay, good. 


Mike: By the standards of our show, this is weak shit. This is fine. The trauma meter that is always bouncing in the red at the bottom of our show at all times? This is green to yellow.


Aubrey: [laughs] Okay, good to know. Okay. “A PhD student from a Turkish University called to interview to be a visiting scholar for six months. When she arrived, I gave her a data set of a self-funded failed study, which had no results. It was a one-month study in an all-you-can-eat Italian restaurant buffet where we had charged some people half as much as others. I said, "This cost us a lot of time and our own money to collect. There’s got to be something here we can salvage because it’s a cool, rich and unique data set." I had three ideas for potential Plan B, C, and D directions since Plan A had failed. Every day, she came back with puzzling new results, and every day we would scratch our heads, ask "Why?" And come up with another way to reanalyze the data with yet another set of plausible hypotheses. Eventually, we started discovering solutions that held up regardless of how we pressure-tested them. I outlined the first paper and she wrote it up. This happened with a second paper and then a third paper, which was one that was based on her own discovery while digging through the data.” 


Mike: What do you think? 


Aubrey: So, basically, he talks about bringing in this PhD student to help out at his lab. He's essentially asking her to keep reinterpreting the data, basically, until she finds something. Every day, she goes and reinterprets the data and brings it back to him and he goes, “They're not quite it, reinterpret it again. That's not quite it, write this paper and write it differently.” It seems real fucking wild to go back to the same data set again and again and again, and go, “What about this? What about this? What about this? What about this?” Of course, interpretation is always, always, always part of the deal when you're doing research. Everything gets interpreted by humans. There is nothing that is fully, fully, fully objective as we want to think there is. There's a point at which either there are conclusions to drop or there aren't. When you start to force it, you start to change the shape of the data itself. It feels almost like photoshopping. There's a point at which you're changing the contrast and the brightness, and then there's a point at which you're actually just manipulating what's in the photo.


Mike: Yes. Do you want to hear the titles of the papers that came out of all of this data digging? 


Aubrey: Oh, God.


Mike: Low prices and high regret. How pricing influences regret at an all you can eat buffet? Lower buffet prices lead to less taste satisfaction. How traumatic violence permanently changes shopping behavior? And also remember the study we mentioned earlier that men eat more in the company of women? That's one of the five studies that they get from just scraping this data, basically, to death to find any associations in it.


Aubrey: They didn't set out to be like, “Hmm, let's take a pool of people who've experienced traumatic violence and see what happens to them in a grocery store.” They were like, “We're doing a study in a grocery store. What happens when we look at just the people who've had experiences with traumatic violence?" basically, right? 


Mike: Yes.


Aubrey: That's a big oops.


Mike: Yeah, it's bad. 


Aubrey: And it's more than oops. 


Mike: The reason why I find this so delicious, this is the cute opening anecdote to a blog post that isn't about this. The blog post is basically, this rise and grind bullshit where he's comparing this hardworking, unpaid Turkish researcher to a paid grad student in his lab who refused to do this for me. The whole thing is this fucking subtweet of this poor woman who left his lab and didn't want to do this wildly unethical research. So, he says, “Six months after arriving, the Turkish woman had five papers accepted or submitted. In comparison, the postdoc left after a year and also left academia, with one quarter as much published as the Turkish woman. I think the person was also resentful of the Turkish woman."


Aubrey: Goddamn it. It's also I will say in this blog post, it feels extremely wild to watch someone commit career suicide without knowing that that's what they're doing. 


Mike: Right. I'd have no idea. 


Aubrey: It's really something. 


Mike: But are you familiar with this term p-hacking?


Aubrey: It's one that I've heard. My understanding is that it is this general practice of you interpret the data so much that you start to manipulate it.


Mike: Basically, yeah. I actually think that a better term for this is “harking,” which stands for hypothesis after results are known. 


Aubrey: Oh, that's a great acronym. 


Mike: It's good. And it's a good word too. [crosstalk] --harking at the moon and stuff. This is basically what Brian is describing here. Where it's like you've gathered all of this data, the central question that you're trying to answer, you didn't get the result that you wanted, or it's inconclusive, or whatever. So basically, you just start systematically going through your data and being like, “Well, what about men eating with women? What about older people and pizza? What about salad?" You just start going through it and being like, “Well, is there anything else here?” This is a problem in science generally, but it's especially a problem in nutrition research. I think that's something that people don't really know or haven't really internalized, is that there's essentially no way to research nutrition, because you can't really induce diet changes in people in any sort of scientifically robust way. This is why essentially, every study that compares like the Atkins diet to the Ornish Diet, none of them actually find interesting results, because nobody can stay on these diets very long,


Aubrey: Right, almost all of those studies are like-- there's a little section where they just mention briefly, like, “70% of the people on this diet dropped off. Anyway, the results are, blah, blah, blah.” You're like, “Well.” 


Mike: Exactly. The only thing that leaves you is to survey what people are already doing. This is how you get a billion of these studies where they'll take 10,000 people or these giant cohorts, and they'll ask them a bunch of questions. Do you eat blueberries? Do you eat apples? Do you have cancer? Are you tall? Are you short? Are you redheaded? And then, you can publish the associations that you find. People who eat breakfast every day weigh less than people who don't eat breakfast every day. Every day, you can find a study coming out that is along these lines.


Aubrey: Right. People who ate full fat dairy as a kid are more likely to be thin in adulthood-- [crosstalk] 


Mike: Exactly. 


Aubrey: It blows people's brains out of their heads and is one of the sorts of associations where you're like, “Okay, but what else does that mean?”


Mike: Exactly? There's a really good series of articles by Christie Aschwanden at BuzzFeed who does a deep dive into the way that these large-scale studies are done. The main thing to know about these huge survey studies where they're asking people about their health conditions and about their weight, is the data is total trash, because there's really only two ways that you can get information from people about what they're eating. The first way is you do these 24-hour recall studies. You keep a diary for a day, and then you write down like, “Today, I had a sandwich for lunch, and then I went to McDonald's for dinner,” or whatever. But, of course, the problem with that is that first of all, the minute you start keeping a food diary, you start eating differently. 


Aubrey: Yeah, that's right. 


Mike: If somebody tells you to write down everything you're eating, you're probably going to eat better that day. Even that, Christie in her article talks about, like she goes to an Indian place and eats a curry for dinner. And she's like, “Well, how many calories was that? How many grams was that? What were the ingredients in that curry?” None of us know the weight of what we're eating or whatever.


Aubrey: Totally. Even when you do have screen-ahead kinds of foods that you're eating, if you're like eating a stalk of celery, the difference between a small stalk of celery and a large stalk of celery is subjective.


Mike: 24-hour food diaries are problematic in their own way, so a lot of studies will do food questionnaires. They're called food frequency questionnaires, which is a pretty standard methodology for these kinds of studies that include a huge battery of questions about how often in general you eat various foods. As you are probably guessing, this is also super problematic.


This is an excerpt from Christie’s article in BuzzFeed where she actually took one of these questionnaires. "Some questions, how often do you drink coffee were straightforward. Others confounded us. Take tomatoes. How often do I eat those in a six-month period? In September, when my garden is overflowing with them, I eat cherry tomatoes like a child devours candy, but I can go November until July without eating a single fresh tomato. So, how do I answer the question? 


Questions about serving sizes perplexed us all. In some cases, the survey provided weird, but helpful guides. For example, it depicted what a half cup, one cup, or two cups of yogurt looked like with photographs of bowls filled with various amounts of woodchips." I don't know why they didn't just use bowls filled with yogurt, but whatever.


Aubrey: It seems really odd to choose-- [crosstalk] 


Mike: That seems like an easy one.


Aubrey: You know what it is? It's the commercials for tampons and pads that are like, it's just mystery blue liquid. You know that time of month, when just there's a bunch of Windex that shows up in your underpants? 


Mike: [laughs] 


Aubrey: This is that.


Mike: "Other questions seemed absurd. Who on this planet knows what a cup of salmon or two cups of ribs looks like? I noticed that when I was offered three choices of serving sizes, my inclination was to pick the middle one, regardless of what my actual portion might be." There's quite a few studies of how bad these are. In one of these large cohort studies, they found that people were underestimating their calorie counts every day by as much as 800 calories.


Aubrey: Whoa. 


Mike: It basically makes all of these comparisons are completely invalid because you can't say that blueberries prevent glaucoma or something, you don't actually know how much blueberries people are eating.


Aubrey: This feels an inroad to the studies that we're constantly getting on foods that are controversial nutritionally, like cranberry juice is really good for you. No, it's really bad for you. Dark chocolate, have it every day. Never have it.


Mike: Oh, my God.


Aubrey: Red wine, drink it for your heart health, but not after age 70, or whatever the things are, that we're constantly getting conflicting information about a handful of things. Eggs were this way for a long time.


Mike: Fucking eggs, man. 


Aubrey: Again, it makes it feel like it is sort of impossible to know, as a consumer, what you should and shouldn't be eating? Actually, the answer here is to be more transparent about like, it's really hard to find conclusive findings. 


Mike: Yeah, we just don't know.


Aubrey: We're not good at that. We don't have media systems that are especially good at-- display things as take it with a grain of salt. They're just a bunch of systemic and individual problems. Also, I would say, we're asking people to do this thing, which is estimate the size and amount and weight and caloric value of things without really having any training and how people should do that. Generally speaking, we're all pretty bad at that and we shouldn't actually get good at it, because when we get good at it, a lot of us develop eating disorders. 


Mike: Yeah, no kidding. She has a really interesting section in her article, where she talks about, because this questionnaire includes 54 questions, this is where p-hacking comes in, because you're getting 54 variables. She says, “The food frequency questionnaire we used produced 1066 variables and the additional questions we asked sorted survey takers according to 26 possible characteristics. This vast data set allowed us to do 27,716 regressions.”


Aubrey: Holy shit.


Mike: According to the data that they collected, people who eat cabbage are more likely to have innie bellybuttons. People who eat shellfish are more likely to be right-handed. People who eat more fried fish are more likely to be Democrats. People who eat bananas have higher scores on the SAT verbal. 


Aubrey: Yeah, I ate a lot of bananas. 


Mike: These are all statistically significant results, by the way. These are all technically publishable.


Aubrey: I also think like part of the backdrop of findings like these and interpreting findings like these and people just running with them, is this desire to believe that scientifically we have arrived, that we know everything that is knowable. We have reached the pinnacle, and we are there now. And now, we're looking out onto all of the world as it is. Rather than going, “Wait a minute. Sometimes, people are not as meticulous as we want them to be.” Or, sometimes we put wishful thinking into our science, or sometimes there's shit we don't know and techniques that are developing now that will help us in the future. Instead, what we get is a conversation about science, that is like, “The science says this. Therefore, it's true. Therefore, you've got to get smaller plates.” Where you're like, “Well, that's not the whole picture.”


Mike: I also think that one of the fundamental misunderstandings here, and this comes up so much, is that when you hear that something is a significant result, that makes you think that it's big. If I say watching Legally Blonde had a significant effect on my life, you'd be like, “Okay, it's a big effect.” But the term 'statistically significant,' all that means is that it's unlikely to be due to chance. This is from a Nature article about this. “Critics bemoan the way that P values can encourage muddled thinking. Last year, for example, a study of more than 19,000 people showed that those who meet their spouses online are less likely to divorce than those who meet offline. That might have sounded impressive, but the effects were actually tiny. Meeting online nudged the divorce rate from 7.7%, down to 6%." So, those are statistically significant results, but people who met online have a 1.5% lower divorce rate is not that interesting.


Aubrey: And also, if you're using a site like say, eHarmony, where they're like, “It's science. We're matching you based on science,” that also like increases your buy-in to the method of meeting and makes you feel like it's somehow more legit. Again, there are so many variables here that could account for these differences. When you just say if people who met online are less likely to get divorced, people are like, “Oh, I should be looking online.” Rather than going, “Well, wait, what does that mean?”


Mike: Yeah, This also comes up a lot in anything involving mortality, that you always hear these things, like, "Eating nuts reduces your risk of prostate cancer by 40%." And then you look at the actual numbers, and it's like you have a 3 in 100,000 chance, and that goes down to a 2 in 100,000 chance. It's not clear to me that I need to change my dietary habits, to make this extremely rare thing like slightly rarer.


Aubrey: Totally. In our brains, because most of us are accepting this news pretty passively, and again, pretty uncritically, we hear that as eat nuts three times a week, and you definitely won't get prostate cancer. Part of this sort of breakdown happens in the research itself. Part of it happens in how the research is presented in the paper. Part of it happens in how the research is interpreted and reported in media. And part of it happens at the point of consumption, which is the point which we hear it and translated into what we're going to do in our daily lives. There are breakdowns at every step along the way in this process.


Mike: I also think the fundamental point about all of this research to, especially these large surveys, is that they don't show causation, that they're very limited in what they can show. All they can show is associations. There's ways that you can control for poverty, you can control for education. I'm a little skeptical of how much statistical controlling you really can do. But the fundamental fact is that all you can find is associations. And oftentimes, those are measuring a third thing, there's probably something independent that is affecting how many bananas you eat, and your score on the SAT verbal.


Aubrey: Bananas as a snack are also speaking to a really specific racial class and cultural background. If you're in an immigrant family, your afterschool snacks might be a different thing. That doesn't make you less likely to do well, y our SAT verbals. There are other factors at play here. [exhales] It feels really challenging. I find myself getting really angry. It's we're talking about all this-


Mike: Dude. 


Aubrey: -because it's huge, weird, not intentionally-- I don't even know maybe intentionally. It's a grift economy.


Mike: This is what's so hard about this is because I think for people of good faith, if you're actually trying to find out which nutritional habits are the best for promoting health, there's no perfect way to gather that information. There's no good way to answer that question. Most of the people in this field are trying to triangulate, zigzag their way to real answers. But the problem is that these methodologies, the gaping holes in these methodologies leave them vulnerable to grifters and also vulnerable to the incentives of science. One of the things that is really important in Brian Wansink’s story is this idea that you have to publish. If you want to get tenure, if you want to get noticed in your field, you have to publish as much as possible. For a lot of people, if you've gone to all of this work to gather this data, you spent months watching what people are doing at a pizza buffet, it's like, “Well, fuck, I can't just get rid of this. I spent a ton of money. There's grant money on the line.”


Aubrey: Yeah. I also feel because of the ways in which Science has been used as a political football in recent years, particularly around climate change, particularly around-- COVID is a great example. Do you believe the science, or do you not believe the science? So, there has become this reaction on the left to be like, “We believe in science,” which means that we accept a lot of this stuff uncritically, that we slip slide into this mode of just like, “Whatever science says is the truth,” without really a recognition of what most researchers and most scientists will tell you, which is that science is a series of very active participatory conversations. The point of science is to figure out things we don't know.


Mike: And it's a process.


Aubrey: And it's a process. 


Mike: This actually brings us to the Original Pizzagate.


Aubrey: This is not comet pizza. 


Mike: No. Thank God.


Aubrey: This is not QAnon. This is a different pizza-related scandal.


Mike: This is just a clickbait title that I'm giving this episode, just to be clear. After this blog post comes out, the comments section is an absolute Red Wedding. Everybody in the comments is like, “This is why I left science.” “What you're describing is exactly the problem with science.” So, there are four grad students, random people, they're not official investigators. They're basically just people who read this blog post, and they're like, “This dude sucks.” Their names are Tim van der Zee, James, Heathers, Nick Brown, and Jordan Anaya, and these four dudes dive deep into Brian Wansink’s work. 


The first thing that they dismantle are these pizza buffet studies that he was talking about that he sent to this Turkish researcher. One of the first things that they find is in these four studies that are all based on the same data, they have a total of 95 references to Brian Wansink’s other work. In the literature review, they'll say like, "Blah, blah, blah, we know larger plates, blah, blah, blah." And yet, none of them had any links to each other and none of them even mention that there's been any other publications with this same data. So, that's already a statement that, like, “We know what we're doing here.”


Aubrey: This is an active choice to kind of bury the lede.


Mike: Yes. The other main thing that they find in these pizza studies is weird, statistical irregularities. We have to do some math to understand this. I'm going to try to make this as simple as possible, partly because I'm not sure that I understand all of this, but I'm going to try to present it as well as I can.


Basically, imagine if you had a sample of 100 people, and you're trying to figure out, do people like broccoli? Yes or no. And you're serving them, and the only two options are yes or no, they can't say I don't know. They have to say yes or no. If I did that study, and I came to you, and I said, “32.5% of people like broccoli?”, that's an impossible number. If I'm serving exactly 100 people, there's a finite number of sort of results that I can get. If I say 32.5% of people, that would mean that 0.5% of a person is in that data. Does that make sense? 


Aubrey: Right. If you're doing a study of three people, which you wouldn't because that's too small, your options are 33.333%, 66.666%, or 100%. If it's two people, your options are 50% or 100%.


Mike: Yes, that's better than my broccoli thing. Even if you have 672 participants, there's going to be a finite number of options that you can have. Even though it's going to be a much larger number of options, it is also going to be finite. One of the things that they find when they start going through these pizza studies, is that a lot of the numbers are impossible for the regret data, people are rating their regret on a 1 to 7 scale. I don't regret it, or I regret it super-duper much. One of the samples I think is people who ate more than three pizza slices, there's only 10 people in that tranche. So, if all 10 people save 7, I regret it the maximum amount, to get the average, you divide it by 10 because that's the number of people. A score of a total of 70, divide that by 10, you get 7. If everybody says 7, but one person says 6, you'd get 69. And then, you divide that by 10, and that'll be 6.9. And you can keep going all the way down, 6.8, 6.7. 


Aubrey: It's basically like, you'll get to 1/10th of one percentage point, no matter what. 


Mike: You can't get pie out of that. 


Aubrey: That's right. 


Mike: What they notice in the tables of the published tables of these pizza studies is that one of the averages is 2.25 and another one is 3.92. 


Aubrey: Aren't they then just fully making up numbers? 


Mike: This is the thing. To this day, we don't know exactly what happened. The only thing that makes any sense is that they're doing this from different sample sizes. Maybe they were using 11 people for these calculations and forgot to replace them. But what this indicates is that it's p-hacked to fucking death.


Aubrey: It is fascinating to me that all of this past peer review with very simple arithmetic issues, this isn't even like, how do you interpret data, blah, blah, blah, blah. This is just like, do all the numbers in the first four columns, add up to the number in the fifth column, no?


Mike: No, they eventually find 150 mathematical impossibilities. 


Aubrey: [gasps]


Mike: Another really weird thing is that the sample sizes don't add up. So, if you have 100 people in your study, and then it's like people who eat no pizza, people who ate one slice, people who ate two or more slices, that's your whole study, that should add up to 100. But it doesn't add up to 100.


Aubrey: Right. I'm thinking about this the data analysis that they're doing being a little bit like Hansel and Gretel style, that they didn't quite leave the trail of breadcrumbs to get them back out.


Mike: Yes. So, basically, after Pizzagate, after they look at these four papers, everybody starts looking at this guy. 


Aubrey: Yeah, for sure. 


Mike: James Heathers, one of the guys that's actually looking into this, designs a-- it's called SPRITE, I forget what it stands for, but it's this statistical tool that can actually reconstruct results. So, they start going back through his papers. Remember, the popcorn study? 


Aubrey: Yeah, the bad-tasting popcorn.


Mike: That one falls apart. There's numbers in there that should be impossible. To be fair, they say that that's not as bad as some of the other ones. There's one where it's one of the workplace wellness ones, where if you're sitting near a candy dish, you eat more than if it's placed far away from you, that one falls apart. Another one that gets debunked is the bottomless bowls study. It just has a lot of data in it that literally is impossible. There's no way to get those averages and standard deviations from real data.


Aubrey: It's so tricky, because none of this means that any of this is definitely true or definitely untrue. There's a bunch of stuff we thought we knew, and we don't actually know. And it's gotten so far out into the public imagination and into our collective bloodstream, that you can't unring that bell. By the time people just start to intuitively believe and understand that men eat more pizza in front of women, you can't be like, “Well, actually, mathematically, those findings were--" [crosstalk] That doesn't mean anything to anybody.


Mike: Yeah. Also, I would not be remotely surprised if it turned out that if a candy dish is near you, you end up eating more candy during the day. I think that's extremely plausible. I also think that this is the whole point of science, is to actually confirm things that seem plausible. It seems super plausible that the sun rotates around the Earth. But then, you look into it, you're like, “Whoops, it seems like this very common-sense plausible thing turns out not to be true.”


Aubrey: This is the episode when Mike comes out as a flat earther. 


Mike: I know. [laughs] 


Aubrey: Listen, there is a glass dome. There are flat edges. Why is there a horizon? I'm just asking questions.


Mike: Did you not know that I was a Ptolemy stan? I like [crosstalk]. And then, it's from this mathematical stuff, it's just a nonstop avalanche of ethical, weird stuff. There's a huge amount of self-plagiarism. Entire articles that he's basically repurposing for book chapters and vice versa, tons of that stuff. My favorite one is apparently he wrote some weird article about World War II veterans, and how trauma affects their eating habits, something, something. This is right there in black and white. In his sample, 20% of the participants are women. The whole thing is based on combat veterans, and trauma among combat veterans, and it's like, “Well, women didn't fight in World War II.”


Aubrey: “We're doing a study on the Tuskegee Airmen. And in this sample, we're going to talk through the 50% of them that were white women.” And you're like, “What? No.”


[laughter] 


Mike: We also get in this wave of downfall stuff. We also get the complete collapse of all of this school lunches. Shit.


Aubrey: What happens with the school lunch? 


Mike: Oh, my fucking God.


Aubrey: Is it still in play? 


Mike: The school lunch stuff, the program is now defuncted. I think that whatever, there's probably schools that have bowls of fruit next to the cash register, I think that some of the principles are probably still in play, but as the sort of USDA program, it's gone. Their website, you can only look at it on archive.org. The thing is toast. There's a researcher named Eric Robinson who starts looking into this stuff. The first thing that he finds, it's actually pretty bad and pretty bad that nobody noticed this before, but they started implementing this program in 2014. The first randomized control trials of these principles didn't start until 2014. So, basically, when they implemented this program, they had no data. One of the things that Eric Robinson also notes in his paper is that there's these 15 strategies that schools were supposed to be implementing, it appears to this day, only six of them have ever actually been studied. 


Aubrey: Yeah, that just seems wild. Also, I will say, there is already so much that we expect out of schools. We're expecting teachers and school administrators and school staffers to carry the weight of so many of our social anxieties, and this also becomes another one that they're now wholly carrying. All of our weird shit around health and weight and whatever, that kids generally don't have in the same way, this is part of how we ensure that kids pick that stuff up, and that we sort of projected onto them.


Mike: Also, this was the thing that I was biting my tongue to keep from saying earlier when you were talking about the Elmo stuff. The Elmo study, choose an apple with an Elmo sticker or a cookie.


Aubrey: Yeah.


Mike: It turns out the study wasn't on 8- to 11-year-olds. It was on a 3- to 3-year-olds. 


Aubrey: Woo, that makes more sense. 


Mike: You sniffed it out. Millions of researchers did not sniff this out, Aubrey, you knew.


Aubrey: What? 


Mike: [laughs] 


Aubrey: Truly, anyone who has spent any amount of time around, a 9- or 10-year-old can tell you that even if they still like Elmo, they know that they aren't expected to.


Mike: This is a fucking wild error to make. I thought it was among 8- to 11-year-olds, it was actually 3- to 5-year olds in daycares. So, this has no application to elementary schools. 


Aubrey: Well, and also, again, children ages three to five are in a completely different state of brain development, and impulse control. It's not quite like you should be studying another species, but it is like so far off the mark.


Mike: Well, yeah, the whole concept of choosing food is very different for a 4-year-old versus a 10-year-old. There's also another fucked up thing. This actually shows up in a lot of lunchroom studies. I've been reading studies all week, that in this study where they made the chocolate milk harder for the kids to grab, what happened is they took more milk, so it's like, “Yay, they're taking milk. It's not flavored, it's not sugared,” but then they're not actually drinking it. Getting kids to take vegetables is a totally different thing from getting them to eat vegetables.


Aubrey: It is very easy to get my nephew to put vegetables on his plate. That's fine, he will do that without objection. When you're like, “Hey, you've got to have a couple more bites of vegetables.” He'll be like, “How big of a bite? Let's negotiate.”


Mike: There's also-- this is one of the fucking pettiest things I've ever seen in an academic paper. There are randomized control trials that show that smarter lunchrooms do actually improve the number of fruits and vegetables that kids are eating. This is evidence that Brian Wansink was using for years, and he would talk about it in his TED talks and, like, “We're doing this and it really improves things for kids.” And then, when Eric Robinson goes back to the data, the studies show that after all of these interventions in schools, the biggest sort of study that has been done on this, shows that kids are eating 0.1 unit of fruit more than kids who aren't getting this intervention. He has a whole page of his paper dedicated to photos of this. So, he goes, “Figure 1, a small apple. Figure 2, 10% of an apple.” And it shows like this lonely, little slice of an apple on a plate, and he clearly took the photos at his office. They're in this shitty break room. 


Aubrey: I love everything about this. 


Mike: That's fucking brutal, dude. You could have just said it. [laughs] 


Aubrey: There is a real special place in my heart for the extreme pettiness that is reserved for very high-minded fields. [laughs] 


Mike: Love it. Basically, it appears that all of these interventions increased the amount of fruit that kids ate by 1/10th of an apple. We're not talking about a revolution in children's consumption here. We're talking about extremely modest effects, and that was never how he was describing them publicly.


Aubrey: It's so hard because this feels like a real Emperor's New Clothes kind of an episode.


Mike: Yeah. This is honestly my biggest revelation from this, is that there's this statistical stuff. There's the p-hacking stuff, but the worst thing that he did was out in public. One of the things Eric Robinson mentions is that he would have these studies where we find kids are eating 0.1 unit of apple more, and then in the conclusion of the article, he'd be like, “This is an effective intervention for childhood obesity.”


Aubrey: God fucking damn it. 


Mike: And you're like, “Well, it's very evidently not.”


Aubrey: You have to line up so many dominoes before you can flick one and have them all fall the way that you think they should. This guy essentially set up two dominoes, and they didn't make it past the finish line. And he was like, “We did it.”


Mike: Another example in here is that there's articles where in the abstract, he says that giving kids presliced fruit increased fruit consumption by 71%, and then you read through the paper, and it actually increased it by 4%. [laughs] 


Aubrey: If you and I, as people who are not-- we don't have math degrees, if this stuff jumps up out to you and I just at face value, that feels like, “Uh-oh.” 


Mike: It's not good. 


Aubrey: It's not great.


Mike: This is actually the next stage of the downfall. There's this yearlong period where the Pizzagate stuff is happening, and there's statistical analyses. And there's all of this sort of behind-the-scenes intra-academic debates about this Brian Wansink guy, but it hasn't really bubbled up to the surface, sort of normies, we're not really noticing this. This is so weird, the next stage of this downfall basically happens with a tweet thread by The Joy of Cooking Twitter account.


Aubrey: [laughs] Oh God, I really love the idea of a nutrition researcher getting owned by The Joy of Cooking.


Mike: Fucking owned.


Aubrey: By my grandma's cookbook. [laughs] 


Mike: So, do you remember The Joy of Cooking study that he did? 


Aubrey: Sure. 


Mike: They had of course seen his study. They'd seen it's been cited more than 30 times, it shows up in media reports. It's something that like everyone just mentions and cute little parentheses, whenever Joy of Cooking comes up. They're like, “Well, the portion sizes are so much bigger.” There's this great New Yorker article by friend of the show Helen Rosner, who interviewed people at Joy of Cooking about what it felt to be the target of this study and trying to debunk it for years, and the world wasn't ready. Finally, after things start to domino out of place for Brian Wansink, they put out this thread saying that, first of all, the entire idea of his original study was that there were all these different recipes throughout time, he could only find 18 that appeared in every single Joy of Cooking to compare to each other. The Joy of Cooking is like, “We found 275 recipes that were in every single edition.” And then, they do this whole thing about-- it's absurd to say that portion sizes have increased when most recipes in the book aren't meant to be eaten all at once. One of the recipes that he says has gotten bigger over the years is gumbo. It's like a big ass pot of gumbo.


Aubrey: Yeah. Look, my reference point for this whole section is just going to be The New York Times cooking section and the comments on those recipes. Have you ever looked at the comments in The New York Times cooking section?


Mike: Oh, aren't they like, I replaced the chicken with beef and the rice with Styrofoam chips? It's all like I modified it.


Aubrey: And it was not very good, one star. [laughs] Well, come on, dude. That feels the part of the challenge here. It's like you can't and don't know. Also, “Hey, Brian Wansink, you love your plate sizes so much, how big are the plates that people are eating the gumbo off of, hmm?” Good Lord.


Mike: That study was basically trashed. This is the overall frustration. There were dozens of articles written about this. They're like Joy of Cooking portions are getting bigger. It's just not a story that means anything, and it never was. 


The final chapter of the downfall, and I think probably the most important one, is a reporter named Stephanie Lee at BuzzFeed does a public records request. A lot of the people that Brian Wansink has been corresponding with are employees of universities, and universities are public institutions, and you can FOIA them. So, she gets this huge trove of emails from New Mexico State University where one of his collaborators works. She basically finds that all of this was totally deliberate.


Aubrey: What?


Mike: He was basically running this lab as a publication factory in a very explicit way. There's all these emails about-- the same way that it was with the pizza study, where they've gathered all this data, and then whatever grad student gathered it, has moved on. And then, he'll assign another graduate student to mess with the data until they can get something publishable out of it. And so, he says in one of the emails, "A lot of these papers are laying around on our desktops, and they're like inventory that isn't working for us. We've got so much huge momentum going. This could make our productivity legendary.” 


In one case, there's this data that’s how people shop in grocery stores where they don't speak the language. And it's just like, "Isn't all that interesting? They didn't really find-- nothing jumped out of the data?" And he's like, “Oh, we'll keep looking at lower and lower tier journals until you can get it published.”


Aubrey: This is such a weird capitalism-forward approach to scientific findings. It's really odd. That feels really odd to me.


Mike: Also, media forward, because in a lot of these emails, people will come to him with ideas and be like, “Ooh, that'll definitely go viral.” This is a very explicit goal of his. He's even at one point training his grad students how to pitch their findings to the media. They'll have little sessions, where they'll practice their elevator speeches to media sources.


Aubrey: Which isn't in and of itself nefarious. You practice elevator speeches for all kinds of stuff. But again, if that is the focus, you're not going to have findings that are as useful or as solid as you would want them to be.


Mike: Eventually, The Cornell Sun, the newspaper, the university interviews his former grad students, and they say that they felt really uncomfortable with the way that they were pressured to manipulate data, to frame things for the media, to try to get these things to go viral. It seemed like this was a very well-known problem among people who worked with him.


Aubrey: Yeah. There's this intense power dynamic there, that he is a nationally renowned researcher, you are a grad student or an employee who is working for him. This is the person you would go to for a reference. There is not a neutral power relationship here.


Mike: I had a bunch of other studies that I was going to debunk, but I think you get the idea. Most of the stuff doesn't hold up. I feel there's the debunkable articles. But then, I think the much bigger problem with his work is this thing of him deliberately framing articles to get media play, and a lot of these articles, a lot of them are just fucking dumb. There's a really interesting article by this child’s marketing researcher guy. This is way before this sort of the scandal happened. This is in 2014, who looks into Brian Wansink. Remember the study on cartoon eyes looking downward at children? 


Aubrey: Uh-huh. 


Mike: This actual marketing researcher writes this long essay who's like, “This is dumb. This is not a real study. You can't measure the angle of eyeballs in cartoon characters.” He includes photos with a Trix rabbit. The Trix rabbit is looking upwards, but his eyes are tilted down. And he's like, “Well, does this count as down or straight on?” You can see him sort of sputtering in the text. He's like, “This is dumb. Why was this published?” And, of course, the reason why it was published was because you get media out of it.


Aubrey: Yeah. I could totally see as a marketing person that you'd be like, “We're not doing that. We're doing other things but we're not doing that.” What? 


Mike: Well, manipulating children in other ways. 


Aubrey: You're making us look evil in the wrong ways. 


[laughter] 


Mike: In 2018, he's pushed out of Cornell. In September of 2018, the Journal of the American Medical Association retracts six of his papers. These grad students, there's a running tally of all of the articles with plagiarism and conclusions that aren't supported by the data, etc., and it's up to 52 publications. Some of them are self-plagiarism, which I honestly don't put in the same category as faking your data kind of stuff. But also, they don't include the papers were-- they just shouldn't exist, like the cartoon study. There's been 13 articles that have been officially retracted, and 15 of his articles have been officially corrected.


Aubrey: Out of how many articles altogether?


Mike: He actually says on his website, he has like a brianwansink.com. He says, “7% of my research articles were retracted.” 


Aubrey: Oh, brother. 


Mike: That's his little way of downplaying how severe this is. But it's not clear how many articles people have looked into. The denominator is not his total body of work. The denominator is how many did people investigate, because I noticed that none of these debunkings related to his work on workplace wellness. So, I was like, “Okay, Brian Wansink Workplace Wellness,” just googled it, found a random paper. And then, the sample sizes didn't add up in the first paper that I looked at, and it also had conclusions that were not remotely supported by the data at all. Again, the first one that I looked at. I think people just got sick of debunking this guy's papers after a certain point, but I think that a much larger number of them also wouldn't hold up to scrutiny if you took a look.


Aubrey: Yeah. I think that makes sense. Also, I would not be surprised if he was like, "7% of my papers have been questioned or discredited," or whatever. And then you look at the data set and you're like, “It can't be 7%.” 


[laughter] 


Mike: That’s mathematically impossible, Brian. Have you learned nothing?


Aubrey: I think it's worth looking at him and his work very closely, and I also think in the same way that his work encourages us to look at folks on an individual level, and there's an impulse to resist there, I think there's an impulse to resist here, which is only looking at him and not also looking at like, “Hey, all of this stuff passed muster for all the systems that we have. This went through the entire peer review process. This got published in the Journal of the American Medical Association, the New England Journal of Medicine," whatever the biggest journals are. That also feels a really challenging layer to add on to all of this, is this isn't an exception, this isn't someone who's skirted the system, this is someone who went through the system, and this is what came out.


Mike: Exactly. I think there's a huge media story here too in that the New York Times should not be quoting anybody 60 times. That, just to me, is just a huge red flag when you're going back to the same source over and over again, because then you have this mutualistic relationship between the journalist and the researcher. I listened to a really interesting podcast series on the The Dr. John Berardi Show about the scientific debunking as an institution. They said something really interesting. They said that, “In fields of science, where not very much is known, there's just a lot of mysteries still to solve, it's oftentimes the most confident people, not necessarily the most knowledgeable people or the most careful people, who get the most attention.” I think that Brian Wansink really took advantage of that, that he's like a good public speaker, he's handsome, he's straight and white and cis. He's just out in public giving good TED Talks, writing very pop easy-to-read books. 


I still cannot get over the fact that all this stuff about weight loss, use smaller plates, eat one fewer candy bar a day, you’ll lose 27 pounds, he never, it appears, even attempted to test any of this. That’s really incredible to me, because his entire thesis, his entire career was dedicated to this idea that we can make inadvertent small changes to our food environments and lose a bunch of weight. Well, why not take 100 families and swap out 50 of them with smaller plates and see what happens? People can't stay on the Atkins diet for six months, people can eat off of smaller plates for six months. That's actually a pretty easy test to do. And yet, he never even tried.


Aubrey: We talk in food world and nutrition world about health halos. Things that seem to take on more value and almost more moral value because they appear to be healthy. There's a little bit of a health halo effect with nutrition research. We are all in this sort of constant state of desperation for more concrete answers than the various and sundry diet marketing that we're exposed to. I think when someone comes along who is from the academy, who is trained in the scientific method, and does something that seems official and more concrete, we put folks who do that research up on a pedestal and put that research up on a pedestal in a very uncritical way.


Mike: Yes. This is the story of how far you can go if you are telling people things they want to hear. I should also mention, I can't believe you just brought up the term 'health halo.' Do you know who coined the term 'health halo'? Brian Wansink.


Aubrey: No, fuck. Come on, really? 


Mike: Yeah, dude.


Aubrey: Michael.


Mike: This brings us to our happy epilogue. Do you want to hear the happy epilogue? 


Aubrey: Ooh. Let's hear it. 


Mike: I was totally surprised by this, actually. In the research for this episode, I did a lot of reading on school lunches, generally. Did you know that school lunches have gotten way better? 


Aubrey: Really? 


Mike: Yes. In 2010, the Obama administration passed the Healthy, Hunger-Free Kids Act. There's all these studies showing that kids are eating more fruits and vegetables. Kids are eating more whole grains. A lot of schools took chocolate milk and strawberry milk out of schools. A lot of them took soda out. There's all this data now showing the average school lunch is significantly more healthy than the average bagged lunch.


Aubrey: I'm so glad to hear it. It sounds more nutrient-dense foods. There are more foods with fiber in them. There are fewer foods that are just way high sugar.


Mike: Yeah, a lot more food is made from scratch now. It's obviously not perfect. It's not as good as other countries. There's still huge inequalities. It's not perfect but I do think that it's worth noting sort of these kinds of improvements. I also think that it's worth noting, like what improves kid's health, and it's laws and fucking money. One of the main things that went along with this act was way more money for school. I think it's $3.38 per meal, and it used to be like $1.30 per meal. Yeah, when you give schools more money to feed kids, they feed kids better.


There's all the hidden stuff with Brian Wansink’s work, but then the more visible stuff is this is something he was not interested in at all. He goes out of his way throughout both of his books to be, like, “Oh, we shouldn't take chocolate milk away from kids, because then they'll avoid school lunches altogether. We don't want to change anything. That's too paternalistic.” And it's like, “No, Brian, we should take the chocolate milk away. I feel fine about there not being chocolate milk in schools, dude.” 


Aubrey: Also, we live in a world of forced choices. It's happening all around us all the time. That is part of the central conceit of his work. It seems really weird to say, “We're going to stop short of forcing choices," even though all of his research is forcing choices.


Mike: Exactly. The implication, the obvious implication of his work is that we should force choices, that these are designed environments. There's no such thing as a non-socially engineered food environment. We are surrounded by social engineering at all times. We might as well engineer good environments, especially for kids. But it's like, as soon as it came to any actual mandatory thing of forcing kids to not have those choices, he would completely freak out and be like, “No, no, no, the corporations have to be our partners.” It's very clear what his actual project was the entire time. 


Aubrey: Ultimately, these are incredibly complex issues, and to focus just on the individual choice/life hack stuff feels very shortsighted to me.


Mike: In conclusion, don't life hack, write to your senator, and make something from the 1936 Joy of Cooking tonight.


Aubrey: Also, if there are any researchers or science reporters listening, put Elmo stickers on the good research.


Mike: [laughs] 


[music] 

People on this episode

Head Shot

Aubrey Gordon

Host
Head Shot

Michael Hobbes

Host