Katy Milkman: Imagine you run a small sales force and have a policy of calculating everyone's sales numbers at the end of each month. To prevent slackers from staying that way, after each tabulation, you pull the worst-performing employee into a private conference room for a stern conversation. Now imagine you run the numbers to try to assess whether those stern talks are having their desired effect. You discover that your worst performer is almost never the same person twice, and that in fact, there's a very reliable month-over-month improvement in the performance of whoever you pull aside.
You'd be pretty excited, right? Clearly your tough talk is getting results. This story is actually not so hypothetical. In fact, when he was a pilot in the Israeli Air Force, economics Nobel laureate Danny Kahneman heard a similar story from a flight instructor who insisted that when he gave harsh feedback after novice pilots had a bad flight, he saw big benefits.
Danny relates this story in his bestselling book Thinking, Fast and Slow, but Danny was pretty sophisticated about statistics and had a feeling that there was an error in the logic that was leading to this instructor's conclusion about the benefits of delivering criticism. In fact, he was quite confident it wasn't the tough talk that was improving the performance of struggling pilots. It was something else. In this episode, we'll talk about the fallacy that tripped up that Israeli Air Force flight instructor and my hypothetical sales manager.
It's a misconception that can affect scientific research, investment decisions, and even how you interpret the performance of your favorite athletes.
I’m Dr. Katy Milkman, and this is Choiceology, an original podcast from Charles Schwab. It's a show about the psychology and economics behind our decisions. We bring you true stories about high-stakes choices, and then we examine how these stories connect to the latest research in behavioral science. We do it all to help you make better judgements and avoid costly mistakes.
Albert Chen: Sports Illustrated was my first job out of college. It was kind of my dream job to have the opportunity to work at this publication, which at the time drove the conversation within the sports world.
Katy Milkman: This is Albert.
Albert Chen: My name's Albert Chen. I am a writer, podcaster, and formerly a senior editor at Sports Illustrated magazine.
Katy Milkman: Albert spent nearly 18 years at the storied magazine, and he appreciated the influence it had on the world of sports.
Albert Chen: Sports Illustrated was a huge driver of just what people were talking about, who the biggest stars were. Just to give you an idea, in its heyday, Sports Illustrated had 20 million readers.
Katy Milkman: With that kind of readership, being featured on the cover was a huge deal.
Albert Chen: Simply because of how big of a publication it was. We're talking about a time before Netflix, before podcasts, before ESPN streaming and streaming services.
Katy Milkman: And of course, it remains a very big deal to be a Sports Illustrated cover model, as we saw when the magazine recently placed 81-year-old Martha Stewart on the cover of its swimsuit issue. That decision was front page news around the world. You'd think it's a pretty big honor to appear on the cover of Sports Illustrated, but not everyone's keen on the idea.
Albert Chen: In 1996, the New England Patriots were on a run to the Super Bowl, and their coach was Bill Parcells, this legendary gruff coach who suffered no fools and was an intimidating presence. But after the Patriots won the 1996 AFC title game, which got them to the Super Bowl, he phoned his daughter Jill, who worked at the time at SI's marketing department. And after he got Jill on the phone were just two words, "no cover."
Katy Milkman: The Patriots were going to the Super Bowl. So why in the world would coach Bill Parcells not want to be on the cover of the biggest sports magazine in the world? It was because of the Sports Illustrated jinx. Legend had it that individuals or teams who appeared on the cover would be subsequently cursed. Their fortunes would fade after a cover feature. Their shot at winning big events like the Super Bowl or Wimbledon or the PGA Championship would be doomed. Let's look at a couple of examples of the jinx in action, starting with Andy North.
Albert Chen: In 1978, Andy North was a 28-year-old golfer who had never won a major tournament. To the surprise of the golf world, Andy North won his first major tournament, the U.S. Open, which is one of the biggest golf events of the year. He's on the cover of Sports Illustrated. It's a shot of him making a critical shot on the 18th hole to win the tournament, and it was a big deal. I mean, an American winning the U.S. Open is always a big deal, and it was a big surprise that he came through in the clutch.
Katy Milkman: It was a spectacular achievement. North had never won a major championship and he'd only had one minor tournament win before the US Open. He seemingly came out of nowhere, and then he fell back into nowhere after the Sports Illustrated feature.
Albert Chen: He failed to win another event on the PGA Tour for seven years, and that is a very long time when you're a 28-year-old. And not only did he not win a major tournament like the Masters or the U.S. Open, but he failed to win any tournament period. And to go seven years without winning one after winning one of the biggest events of the year is a pretty significant and notable drop-off, and that's a very long stretch to not win a single tournament.
Katy Milkman: But then Andy North's story took another surprising turn in 1985.
Albert Chen: He wins the U.S. Open again, and it's his second major tournament win. It's another U.S. Open, and again, he makes the cover of Sports Illustrated. Not very many golfers make the cover of Sports Illustrated, and so for Andy North to make it again in 1985 tells you that he was a big national story at the time.
Katy Milkman: But again, Andy North's time in the spotlight would be short-lived.
Albert Chen: That would turn out to be the last time he would ever win any sort of PGA golf tournament for the rest of his career. The jinx turns out struck twice when it came to Andy North, unfortunately.
Katy Milkman: The Sports Illustrated jinx appeared many times over the years. There are numerous examples of athletes and teams who made the cover and then flubbed a game or had a disappointing season or bowed out of sports altogether. This next example is a kind of mirror image of the Andy North anecdote. Serena Williams didn't come out of nowhere in this story. In the early 2000s, she was already the biggest women's tennis star in the world. But in 2015, she reached a new level.
Albert Chen: Serena was on a remarkable run. She was ranked number one in the world for the third longest run in women's tennis history. She had at one point a 27-match win streak, which is very, very hard to do in tennis. And she won the Australian Open, the French Open, and Wimbledon, the first three major tournaments of the year, and was on the cusp of doing something that no tennis player, male or female, had done in over 27 years, which was win tennis' Grand Slam, which is winning all four major tournaments. She was having arguably the most dominant season of any tennis player, male or female.
Katy Milkman: It's no surprise then that Sports Illustrated would want to have her on the cover. She was big news in 2015.
Albert Chen: Headed into the final major tournament of the tennis season, which is the U.S. Open in September of 2015. In August of 2015, Serena was on the cover of Sports Illustrated, which was promoting her possible and some would say very likely Grand Slam victory. The cover line was "All Eyes on Serena." It had become a story that was bigger than tennis, that was bigger than sports and drawing in attention from all media across the world because of this historic run. Getting a ticket to the semifinals and the finals of the U.S. Open that year, it was the toughest ticket in sports.
It far dwarfed whatever was going on in men's tennis, certainly at that point, but it was just the biggest story in sports, period. I just remember every match of hers at the U.S. Open was just this star-studded event with celebrities in attendance. Every match was just this huge deal because everyone wanted to be watching live when Serena made history, because it did seem inevitable just because of how dominant she had been through that entire 2015 season.
She had reached the semifinals of the U.S. Open and had won a tough match against her sister Venus and others throughout her U.S. Open run that presented the biggest challenges for her to actually accomplishing the Grand Slam and winning the U.S. Open title. When she reached the semifinals, and when everyone looked at who was in the semifinals with her, it really just felt inevitable that she was going to really cruise to winning the title, and she was facing in her semifinal match an Italian player named Roberta Vinci, who was an unseeded player.
She pretty much wasn't on the radar of anybody and certainly seemed incredibly unlikely to beat Serena Williams in the semifinals. I recall watching the match on TV and just being completely stunned by it. It was a three-set match where Serena was just making these—uncharacteristic for her—errors, hitting balls into the net, hitting balls long, double faulting. It wasn't the Serena Williams that you were accustomed to seeing. Based on the reaction on Roberta Vinci's face after she won match point, she was completely stunned and shocked by the moment just like everybody else.
Serena, as you could imagine, she was in tears. She was completely devastated by this moment, by losing. The reaction was just like, "Did that just happen? Did Serena lose to Roberta Vinci? Is this how this incredible run is really going to end?" Yeah, it was a completely shocking sports moment.
Katy Milkman: It was one of the biggest upsets in tennis history, and it happened right after Serena appeared on the cover of Sports Illustrated. Unlike Andy North, Serena quickly returned to her winning ways.
Albert Chen: She went on to win Wimbledon in 2016, and she would go on to win the Australian Open in 2017.
Katy Milkman: That said, she would never recapture her record-breaking performance in that remarkable 2015 season, an incredible year even with that U.S. Open loss. Over the years, Albert has thought a lot about the many examples of superstars appearing on the cover of Sports Illustrated and then experiencing a setback. In fact, he co-wrote a story on the Sports Illustrated cover jinx for the magazine back in 2002. The question was, does the jinx really exist?
Albert Chen: My role with another fellow researcher named Tim Smith was to go through all the covers in the history of Sports Illustrated up until that point and to see if the cover subjects were jinxed or not. This was an age well before digital archives, so we went through the Sports Illustrated library of archived issues, and one by one with legal pads in hand going through and documenting the subject on each cover of an issue.
The magazine's first issue came out in 1954, so there were 2,456 cover subjects. What our results showed was that there was a jinx rate of 37.2%, which I'm laughing at the specificity of that number. You have to be precise when you're talking about the SI cover jinx.
Katy Milkman: 37.2% of athletes saw a game or tournament loss or a decline in performance, or they experienced an injury in the weeks and months following their cover story.
Albert Chen: In other words, 913 covers featured a person or a team that had some measurable form of misfortune that essentially reached our definition of a jinx.
Katy Milkman: Albert and his colleagues came to believe that the jinx was actually a real thing, but they were pretty skeptical about some of the superstitious ideas explaining what caused it.
Albert Chen: There's a lot of superstition in sports, whether it's Tiger Woods insisting that he wears a red shirt on the final day of every tournament that he plays in, or a baseball player insisting to use his favorite at bat. There's a story of Michael Jordan always wore his University of [North] Carolina shorts underneath his uniform every game. I don't believe that stuff has an impact on the black-cat sort of level of superstition, but I do believe the jinx does exist.
In that, for starters, they are on the cover at a point in time when fame is coming suddenly at them, and there's this new level of stress and pressure and expectation. The second reason is that the jinx is really nothing more than what statisticians call regression to the mean, and others would call water seeking its own level.
Katy Milkman: For golfer Andy North, whose wins were exceptions to his average performance, it makes sense that he would return to a normal level of winning, which is to say not winning much at all between and after his cover appearances on Sports Illustrated. For Serena Williams, no matter how outstanding she was, there was still always game-day luck. Landing on the cover at such an incredible high point in her career, it's no wonder she had a bit of a disappointing loss shortly thereafter.
A perfect streak can't last forever, and her incredible 2015 season was an outlier in an extraordinary career, but not one without occasional defeats. As Albert mentioned, a Sports Illustrated cover came with real pressure from the press and from fans, especially for rookie athletes, and that pressure could potentially have negatively affected players' performance on the field or the court.
But often the jinx was just that the cover issue coincided with a pinnacle of success for the athletes featured. Odds were that their next game or match would not be quite as good as the amazing one that had landed them on the cover.
Albert Chen: We're capturing these teams and these athletes at the moment of their peak, and they have nowhere else to go but down.
Katy Milkman: Albert Chen is a writer and podcaster and formerly a senior editor at Sports Illustrated. He's the author of the book Billion Dollar Fantasy. You can find links to Albert's work and his book in the show notes and at schwab.com/podcast. As you heard Albert mention, one of the explanations for the so-called Sports Illustrated jinx is related to the mere fact that athletes only manage to land on the cover of the magazine after an extraordinary run of success, and perfect performance just isn't something that can last in sports or in any other arena of life.
The technical term for what Albert is describing here is regression to the mean. It's actually the very same concept that convinced the flight instructor Danny Kahneman encountered in the Israeli Air Force, that negative feedback was phenomenally helpful to his student pilots. It's the reason a sales manager who denigrates their lowest performing salesperson each month could expect to see that salesperson's results reliably improve after the tough talk.
I've invited an expert in statistics to join me to explain regression to the mean, which pathbreaking behavioral scientists Danny Kahneman and Amos Tversky proved most people tend to neglect in a seminal 1974 paper. We'll talk about what it is, the fact that most people neglect mean reversion, which can be perilous, as well as specific ways in which ignoring regression to the mean can have negative consequences.
Elizabeth Tipton is an associate professor of statistics and data science at Northwestern University. Hi, Beth. Thank you so much for taking the time to talk to me today.
Elizabeth Tipton: Hi, Katy. I'm delighted to be here.
Katy Milkman: I wanted to start by asking if you could define regression to the mean for our listeners. What is it?
Elizabeth Tipton: Regression to the mean is a statistical phenomenon. It occurs whenever we see repeated measures on the same subjects, which we might think of as people. The basic idea is that the measures at two time points or two or more time points are not perfect measures. There's some error in the measures. And as a result, those people who score very low or very high on the first measure are more likely to have less-extreme measures on the second measure. That's to say they're more likely to regress back towards their mean on the second measure.
Katy Milkman: If you were looking at two students, say, or actually let's talk about 100 students, and gave everybody the same math test on Monday, and then you gave it to all of the students again on Tuesday, what are the implications of regression to the mean for thinking about the data you'd collect on Monday giving math test one that's supposed to measure a concept, and then on Tuesday giving math test two that's also supposed to measure that same concept?
Elizabeth Tipton: In the perfect case, your math test has no error in it, and it perfectly captures every person's true knowledge of math. In that case, how they score on Monday and Tuesday should be the same. Kind of a worst-case scenario is that it's just all noise. It's all just random error, in which case, how you score on Monday and Tuesday, completely unaligned. What we see in real life is something in between those two extremes, which is that they're correlated with each other, but they're not a perfect correlation.
People who score really well on Monday, let's say your top achiever on Monday's test, some part of that achievement and their top score is their true score, that they're really good at math, but some part of that is maybe an unusually lucky day, that they had breakfast, and they were feeling great, and they happened to get items that were perfectly aligned to things that they knew.
Whereas on Tuesday, they maybe don't have that same luck part, so they still have their strong math knowledge, but they don't have as much luck on Tuesday, and so maybe their score is slightly lower and closer to the mean than it was on Monday. One of the things I wanted to emphasize here that I think people often get confused about is that because it says regression to the mean, it makes you think, well, everybody is moving back towards the average, right?
You think, well, but there must be some people who are actually above average and some people who have below average, I don't know, whatever the skill is or that's being tested.
Katy Milkman: Baseball batting averages.
Elizabeth Tipton: Right, yeah. I was thinking of bowling as my example here. My husband did a lot of bowling as a teenager, and so he's quite good. When we go bowling, he will get a strike, and then it will be followed by a spare. He starts out strong with that strike, and it's likely to be followed up with some pretty good frames thereafter. I am maybe one of the worst bowlers ever.
Katy Milkman: I'll fight you for that title.
Elizabeth Tipton: I can easily have a run of gutter balls and then all of a sudden get a strike. When I get that strike, for me, that is completely random luck. I am not doing anything on purpose to get that strike. That is just me getting a really lucky hit basically. It's very common that the thing that comes after that is another gutter ball, not that I've suddenly become good at bowling, though that is sometimes what I want to think, right?
It's clearly luck, whereas there is actual skill. My husband does better or worse. He's not always getting strikes or something, but he's more likely to hit nine pins on average than I am. What I'm trying to emphasize is that everybody has kind of a true score.
Katy Milkman: Skill, your skill level, how good you are at …
Elizabeth Tipton: Exactly. But then also in measures, because it's really hard to measure your actual skill or these actual things that you know, there's also this little bit of luck. It can be depending on the measure, a lot of luck or a little luck can be involved in that, depending on the test is long or it's short or how well it's made. But we can all have lucky days and unlucky days on measures.
The reason that good or bad luck matters and why regression to the mean matters is if we are selecting people, and we're selecting people based on this noisy measure, and we're selecting the lowest or the highest performers, we're set up in a situation in which we've selected on people's luck. We've either selected their best luck or their worst luck in those cases. It's likely to be followed up on the next time by them being closer to their true score, less unlucky or less lucky, than they were the previous time.
Katy Milkman: What is your favorite example of a situation where, if people ignored regression to the mean, they might make real mistakes?
Elizabeth Tipton: I think we see this with test scores a lot. You take people who scored very poorly on a fall test, for example. Let's say students take the MAP test.
Katy Milkman: Sidenote to listeners, the MAP test is a widely used student assessment in K through 12 schools that's designed to measure academic progress. It's called MAP to abbreviate Measures of Academic Progress.
Elizabeth Tipton: They don't do well on the fall test, and you do whatever it is you think you're going to do to help those kids, and you're focused only on the kids that were low performers. Maybe your school says, "Let's focus in on the bottom 20%. We've really got to raise their scores," and you do whatever it is that you're going to do to help them out.
And then you take them on the winter test, they are likely to increase their score partly because you took these kids who did really poorly, the ones at the very bottom. Those at the very bottom are a mix of those who really are struggling and don't know the material and some kids also who it's a mix of that plus some really bad luck. Maybe they were really tired the day they took the test, for example.
Katy Milkman: That's a great example when it would be a problem to ignore regression to the mean if you only focus on helping the kids at the bottom and you miss some who are close to the bottom.
Elizabeth Tipton: Right, or you focus on them and then you think that whatever you're observing, that whatever you did, must have made them better, right?
Katy Milkman: Right, because it might just be regression to the mean. You took the extreme outliers, and the next time they look better. I'm curious if you feel like there's a way that it clicks for you when you teach students that they need to pay attention to mean regression when they're looking at baseball statistics or the stock market performance of different hedge funds or how well different stores’ sales will continue to look a year out.
Elizabeth Tipton: Yeah, that's a great question. I'll say two things here. I think one is, in general, I think reminding people constantly, my students and even experts, it's easy for all of us to make these mistakes, that any measure we see includes some part that is true and some part that is random. Anything you see is not measured perfectly, and it's easy to be tricked into thinking that everything we see, that's a true measure. Reminding yourself, wait, that's probably an imperfect measure.
And if it's an imperfect measure, I should be hesitant to believe that these extreme values based on one measurement are likely to be consistent. The other part that I try to emphasize too, and this is partly because I think I was confused myself about regression to the mean for a long time, this idea that we would all regress back to the average seemed very confusing to me. Why would we all go back to the average? Surely there are some people that are better at baseball than others, for example.
Framing it as each person has their own mean and that there are people who are better at baseball. Also, those people who are better at baseball have some good days and some bad days. There are people like myself who are terrible at baseball and really all sports have some good days and bad days. It's almost more intuitive if you focus on the individual and that this is one measurement of many measurements for that individual can help you tease apart those two things, that some part of this is true and some part of this is just good or bad luck.
Katy Milkman: What do you think our listeners should do differently understanding regression to the mean and people's general tendency to neglect this phenomenon? How can we live our lives in a more productive way with a little knowledge of this issue?
Elizabeth Tipton: If you're in a situation in which you are looking at the top scores for something or the bottom scores, remind yourself that there is noise. Remind yourself that some kids or people had good luck, and some had bad luck. And that as a result, you should not make a major decision based on one observation. If you do want to make that decision based on the observation, you need to pause and remind yourself how are you going to take that luck into account?
Katy Milkman: I love that answer, and it's a perfect place I think to wrap up. Thank you so much for taking the time to talk to me today, Beth. This was really enlightening.
Elizabeth Tipton: You're welcome. Thank you for inviting me.
Katy Milkman: Elizabeth Tipton is an associate professor of statistics and data science at Northwestern University. You can find links to her work in the show notes and at schwab.com/podcast.
As I mentioned earlier, mean-reversion neglect is a bias that was first brought to the attention of behavioral scientists in 1974 by psychologists Amos Tversky and Danny Kahneman. Because we tend to neglect measurement error when thinking about the best way to interpret data, we overlook the reality that extreme performance typically reverts back toward the mean.
Mean reversion can easily explain why landing on the cover of Sports Illustrated appears to be a curse. You have to have an extraordinary run of good luck combined with excellence to attain the kind of success that leads to a cover story. And after that type of outlier event, simple statistics tell us that your luck and performance will most likely revert towards the mean, but understanding mean reversion and curing yourself of the tendency to neglect it won't just help you think more logically about the fallacy that it's bad luck to end up on the cover of Sports Illustrated.
And it's not just a bias with implications for interpreting people's successes and failures. Regression to the mean is relevant for thinking about the performance of stocks, the weather, traffic patterns, and really any repeated observations you make of the world. The top-performing hedge fund this year is unlikely to be the top-performing hedge fund next year, not because of a jinx but because of regression to the mean. A town that sees more snowfall than ever before is unlikely to be equally inundated the next year.
Once you appreciate that outliers tend to regress toward the mean, you can make better predictions about the future and draw more logical conclusions about how your handling of extreme performers actually affects their subsequent success. None of this is to say that excellence or poor performance is always an outlier. Skill is very real. Differences in things like weather or traffic conditions in different locations are no joke, and these things can and do also change over time.
It's just that almost all observed outcomes are a combination of true quality at a point in time and luck. When wouldn't you see regression to the mean? Well, when there's really no luck. For instance, if I measure the height of every MBA student in my class on Monday to the nearest inch using an excellent ruler, and then measure their height again on Tuesday, I wouldn't expect to see much regression to the mean because luck doesn't change your height on a one-day timescale once you're fully grown.
But if I measured my student's skill at flipping heads with fair coins, it would be all luck, and Monday's performance would fully regress towards the mean on Tuesday. Hopefully, this episode will help you appreciate the combined forces of nature and luck in most of the outcomes you see around you, and hopefully you'll become less bullish about the impact of berating low performers and less suspicious about the Sports Illustrated jinx. Neither is likely to matter anywhere near as much as you thought.
You've been listening to Choiceology, an original podcast from Charles Schwab. If you've enjoyed the show, we'd be really grateful if you'd leave us a review on Apple Podcasts, a rating on Spotify, or feedback wherever you listen. You can also follow us for free in your favorite podcasting app.
And if you want more of the kinds of insights we bring you on Choiceology about how to improve your decisions, you can order my book, How to Change, or sign up for my monthly newsletter, Milkman Delivers, at katymilkman.com/newsletter. Next time you'll hear about the mistakes we make when we receive new information. I'm Dr. Katy Milkman. Talk to you soon.
Speaker 4: For important disclosures, see the show notes or visit schwab.com/podcast.
