Announcer: In the beginning, there was a vision, a bold plan to turn DC Comics and its characters into larger-than-life mythical heroes on film, an operatic saga of gods amongst men complete with darkness and hope, sacrifice and destiny. The monumental task of shaping this cinematic universe fell to visionary director Zachary Edward Snyder. An ambitious serialized arc began to build around their central hero, Superman. But up, up high in the ranks of Warner Bros. Studios, a battle of ideals was raging, a war that would ultimately shake the very foundation of what Snyder and his team were crafting, the DC Extended Universe.
Katy Milkman: A lot of the time, the drama in Hollywood extends beyond the screen. From production budgets to box office earnings, Hollywood movies are a high-stakes business gamble. On screen, the characters tell the story, but off-screen at the box office and in critical reviews, the numbers tell the story. In this episode, we'll look at our very human tendency to fixate on numbers, why we often value hard numbers over words or graphs or other equivalent representations of the very same information, and what we should be aware of when we face a decision involving numbers, numbers that might not tell the whole story.
I'm Dr. Katy Milkman, and this is Choiceology, an original podcast from Charles Schwab. It's a show about the psychology and economics behind our decisions. We bring you true and surprising stories about high-stakes choices, and then we examine how these stories connect to the latest research in behavioral science. We do it all to help you make better judgments and avoid costly mistakes.
Stephen M. Colbert: The idea was let's bring in these filmmakers who have their own unique takes on things and have their own vision and voice and give them these big budgets to make these huge blockbuster movies.
Katy Milkman: This is Stephen M. Colbert, not to be confused with comedian and late-night host Stephen Colbert.
Stephen M. Colbert: My name is Stephen Colbert. I am a writer and podcaster.
Katy Milkman: Stephen is a commentator on and close observer of the film industry. One of his podcasts breaks down the movie Batman v Superman minute by minute, so he pays very close attention to superhero films, in this case from the DC Extended Universe.
DC is the comic book publisher, which is now a subsidiary of Warner Bros, and it's home to superheroes like Wonder Woman, Superman and Batman. Before the Batman v Superman movie, there was the first DC Extended Universe film, Man of Steel, a film that was greatly overshadowed before it was even released.
Stephen M. Colbert: Man of Steel was going to come out in 2013, so it was already mostly produced and in post-production, and The Avengers came out, and that was the first Marvel Cinematic Universe crossover movie directed by Joss Whedon.
Katy Milkman: A quick aside for anyone who's not familiar, both Marvel and DC started out as competing comic book companies. Now they're also competing movie companies, and they each have their own roster of superheroes. Spider-Man, Black Panther, Iron Man, they're on the Marvel side. Flash, Aquaman, and Superman are all DC characters. It's OK if you can't keep them all straight though. This story will still make sense.
So Man of Steel, a Superman movie from DC Comics, was released after the blockbuster hit Marvel movie The Avengers. And while Man of Steel did pretty well at the box office, it paled in comparison to The Avengers.
Stephen M. Colbert: The Avengers made $1.5 billion at the global box office. It was a huge deal at the time. Everyone wanted their new Avengers.
Katy Milkman: So now Warner Bros. Studios, with its DC Extended Universe movies, was aiming for Marvel numbers.
Stephen M. Colbert: When you're trying to make a film like that that has that big market appeal, you're looking for something that's harder for audiences to dislike. You're not taking big swings. That means you need all demographics to be appealed to in order to fill seats as opposed to saying, "What's the most interesting, unique, best movie we can make?"
Katy Milkman: The filmmaker involved, Zack Snyder, was not a big-market-appeal kind of director. His work is darker in tone, more mature.
Stephen M. Colbert: Zack Snyder is really interesting. People use the word "visionary," which once again isn't necessarily a term of quality. It's just a term of approach where he's got a movie in his head. He'll have a story, and he'll know what it looks like. He knows what it smells like. You'll see a shot, and there'll be an alleyway. He could tell you what's around that alley even if the camera never sees it.
Katy Milkman: He creates unique and striking worlds, but his approach is not for everyone, which means it's not necessarily a recipe for broad appeal.
Stephen M. Colbert: Zack Snyder's polarizing nature, Rotten Tomatoes isn't very friendly to that, and so the reviews that liked it did really like it, and a lot of fans really liked it, but because it was polarizing, that gets punished by Rotten Tomatoes scores.
Katy Milkman: You're probably familiar with the Rotten Tomatoes rating system. Look up any film online, and you'll see ratings as part of the overview. The rating appears as a number, and for Rotten Tomatoes, that number is a percentage. Search Man of Steel and you'll see a 57% rating. Now, 57% doesn't mean it averaged 5.7 out of 10 as a score. It means 57% of critics reviews were positive, and positive can range from feeling mostly OK about the movie to absolutely loving it. It's a big range.
Stephen M. Colbert: Despite the praise that it was getting, it wasn't dominant enough in its score or in its reviews for anyone to say, "Let's not mess with the formula."
Katy Milkman: So the formula was tested.
Stephen M. Colbert: Zack Snyder was pressured to make Batman v Superman funnier. There was a big focus on adding jokes and on making it lighter and not so dark. He still had enough leverage at the time to not completely cave to that. Warner Bros. did step in to make some changes. It was almost a three-hour movie. And then for the theatrical version, they made him cut out 30 minutes. There was some significant plot developments and other details that got lost in that. Stylistically, Warner Bros. wasn't really exerting any real effort there yet.
Katy Milkman: Not yet. Batman v Superman: Dawn of Justice was set to release in March of 2016, but a week before it premiered, the film's Rotten Tomatoes score became available, and it wasn't good.
Stephen M. Colbert: It got a 28% on Rotten Tomatoes. That was a huge moment where it's following up a somewhat controversial movie, and then a week before it comes out it's like this bomb drops of 28%, which is really low.
Speaker 4: What was Zack thinking?
Speaker 5: Oh no, it is bad. All my friends were like, "No way. Henry Cavill rocked it, by the way."
Speaker 4: Our review is in too, and in the end, it's not much better.
Speaker 5: Oh, well.
Stephen M. Colbert: For something that's taking Batman and taking Superman and putting them in the same movie at the same time, a lot of people would make the argument that this clearly is an example of a movie that should have done over a billion dollars. And so considering the low Rotten Tomatoes score, that was seen as, that's the reason why. If only this had the larger audience appeal, this is something that would have made over a billion dollars, and so therefore we're missing out on hundreds of millions of dollars because of how polarizing this is.
Katy Milkman: The next film in the series, Justice League, was just about to start production right as this low Rotten Tomatoes score for Batman v Superman came out. The studio understandably panicked.
Stephen M. Colbert: They were going to leave to go start shooting on location, I think in Iceland, the day after the Batman v Superman premiere, which is before the box office results are even in. But because of that Rotten Tomatoes bomb drop, they actually immediately decided to do some rewrites on the script.
Katy Milkman: The studio delayed production, and Zack Snyder and his team were asked to modify some of the more controversial elements in this next film, Justice League. And again, they were asked to make it funnier.
Stephen M. Colbert: I think there's a lot of humor in his movies, but it's not the poppy, quippy jokes that you would get from a lot of other popcorn superhero movies, especially like the Marvel style that they're known for now. And so there's about two months there where they ended up with a script that was similar but got this work over to try to make it something that would be more appealing, ideally get a higher Rotten Tomatoes score.
Katy Milkman: It wasn't just rewrites. Throughout the filming, executives were on set essentially to babysit Zack Snyder to remind him of what makes a commercial hit. The pressure was rising.
Stephen M. Colbert: He started to second-guess himself a little bit because of the pressure. And so there's just a lot of stress, and also there's a lot riding on it now because instead of riding high on this universe that they're building out, there's now a sense of this is a make-it-or-break-it moment. Not just for the franchise, but also it's a big deal for Zack Snyder because if this pressure has been growing movie by movie by movie, and now not only is the franchise at stake, but this is going to be a big deal for his career also if this doesn't work out.
Katy Milkman: Snyder eventually stepped aside. It was in part due to the studio pressure, but mostly for personal reasons—there had been a tragedy in this family. But the film was nearly done at that point. In a twist, instead of working with what they had, Warner Bros. hired director Joss Whedon, yes, the same director of The Avengers movies. He came in and he did some rewrites and even some reshoots. They had to bring the cast back, and all of this reportedly cost an extra $30 million. Justice League was finally released in 2017.
Stephen M. Colbert: It's been compared to a Frankenstein's monster. You have the Zack Snyder-style scenes with the high-contrast colors and the serious tones, and then you have these other scenes where Superman's face is weirdly CGI'd and the tone is totally mismatched, and they'll even be dialogue points that are really on the nose exposition that seems to almost be apologizing to the audience to undo controversial plot points.
Katy Milkman: With a 37% on Rotten Tomatoes at the time, Justice League did a bit better than Batman v Superman, but worse than Man of Steel. Not a total failure, but also not a success, and fans eventually started piecing together that this movie was not Snyder's vision. From posts he made on social media during production, they determined that Zack Snyder's version of Justice League was out there somewhere, and they weren't going to rest until they saw it.
Stephen M. Colbert: Fans started campaigning online. #ReleasetheSnyderCut is the famous hashtag for that. Whenever there was an interview with someone from the cast or crew, the interviewer would have to ask, "Well, what about this? Does this exist?" Cast and crew started being more open about, they'd say, "Yes, it does exist. And yes, it was better."
Speaker 6: You guys overwhelmed us with Snyder cut questions. Snyder cut …
Speaker 7: When are they going to release the Snyder cut? Give it to me.
Stephen M. Colbert: Eventually the volume got so loud that the studio called up Zack Snyder and said, "So about that, what if we were to finish that up, and then we can use that as the big flagship product to launch HBO Max?"
Katy Milkman: Justice League: The Snyder Cut ended up getting a 71% from critics on Rotten Tomatoes, and it did even better with audiences.
Stephen M. Colbert: It's definitely the highest score of any of the movies that he directed for DC so far. It also got a 92% audience rating on Rotten Tomatoes.
Katy Milkman: That's a score regular users on Rotten Tomatoes gave the film, not the official critics.
Stephen M. Colbert: It finally checked all the boxes for what the studio was supposedly looking for with the other ones.
Katy Milkman: Snyder's cut better reveals the planned arc for this franchise to the point where most recent reviews of Man of Steel and Batman v Superman are more positive. Choices in those movies make more sense, but the Rotten Tomatoes numbers, accounting for hundreds of professional reviews, are mostly unchanged. Those first Rotten Tomatoes scores weigh it down.
Stephen M. Colbert: Creatives have this big struggle against these people who are trying to make decisions from numbers, and they're trying to make business decisions, and ultimately there's no connection there between what's a good creative choice versus a business choice.
Katy Milkman: Stephen M. Colbert writes about movies in the film industry. He co-hosts the Batman v Superman: By the Minute podcast, as well as Screen Rant's Ring of Power podcast. He thinks the least interesting way to talk about a movie is by saying it's good or bad. You can find more details about Stephen's work in the show notes and at schwab.com/podcast.
There's more than one way to rate a film. Film critics make careers out of writing in-depth reviews of movies for columns or TV shows or social media. And you may remember "two thumbs up" from Siskel and Ebert. And of course, Critics Pick from The New York Times or five-star scales from many reviewers, but dominant in the current zeitgeist are aggregated number ratings from websites like Rotten Tomatoes and Metacritic.
But do we put too much faith in numbers? Audiences weighing what to watch have to balance their taste for action versus comedy, films versus TV shows, and much more, and they have to consider those attributes alongside a movie's "quality" according to a number on Rotten Tomatoes.
I've invited my friend and collaborator Linda Chang on the show to talk about why we tend to focus excessively on numbers, attending more to them than other ways of representing the very same information when we make choices with trade-offs. It's a bias we call quantification fixation.
Linda Chang is a behavioral scientist at the Toyota Research Institute and a former MindCORE Postdoctoral Research Fellow at the University of Pennsylvania. For several years, Linda worked with me in Wharton's Department of Operations, Information, and Decisions, and together with two former guests on the show, University of Chicago Professor Erika Kirgios and MIT Professor Sendhil Mullainathan, Linda and I studied quantification fixation.
Hi, Linda. Thank you so much for joining me today.
Linda Chang: Hi, Katy. It's so great to be here. Thank you for having me.
Katy Milkman: Well, I am thrilled to have the chance to ask you some questions about quantification fixation, and I actually just want to start by asking you to define it. What is quantification fixation?
Linda Chang: Yeah, absolutely. So when people face a decision where there's a trade-off, so they're bombarded with different types of information, maybe some represented numerically, some represented with star ratings or with words, we find that they tend to favor options that are more attractive on the numeric dimension, so it's the tendency to overemphasize or focus more on the numeric dimension.
Katy Milkman: Could you say a little bit more about what kinds of options you have in mind where we would face trade-offs? What are these kinds of decisions that you're thinking about, and what would be a numeric component of that decision?
Linda Chang: If you are on the job market, and you're facing decisions between two different job options, you might evaluate any number of attributes, so something like your salary, your PTO, but also the company's culture, the day-to-day of what you're actually doing. Only two of the dimensions that I've mentioned just now, salary and PTO, are more naturally represented via numbers. Whereas something like a company's culture, what you're doing day-to-day, that's more qualitative, something that is a little bit richer, can't be distilled to a specific number, and you might overweight the salary, the PTO, the numeric benefits of the job rather than thinking potentially deeply about the day-to-day experience of what it might be like to take a particular option.
Katy Milkman: OK, and just for listeners who aren't familiar with that acronym, PTO is paid time off, and it would normally be a number of days that comes with a job offer. So what are the kinds of situations where you think people should be aware that they're at risk of letting quantification fixation bias their choices? In other words, could you talk about some of the types of situations where this tendency might matter.
Linda Chang: For anyone who's browsing the web, I think we're increasingly contending with a mix of both qualitative and quantitative (or numeric) information. And so something even as important as deciding what healthcare plan to enroll in, we get a ton of information about these sorts of different options. Some of that information is represented numerically, some of it with words, others with pictographs, so maybe four out of five stars or bar charts, and all of this information we're taking in and evaluating and then hopefully making a good decision for ourselves.
And so I think whether you're making decisions about your healthcare plan, about a raincoat online or camping equipment, all of these things, we're often not presented just with one type of information, but a whole slew of different kinds of information. And so when you have that mix of information, I think this is when quantification fixation can potentially come into play.
Katy Milkman: Yeah, I love that. And as you were describing it, I was thinking about all the websites I use that have that mix of information like Yelp where they give me maybe a numeric rating on average or an average price, but also a bunch of user reviews and stars, which you're saying are different than a number.
And then I'm thinking about buying a house or shopping for an apartment or what have you, constantly we're looking at a bunch of attributes lined up for a number of choices that might suit our needs. And some of them are presented as numbers, and some are presented in different ways. So you're basically saying in those situations the number will loom larger even if it's not the most important component of the choice.
Linda Chang: Exactly. So I think some deliberation probably needs to be had up front in terms of what actually matters in terms of the decision at hand so that we're not swayed by the format, let's say, of the information.
Katy Milkman: Could you describe some of the details of the research showing that people exhibit this bias, that they show quantification fixation?
Linda Chang: Yeah, participants in one experiment were presented with an option to choose between two distinct hotels, hotel A and hotel B. And these two hotels varied on whether they had a higher price and a higher rating or a lower price and a lower rating, and so there's a trade-off between these two dimensions of price and rating.
What we varied between different participants in conditions or scenarios is how price and rating was represented. So for some participants, they saw hotel A and hotel B, where price was given as a number and rating was given as star ratings, and other participants saw ratings represented as numbers and price represented as these filled cash icons.
Katy Milkman: So what did you find?
Linda Chang: We find that people tend to favor the option that is more attractive on the numeric dimension.
Katy Milkman: So if I'm trying to decide between going to the Hyatt and going to the Marriott, and the Hyatt is pricier but also nicer, I am more likely to pick the Hyatt when you tell me it's rating not in stars but you say it's a 4.8, so you make that numeric because now I'm more fixated on the rating.
And now if you change things and you tell me the rating in stars but the price is numeric, now I actually want to go to the Marriott because the Marriott is cheaper and I'm fixating more on price. Is that the right way of thinking about this?
Linda Chang: That's exactly it.
Katy Milkman: You came up with some really clever designs for these studies, Linda, and one of the things I liked in particular was not only did you present people with scenarios, but you had people make real choices that mattered and that had financial consequences for them. Could you talk a little bit about one of the studies, which I particularly love, that you did involving charitable giving decisions?
Linda Chang: Yeah, absolutely. So I think the question of how we make charitable donations, which charities we choose to donate to, is a really interesting and meaningful one. There is a website online called Charity Navigator, which audits charities and rates them on these different dimensions. And so we pulled two of these dimensions out, one being accountability in finance and another being culture and community. And we found two environmental charities, so same particular topic area, that trade off on these two dimensions. So one had a higher accountability and finance rating and lower culture and community score, whereas the other had a higher culture and community score and a lower accountability and finance score.
And so we told participants, "Please choose between these two charities, and we'll make a donation on your behalf, a $1 donation based on the decision that you make," so this is a meaningful decision, not a hypothetical out in the wild. And so we represented these scores either numerically or with these filled bar graphs to look at whether this shifts people's decisions about which charity to donate to, and we found that it does. So when accountability and finance is represented numerically, people are more likely to want to donate to the charity that has the higher accountability and finance score.
Katy Milkman: Got it. So it's really a fixation on whichever of these ratings provided by an outside audit is described with numbers that is going to drive the preference for the charity is what you're finding.
Linda Chang: Exactly.
Katy Milkman: Why is it that people show this bias? What causes it?
Linda Chang: One thing I think that surprised both you and I is that we tested actually a number of different explanations for why this might be the case, and one mechanism that we've identified is what we're calling comparison fluency, so this felt an actual ease of using information to make judgments about differences.
We're constantly bombarded with numbers. We feel very comfortable in making comparisons between numbers. I think when we think back to grade school, one of the first things that we learned is math, addition, subtraction, all sorts of operations that underlie our comparison framework, let's say. So we find that that numbers are more comparison-fluent than non-numbers in terms of at least the kinds of formats that we've looked at in our experiments. And because of this, people rely on the numeric attributes more when making decisions.
Katy Milkman: Because it's so easy for us basically to say, "Oh, 3 is 0.2 higher than 2.8, and so that's got to be the right choice."
Linda Chang: Exactly. So put another way, I think quantification fixation isn't driven necessarily by whether it's possible to evaluate choice attributes, but the ease with which we do so.
Katy Milkman: I'm curious what made you excited to study this topic? What made you decide to devote a couple years of your life to this research project?
Linda Chang: I think in having those conversations with Erika, Sendhil, and with you, I think I really latched onto this idea that we were noticing more and more we seem to be attaching numbers to all sorts of things, and it doesn't seem like that is slowing down at all. With the migration of the qualitative to the quantitative, I think a slightly different phenomenon is the want or the desire to attach numbers to so many different things, which I think is a whole other probably fascinating research project that I hope someone out there is doing, and so it felt like everywhere that I turned, this kind of question was relevant. As someone who studies judgment and decision making, I felt like I was studying decisions that I was making so many times in a day, and so I really wanted to understand what are the consequences, why might this matter?
Katy Milkman: So what advice would you give our listeners about how they can improve their own everyday decisions?
Linda Chang: I think we have to be aware that when we're making decisions, not just what information is presented, but how is the information presented, and I think before any decision is made, we also need to be careful. So we need to maybe take a step back and decide for ourselves what actually matters here? Should all attributes be weighted equally? Does one matter more than the other? Often we're not just dealing with two attributes, but many more than that. And so thinking carefully, I think, about what matters and whether our decisions are aligning with the things that we say matter, I think, is my takeaway.
Katy Milkman: I really appreciate you taking the time to talk to me today about this. Thank you so much, and thanks for being an amazing collaborator and leader on this project.
Linda Chang: It was a true pleasure, thank you.
Katy Milkman: Linda Chang is a behavioral scientist at the Toyota Research Institute and a former MindCORE Postdoctoral Research Fellow at the University of Pennsylvania in Wharton's Department of Operations, Information, and Decisions. You can find a link to her work, work that I was so happy to be part of with University of Chicago Professor Erika Kirgios and MIT Professor Sendhil Mullainathan, in the show notes and at schwab.com/podcast.
Setting financial goals can be hard, and the temptation to compare ourselves to others, especially in the social media age, can be an impediment to creating and sticking with a financial plan. But a good financial plan is based on your goals and starts with important considerations, like your personal values, that go beyond the numbers. Check out the recent Financial Decoder episode "How Can You set the Right Financial Goals?" to learn more about goal setting. You can find it at schwab.com/FinancialDecoder or wherever you get your podcasts.
Quantification fixation is a bias that many people find unsurprising when they first hear about it. "Oh sure, that makes sense that we overweight numbers." That's a reaction I hear a lot. What I find most interesting about this tendency though is that we overweight quantified dimensions of our choices even when we could get the very same information in a different format.
Seeing that a stock got a four out of five for its environmental, social, and governance risk makes you think differently about it. You'll weight its ESG score more in your decision about whether to invest in that stock than if you'd instead seen a graphic depicting the same four out of five rating visually. There's something about literally assigning a number instead of showing a graph or an icon or a word that means the same thing that can make you care more.
Like most biases we cover on this show, this one is likely to trip you up more when you don't plan ahead to dodge it. Being aware of quantification fixation gives you a big opportunity. You can combat it by thinking carefully about the attributes of a choice that really matter to you before you, say, visit a website like Redfin or Amazon or Morningstar, where the information you care about most may or may not be quantified.
And you can choose to convert all information you collect when making trade-offs into the same format, say, quantifying not only the critical reviews for two movies you're considering watching tonight, but also those films' runtime, how much you like their genre, and anything else you've decided should really matter for your choice. Just because counting changes what counts, it doesn't mean you can't fight back. You can control the things you count.
You've been listening to Choiceology, an original podcast from Charles Schwab. If you've enjoyed the show, we'd be really grateful if you'd leave us a review on Apple Podcasts, a rating on Spotify, or feedback wherever you listen. You can also follow us for free in your favorite podcasting app. And if you want more of the kinds of insights we bring you on Choiceology about how to improve your decisions, you can order my book, How to Change, or sign up for my monthly newsletter, Milkman Delivers, on Substack.
Next time, I'll speak with University of Delaware Professor Jackie Silverman about the power of tracking streaks when we're trying to build habits and how you can avoid becoming demotivated if you break a streak.
I'm Dr. Katy Milkman. Talk to you soon.
Speaker 9: For important disclosures, see the show notes or visit schwab.com/podcast.
