Choiceology's Guide to Better Decisions: With Guests James Korris, Carey Morewedge & Jack Soll

Speaker 1: At the tone, please record your message.

Chris Hughes: Hi, this is Chris Hughes. Look, I can't reach my sister, and since you're next door, could you please check on Terry? I'm really worried. I called the super. He's got her apartment open. Just please look around. I've called Terry, but she's not picking up. I don't know where she is. Anything you could find would be a big help. Look, I owe you. OK, I … 

Katy Milkman: That's an opening scene from a video game called MISSING: The Pursuit of Terry Hughes. The game was designed to help train U.S. intelligence agents to avoid decision pitfalls by working through a fictitious missing person's case.

Today on the show, we're bringing you something a little different. We're calling it "Choiceology's Guide to Better Decisions." We've gathered some advice from expert guests who study the science of de-biasing decisions, and we hope their insights will help you make ever more thoughtful choices. You'll also hear from two guests whose work on the video game MISSING has had a lasting measurable impact on its players, improving the decision-making skills of people who work in one of the highest stakes sectors in the United States, intelligence analysis.

I'm Dr. Katy Milkman, and this is Choiceology, an original podcast from Charles Schwab. It's a show about the psychology and economics behind our decisions. We bring you true and surprising stories about high-stakes choices, and then we examine how these stories connect to the latest research in behavioral science. We do it all to help you make better judgments and avoid costly mistakes.

James Korris: This is James Korris. I'm the president and CEO of Creative Technologies Inc. in Los Angeles, a woman-owned small business.

Katy Milkman: Back in 2015, James was contracted to design a game for the government agency IARPA. That stands for the Intelligence Advanced Research Projects Activity.

James Korris: MISSING (Part One): The Pursuit of Terry Hughes, it's a character-centric narrative that follows the first-person player through a series of discoveries. And basically, you are in the role of a person living in New York, in a cool part of Manhattan, but you have a very boring job. Your neighbor is a very exciting, very interesting person, whom you know only very slightly, because even though she's a next-door neighbor in New York, she's almost a total stranger. And the inciting incident that gets the narrative rolling is a phone call from your neighbor's brother who's very concerned because she's been missing for three days, and he can't figure out what's happened to her. So basically, it's a mystery.

Katy Milkman: The game is set up to challenge the player's decision-making skills.

James Korris: OK, so the first thing that pops up here is the user's smartphone. We see that Chris has texted. He says, "Yeah, you're in the apartment. Please look around. Take as many pictures as you like, but pick the best three that you think are good clues to help us figure out what happened to Terry. I hope nobody grabbed her."

Katy Milkman: The clues are cleverly designed to elicit a biased response. In this case, they trip up the player by leveraging people's tendency to jump to conclusions with limited data.

James Korris: By Chris saying, "I hope nobody grabbed her," he's setting up the hypothesis that there were malign actors who broke into her apartment and took her away. So the question is, "Will you be looking for evidence to support that? Or would you consider a counterfactual hypothesis?"

And this is where we start to navigate around the apartment. So starting in the living room, see there's a desk here. So we see there's some papers around. See this bookcase. We move to this document here and pick it up. And you can see that there are restaurants in Los Angeles. And if you think about it for a moment, if someone is generating a list of restaurants in another city, it does not support the hypothesis that criminals broke into the apartment and kidnapped a person. So if you were to take a picture of this, then it would suggest that you do have some interest in exploring counterfactual hypotheses, so we'll take a picture of that.

OK, so here's a chair that was turned on its side. Then we say, "Oh, well that might support the hypothesis that she was taken by force." Bathroom's kind of interesting. So here's what's in the bathroom. Wet towel on the ground, what does that tell us? Let me take a picture of that. And then over here, here's her closet. And one of the things we notice is that there is a suitcase missing. So again, if someone is taken by force, they don't usually pack a bag. So we got the chair that shows maybe by force, the towel, which suggests that she wasn't taken by force, and the missing suitcase, sounds like she packed for a trip. And so, we're going to send them to Chris.

Katy Milkman: As the game progresses, other characters start to share clues, like the apartment building superintendent who comes over to Terry's apartment and shows you some security footage.

Speaker 5: Hi, you're from 12G. I got a call from Chris, Terry's brother. He told me you were in here looking. The guy is massively freaking. Did you find anything? Come with me.

James Korris: Then he takes us into her office and there's more discovery here.

Speaker 5: I've got the security footage from the elevator. Here we go. This was just a couple hours ago. Terry seem like a nervous person to you?

Katy Milkman: This clue might stand out if you remember the Choiceology episode about the fundamental attribution error. In that episode, we talked about our tendency to attribute other people's behavior to their underlying attributes or personality and to disregard how dramatically their situation has likely influenced their behavior.

James Korris: We might assume that she's anxious because there's something seriously wrong going on in her life. Or it just might be the fact that she's in a creaky old elevator in Manhattan, and it's the kind of thing that would make anybody anxious.

Speaker 5: Personally, I think Terry's in some kind of trouble. She always seems short of cash these days. Late on the rent last month.

James Korris: There's lots of stuff to explore on her desk. Here is an electric bill that says it's past due. So this would support the hypothesis that she's having financial problems.

Katy Milkman: After clues are gathered and the first part of the game is over, users complete an after-action review and hear lectures on decision biases that might've tripped them up during the game.

Carey Morewedge: Confirmation bias is a mistake that people make when they have one idea of what's right, and then they search for evidence and interpret evidence in a way that confirms it. Because people are trying to confirm one idea, they don't evaluate all evidence objectively.

James Korris: So that's Carey who's speaking now.

Carey Morewedge: In the game, for example, you search for clues about why Terry wasn't in her apartment. If you only search for clues that Terry had a financial problem, and you didn't check whether she was gone for other reasons, you committed confirmation bias.

Katy Milkman: That's Carey Morewedge. Carey is a professor of marketing at Boston University who studies psychological biases and how to reduce them. Back in 2015, James and Carey worked together on the project we've just described to marry behavioral science insights with game design, and to make the game as effective as possible at improving people's decisions. Carey joined me to talk about the game and about his research on strategies for de-biasing decisions.

Hi, Carey. Thank you so much for taking the time to be here today.

Carey Morewedge: Hi, Katy. It's such a pleasure to be here.

Katy Milkman: I want to start by asking you if you could just describe your 2015 paper about two different de-biasing training programs that you designed, which I think of as really the best work on de-biasing. They had remarkable and durable effects on reducing decision biases. So I'd love if you could just explain the trainings you designed and how you proved their value.

Carey Morewedge: The backstory is I got a call asking me if I'd be interested in building video games to reduce cognitive biases. And that sounded absolutely crazy to me. IARPA, which is an intelligence research program of the United States government, was doing these high-risk moonshot kinds of programs where a bunch of teams compete to see if they can achieve some kind of goal that has flummoxed the scientific community. And the program manager at IARPA who is running this program was a gamer. She got the idea that maybe serious games, which are video games with a learning component, could be used to reduce a lot of the cognitive biases that she was reading about.

And then we had an initial meeting with them, and I said, "The data that you can reduce cognitive biases is pretty thin." When we think about biases and decision-making, I think as a field we've often described that as there's not much that we can do. I accepted that kind of conclusion. But it was also very discouraging because it feels like "Why are we spending all this time discovering these kinds of biases but feeling like we can't do that much about them?"

So this was exciting to me. I didn't think it would work, but I was game to give it a shot. The basic structure of what we did was they had identified six cognitive biases that they thought were really important for intelligence analysis. So for example, one was confirmation bias, the idea that when we're testing a hypothesis, we tend to look for information that confirms that hypothesis rather than dis-confirms it. So there was six of these different kinds of biases, and they basically broke them into two video games, where each game was designed to treat three of the cognitive biases that they identified. And then we competed against six other teams to see if we could reliably reduce these kinds of biases both immediately and then after a delay. And that delay was two or three months based on which game we were testing. Each game takes about 90 minutes to play, but the game's really surprisingly effective at reducing cognitive bias.

Katy Milkman: So how do you measure the effectiveness of the games?

Carey Morewedge: Yeah, that was really interesting too. I looked around, and there weren't amazing scales to measure these biases at an individual level. And so we developed these scales of each of the biases, and so we could go out and measure people's susceptibility to the six different kinds of biases before they played the game, immediately after they played the game, and then two or three months afterwards. And we basically see on average a reduction of their propensity to exhibit these biases of about 30% from immediately before the game to after, and then we see two or three months later, we see the game is still reducing their propensity by about 23%.

Katy Milkman: That's really interesting. And you mentioned that there are six biases. You gave one example. Could you talk a little bit about the full set of six and whether there were any differences in how well teaching people in this immersive way worked across the different biases?

Carey Morewedge: So the six biases were bias blind spot. That's seeing more bias in other people than in yourself. Confirmation bias. Fundamental attribution error, which is that we tend to attribute our own actions to the situation and other people's actions to them or dispositional effects, like that was you versus that was your context. Anchoring. Representative heuristics, so those are kinds of cases where we often use the similarity of situations as a proxy for the probability of situations. And then the last one is social projection. When people try to think about how much other people have a particular attitude or a particular preference, they often use their own attitudes and preferences to predict how other people feel.

So for example, if you find Crocs to be very comfortable footwear and like Crocs, you might think that most people like Crocs. If you think Crocs are horrifically unfashionable, then you would think that no one wears Crocs because they're terrible. That's a sort of schlocky example. But you can see that politics often divides how people feel about different kinds of issues, and people tend to overestimate how many people agree with them on their particular kind of political rift issue. So those are the six biases.

Katy Milkman: And P.S., I have to say, those are great biases to try to tackle. And I think we've covered four of them maybe on the show, but you just gave me two more topics we'll have to cover in future episodes. So thank you for that too. And I'd love to hear how it all worked.

Carey Morewedge: So the biggest surprise to me is how effective these de-biasing strategies are. The second thing is in other papers we used random YouTube clips of people's lectures as a de-biasing intervention. And we've even done things where we just explain the biases to professionals, like intelligence analysts or risk analysts, and they work. And I think actually the biggest problem was the measurement.

Katy Milkman: Ah. So you're saying our pessimism prior to this research about how easily we can de-bias people was because we didn't measure their success dodging biases very well, and you came up with better measurement tools.

Carey Morewedge: Yeah.

Katy Milkman: I will just say as someone who teaches about bias and who obviously spends a lot of time through this podcast trying to teach the world, it's such an encouraging message that even though we'd seen a lot of failures before, de-biasing through education really does work, and it was maybe a measurement issue. But I also want to point out that in your paper you show that these really engaging video game trainings work better than watching lectures by some of the best researchers in our field. And I would love it if you could share a little bit of your insight about why the video game was particularly useful way to improve people's decision-making.

Carey Morewedge: I think the video game basically throws the kitchen sink of what we think works for de-biasing at the problem. To de-bias someone, they need to know what the bias is, they need to know what direction it affects judgment in. Then, giving people feedback helps, and then they need coaching and strategies. And the video game does all of that in as much intensity as possible in 90 minutes.

Katy Milkman: I love that. And of course, watching a passive video doesn't give you that feedback component and it can't be personalized or engaging in the same way as an active video game.

Carey Morewedge: Yeah, and I think there's a layer of bias blind spot to watching a video too, where you see other people commit these biases, and you're like, "Yeah, maybe I do that, but not as much as that person" or "That's not as much a problem for me." And the game actually is like, "Yeah, you showed this bias in this particular scenario, and here's what you would do to do that." So there's kind of an effect of getting feedback on your receptivity to the training. And in our bias blind-spot paper that describes the skills that we used for bias blind spot, we actually find that people who are higher in bias blind spot are less receptive to de-biasing trainings than people who are lower. And so I think shattering that kind of bias blind spot is helpful in itself.

The other aspect that our findings suggest is that we get this transfer effect. So learning about cognitive biases in one domain does seem to help you in other kinds of related problems.

The take-home, I would say, is that I think we've been really underselling the value of a lot of the public outreach of decision scientists, that people are really benefiting from a lot of this kind of work. And part of it is that it's not as exciting for many people to measure individual differences in these biases that have been established as to understand them or document them. As we mature as a group and as a field, I think it's very useful for us to try to think about, "OK, we've done a lot of good trying to show all of these biases. How do we go from the descriptive to the prescriptive and try to improve social welfare that way?"

Katy Milkman: I love that. And obviously birds of a feather, I think you and I both, we'd love for the incentives to change a little bit so there's evaluation not only of the basic science, but also some of the applied work that can have a positive social impact. So I love this research you've done. Is there anything that you do differently in your life as a result of the research you've done and that you've read on de-biasing?

Carey Morewedge: Yeah, I think I'm just more optimistic about our field. And also, I think that it's important for people who do this kind of work to do more public outreach because it seems to work. I really admire people like you who are going out and talking about the science to everyone and trying to get the idea out. And I think that we could do more as a science to try to think about ways to improve on many of the kinds of biases that we've identified.

And there's a lot of important work to be done on what works best. And for whom? And how related are these biases? And is there something that would work for many of them? And just understanding more about the structure of different biases and how they fit together, I think, would be really helpful to move forward. And for listeners who aren't in the science, I think that this suggests that doing things like listening to this podcast or reading a book or watching a talk online, those are all things that can help you improve your decision-making, and it's not futile. Your biases are not all permanent. You can do something about them.

Katy Milkman: Carey, thank you for the very kind words, and also what an incredibly optimistic note to end on. It's so encouraging that the research shows learning works, and that we're not just stuck with these biases. And just like we can learn calculus and algebra and addition, we can learn how to make better choices. So thank you for the incredible work you have done to teach us that it's not futile. And thank you for taking the time today to talk to me and to our listeners about some of your findings and how we can better de-bias ourselves.

Carey Morewedge: Thanks, Katy. It was such a pleasure to be here.

Katy Milkman: Carey Morewedge is a professor of marketing and the Everett W. Lord Distinguished Faculty Scholar in the Questrom School of Business at Boston University. James Korris is President and CEO of Creative Technologies Inc., and is a pioneer in immersive game-based simulation for military learning. You can find links to Carey's research and to a video of the game he collaborated on with James in the show notes and at schwab.com/podcast.

The video game developed by James Korris and Carey Morewedge and their colleagues shows that learning about behavioral biases and developing strategies to counteract them may be more effective than we thought. The robust results point to the value of education around de-biasing.

With that in mind, we thought it would be useful to share some practical strategies to help you make better decisions, a kind of checklist for a broad range of choices that you might encounter at work or at home or anywhere really. You may have heard some of this advice before on Choiceology, but I've asked my next guest to distill it into a checklist that you can take with you as you make your next big or small decision.

Jack Soll is an expert in de-biasing strategies. He's a professor of management and organizations at the Fuqua School of Business at Duke University.

Hi, Jack, thanks so much for taking the time to talk to me today.

Jack Soll: Hi, Katy. It's a pleasure to be here.

Katy Milkman: Well, it's a pleasure to have you. And I'm hoping we could dive right into some strategies for better decision-making. We've done an episode on the effectiveness of checklists, so I'd actually love to get your list of the best advice in that format.

Jack Soll: All right, checklist. OK, so I have four pieces of advice here, probably incomplete, because as we know any time one person sits down and writes down a list, there are probably some things they're not thinking about. But here's four.

So one is be decision ready. And so what do I mean by that? So we often fall back on our intuition and can be biased if we are distracted, if we're tired, if we're hungry, if we're really rushed, we don't really have time to deliberate. And we often make better decisions if we actually, one, know how to make the right decision, so we know what principles to apply, but also, we kind of have some time to think about and apply those principles. So there's a lot of situations where we're not decision ready. Maybe we're angry at somebody, or maybe we're trying to do multiple things at once, that we're distracted. And so at those times, it's best to put off the decision to another time.

Katy Milkman: I love that, Jack. That's a great item number one. What else?

Jack Soll: Yeah, I think it's really important … we miss opportunities because we didn't think broadly enough. And so a second piece of advice is to broaden the frame. A lot of biases actually could be attributed to thinking too narrowly, that we're only thinking about the one option that's in front of us. Should I go to this college? Should I rent this apartment? And when we do that, we only often have one objective in mind. We're not thinking about all the things we care about. So imagine a student choosing a college and wants to be a software engineer. They might focus on which school has the best computer science department, and that might turn out to be the best choice for them. But it's also the case that if they stop and think about it, they might realize, "Well, in the future I might prefer a different career path" or "There's other things I care about a college, like the social dimensions and things like that."

And so I think the advice is to try to think more broadly about what can happen, how might our preferences change, what can happen in the world? And at the end of the day, decisions are limited by the option set. A lot of research has shown that people are not necessarily so great at choosing well, or they might be OK at choosing between A, B, and C. But sometimes none of those options are the right option. Sometimes what you actually ought to do is throw out all those options and invent some new ones. And so that's what I mean by broadening the frame.

Number three on the checklist: take advantage of the wisdom of others. So seek advice. And when doing this, make sure to get independent advice. And so don't tell the other person what you think the answer is because this is going to influence what they tell you. They'll either think that you're asking for confirmation, and they'll tell you what they think you want to hear, or their thinking will be influenced by your thinking. Either way, you're going to be better off if you really want to get good advice is to get independent advice.

And then fourth: experiment. Try things that you think won't work. Do this in a low-cost way. The idea here is to try to generate dis-confirmation, try to prove yourself wrong. One way to do this is empirical. So if you think you know what a website should look like, you could do some A/B testing, and you could create variations that are different from what you think are best to kind of test whether or not your idea is correct. But in other situations, experimentation is difficult, and there we have to rely on the wisdom of others. And one way to do this is to have a devil's advocate, have somebody argue for an alternative point of view.

Katy Milkman: Yeah, I love that. I often also tell my students to try to find somebody who will play the role of devil's advocate and just argue the opposite, too, when I'm worried that I might be too attached to a given path or argument and not thinking broadly enough.

Jack Soll: Oh, for sure. You want to think about reasons why you might be wrong. We often think about why we're going to succeed or why our answer is right. And you want to think about reasons why the answer might be much different from what you thought. And sometimes we're all on our own, and this is a little bit challenging, but ask yourself why you might be wrong.

There is one other way to do this. Kind of this, I guess, fourth point of trying to seek dis-confirmation is to tackle the same problem but on different occasions. So one trick actually that a lot of professors actually use when they're grading papers. Number one is you don't want to have the name of the student on the paper. And it's very tempting to know whose paper you're grading, but if the name's not there, then you could come up with an assessment of the work, which is not tainted by your views about the person. So this is called blinding.

But in addition to that, the advice here would be to do it twice. And it takes extra time, but oftentimes what happens is, at any given moment, there's some noise in our judgment, or we have different things in mind. And so if we look at the same decision problem twice or are grading the same essay twice, we might come up with different answers. And in those cases, if they're different from each other, you could either, if it's quantitative, you can average the grades together. If it's more discrete, or if it's categorical, what you can do is look at it a third time or have somebody else look at it. And so there's a lot of work along these lines in medicine where if you ask a radiologist to look at the same x-ray on multiple occasions, they might come up with different answers. And this isn't really a critique of their expertise. This is just symptomatic of how difficult the task is. But for those kind of tricky ones where they're giving different answers on different occasions, then maybe a third look or bringing in another expert can be helpful.

Katy Milkman: I love that, Jack. And P.S., it also makes me feel better about the fact that I'm that dissertation advisor who after six months reads a new draft from my student of their, say, dissertation and says, "I really don't like the third paragraph of the intro. You should cut it." And they tell me, "Actually, I added it because you told me to." Anyway, I'm often inconsistent in my advice to my students, but I'm like, "Well, you're getting the wisdom of the crowd within."

Jack Soll: Exactly.

Katy Milkman: It's like two independent perspectives. Aren't you lucky? Anyway, that's my framing trick.

Jack Soll: There's more than one Katy Milkman.

Katy Milkman: That's right. That's right. You caught me hungry today. Jack, this has been so enlightening and helpful, and I really appreciate you taking the time to talk with me and share these insights. Thank you so much.

Jack Soll: Sure, you're welcome. It was a pleasure.

Katy Milkman: Jack Soll is the Gregory Mario and Jeremy Mario Distinguished Professor of Management and Organizations at the Fuqua School of Business at Duke University. You can find links to Jack's research, as well as an article we wrote together with John Payne, titled "Outsmart Your Own Biases" in the show notes and at schwab.com/podcast.

Cognitive and emotional biases can have a big impact on your financial life. To learn more about de-biasing techniques and how to approach a variety of specific financial decisions—like how to save for college or what to do with an old 401(k)—check out the Financial Decoder podcast. You can find it at schwab.com/FinancialDecoder or wherever you get your podcasts.

This episode has been a bit atypical. We've shared the great news that just by learning about decision biases, you can improve your judgment. I think we need to run an experiment to test whether listening to the show successfully de-biases people. But there are a lot of different mistakes we make that we need to be aware of. We often seek confirmation of our existing beliefs rather than searching in a balanced way for information. We misattribute people's behavior to fixed features of their personality instead of their situation. And think too narrowly about important problems. And that's just scratching the surface.

The good news is that cures exist for these types of biases. As Jack shared, you can set yourself up for success by being decision ready. Part of that, of course, is being familiar with decision biases. Carey explained just how valuable that can be, but also, you should make sure you're well-rested, calm, and well-informed before making a consequential decision. It can also be valuable to broaden your thinking by deliberately considering a wider scope of possibilities and future preferences than you would naturally. And you can be intentional about collecting information and seeking independent advice from other people with relevant expertise.

Finally, try to prove yourself wrong by testing. That might mean A/B testing or just having a friend play devil's advocate. I hope that the more you listen to Choiceology, the more reliably you'll be able to dodge decision biases.

You've been listening to Choiceology, an original podcast from Charles Schwab. If you've enjoyed the show, we'd be really grateful if you'd leave us a review on Apple Podcasts, a rating on Spotify, or feedback wherever you listen. You can also follow us for free in your favorite podcasting app. And if you want more of the kinds of insights we bring you on Choiceology about how to improve your decisions, you can order my book, How to Change, or sign up for my monthly newsletter, Milkman Delivers, on Substack.

Next time, we'll bring you stories about two very different battles from the American Revolution, and we'll share how people make decisions in the face of real versus hypothetical losses.

I'm Dr. Katy Milkman. Talk to you soon.

SPEAKER 8: For important disclosures, see the show notes or visit schwab.com/podcast.