Super Forecasting

...

The Big Five is one of the most widely-accepted personality tests. Subjects are tested for five personality traits: openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism. The tests revealed that superforecasters score high in openness to experience, which suggests a preference for cognitive variety and intellectual curiosity. Superforecasters are interested in the world and

are willing explorers. Superforecasters also spend time thinking about their own process and constantly seek to improve. When collaborating, the superforecasters often leave lots of comments in their online discussions, which allow them to recreate their thought processes and improve them when possible. Timely and accurate feedback is an essential element of improvement. Superforecasters embrace feedback. The superforecasters rarely use sophisticated mathematical models to make their forecasts, but they are uniformly highly numerate. Comfort with numbers is a prerequisite for making good forecasts but fancy quantitative models are not.

Superforecasters, similar to the foxes in Tetlock’s study of expert political

judgment, tend to be pragmatic in their methods. Rather than looking at all aspects of the world through a single lens, good forecasting requires

considering multiple points of view. There is good evidence that the aggregation of diverse points of view, done correctly, improves the accuracy of forecasts. You can gain from diversity by capturing the views of different individuals, for example, investors in a stock market, or by aggregating multiple views in your head. Sometimes the answer is to survey the views of others and ask them to criticize your view. Other times you can simply think about the same topic at different times and create a crowd within your head. No matter how you get there, taking in the views of others is valuable.

Carol Dweck, a professor of psychology at Stanford University, is best known for her work on mindset, or a way of thinking. She suggests individuals can be placed on a continuum, with a “fixed mindset” at one extreme and a “growth mindset” at the other, based on their implicit belief about the source of ability. People with a fixed mindset believe that ability is innate and therefore can’t be budged. People who say, “I’m just bad at math,” have a fixed mindset. Those with a growth mindset believe that ability is the result of hard work and effort and therefore can improve over time. Superforecasters fall on the growth mindset side of the continuum. They believe that there is always room for improvement and seek ways to get better.

Another quality that the superforecasters have is grit, a term popularized by Angela Duckworth, another one of Tetlock’s colleagues at the University of Pennsylvania.Grit is perseverance in the service of long-term goals. It entails the ability to overcome failure and obstacles along the way of achieving an objective. Combine a growth mindset and grit and you have an outstanding formula for personal development and

improvement. Tetlock and Gardner call the combination “perpetual beta.” A product in beta is nearly complete but hasroom for improvement. Perpetual beta suggests a desire for ongoingimprovement.

Experiment/Game: The simplest way to capture the wisdom of crowds is to aggregate the estimates of a large number of people with diverse views. For example, you can let lots of people examine a jar filled with jelly beans and ask each of them to provide an estimate of the total. The collective prediction will beat the average individual predictions and will generally be very close to the actual number of jelly beans. Calculate the Brier score of the person and then gift him the jar.

Calibration and Resolution are two key metrics of Brier score.

Keeping score is crucial because it provides feedback and therefore an opportunity to learn. Forecasters want to improve in two ways. One way is called “calibration,” which means that your forecasts line up with

the outcomes. For example, you are well calibrated if you say that certain events will occur with a 40 percent probability and they actually happen 40 percent of the time. Another way to improve is what the authors call “resolution.” Resolution means that when you are sure something is not going to happen, it doesn’t happen, or when you’re sure it will happen, it does.It’s a measure of conviction. Good calibration and resolution are correlated, but they are distinct. You want to sharpen your skills in both ways.

Have to find the reasons why the prediction was wrong

Appendix: Keeping Score with Brier

Psychologists commonly use the Brier score as a method for gauging the accuracy of probabilistic forecasts. Glenn Brier, a meteorologist, developed the score in the 1950s.In its simplest form, the Brier score

measures the square of the forecast error, or (forecast − outcome). For binary events, the value of the outcome is 1 if the event occurs and 0 if it does not. As in golf, a lower score is better.

You can express a Brier score either on a scale of 0 to 1, or 0 to 2, depending on the calculation. We follow Brier’s original approach and place our results on a scale of 0 to 2. When calculating the Brier score this way, you consider the squared forecast error for both the event and the non-event.

Exhibit 5 shows a meteorologist’s probabilistic forecasts for whether it will rain over the next four days. For example, on Day 2, she forecasts an 80 percent probability that it will rain. Likewise, we can say she forecasts a 20 percent probability that it will not rain. Because it did rain, we place a 1 in the outcome column below “Rain” and a 0 in the “No Rain” column. Her Brier score for that day was 0.08. For multiple forecasts, the overall

Brier score is the mean of the scores for each forecast. The meteorologist’s overall Brier score comes to 0.25.

Exhibit 5:

Calculation of Brier Score

– p. 21

Day

Rain

Forecast Outcome

No Rain

Forecast Outcome

Brier Score

Calculation Result

1