Every day each of us makes judgments about the future in the face of uncertainty. These judgments can have a huge impact on our lives, so it’s really important that we make them as accurately as possible. But what can you do if you have limited information about the future? Well, the Open Philanthropy Project commissioned us to create a new web app that helps you practice making predictions, with the goal of honing your ability to make accurate judgments in uncertain situations. You can now try the Calibrate Your Judgment tool on ClearerThinking.org by clicking here! To get started, you simply have to make a free account (so that it can track your progress over time). We’ve also compiled some simple tips to help you make more accurate predictions, which you can read below. But first we’ll briefly explain why the new tool is valuable. The Calibrate Your Judgment app helps you learn how to quantify judgments in the form of probabilities, also known as probabilistic forecasting. Organizations implicitly rely on forecasting when deciding which major projects to undertake, which initiatives to support, and how much funding to allocate to different research areas. One such organization that has to make many challenging decisions in an uncertain world is the Open Philanthropy Project, whose mission is to use research and grantmaking to improve lives as effectively as possible. They approached us with the idea for creating the Calibrate Your Judgment web app. The aim of the web app is to help you become “well-calibrated.” This means that when you say you’re 50% confident, you’re right about 50% of the time; when you say you're 90% confident, you're right about 90% of the time; and so on. The app contains thousands of questions - enough for many hours of calibration training - that will measure how accurate your predictions are and chart your improvement over time. Nobody is perfectly calibrated; in fact, most of us are overconfident. But various studies show that this kind of training can quickly improve the accuracy of your predictions. Of course, most of the time we can’t check the answers to the questions life presents us with, and the predictions we’re trying to make in real life are aimed at complex events. The Calibrate Your Judgement tool helps you practice on simpler situations where the answer is already known, providing you with immediate feedback to help you improve. How else can you get better at making predictions? Take a look at the tips below, which includes useful recommendations compiled by AI Impacts and advice given by Philip Tetlock, author of Superforecasting: The Art and Science of Prediction.
Tips for making better predictions
1) Get used to thinking in ranges and probabilities.
When you want to make a prediction, try not to use words like “inevitable,” "probably," “impossible,” or “maybe.” Instead, approach the claim with a range and a probability. For example, if you want to estimate the year Einstein won the Nobel Prize for physics, first give a range of the dates in which you think he could have won the prize. Think hard about how confident you are that the date really falls within that range, keeping in mind that people usually give ranges that are too narrow. Then assign a probability to that range - are you 70% sure the date lies in that range, or 90% sure? You can use this technique to approach questions like “when will Donald Trump leave presidential office? Will it be by the end of 2020, or another date in the future?” Assigning a range of dates and a probability of this event happening within those dates will help you practice more evidence based thinking. If you’re uncertain about the truth of a claim, instead of just labelling it either “true” or “false,” assign it a probability.
2) Strike a balance between under- and overconfidence.
Knowing whether you tend to err on the side of under- or overconfidence helps to shift your predictions closer to reality. Most beginners to forecasting are overconfident, which you can combat by reducing your initial gut feeling of confidence (by, say, 5-15%). Or, if you are predicting the range of a value, such as a range of heights that you're 90% confident the height of Mount Everest falls within, you can make your range a bit bigger than your intuition suggests. If the ranges you chose were too narrow, that means you are overconfident. If you ranges you chose are very wide, this means you’re underconfident. You can also try making 10 practice estimates at a 90% confidence level, and then seeing whether you really do get 9/10 answers in the correct range. The Calibrate Your Judgment web app is perfect for this, but you can also try it by making predictions about what will happen in your life or the world over the next year. PredictionBook is another tool that's great for tracking real life predictions that you may want to check out!
3) Break intractable problems into tractable sub-problems.
It can be useful to break down big, intractable estimations into smaller, more manageable questions. For example, if you want to know how confident you should be about whether Brexit will occur in October 2019, you can try making predictions about (a) whether there will be a general election in October 2019, (b) whether parliament will allow the UK to leave the EU without a “deal,” and (c) whether the Prime Minister in power is willing to ask for an extension to the Brexit date (check out this helpful flowchart which tackles exactly these questions). Based on the likelihood of these events, you can try to estimate the probability of your initial question occurring. This technique is known as a Fermi estimation.
4) Take the outside view, then the inside view.
It’s tempting to base your prediction on the first intuition that occurs to you. Instead of starting with your own personal instinct, consider what the outside perspective says; for instance, how have similar predictions to this one turned out in the past? Or, what usually happens in situations like this one? When you have taken this information into account, only then consider the specific evidence from your experience, and use this to adjust your final prediction. For instance, if you're trying to predict the chance that your friend cancels on you for dinner tonight, you can start by asking "how often does this friend cancel plans?" That's your "outside view" perspective. Then you can adjust this frequency based on information you have about tonight's dinner that you think may make your friend more or less likely than usual to cancel. This technique is similar to "reference class forecasting."
5) Seek alternative hypotheses.
One of the easiest ways to consider alternatives is to ask yourself, “what are some reasons my judgment might be wrong?” Tetlock calls this “dragonfly eye perspective” - valuing diverse views and combining them with your own perspective. Combining the judgments of multiple people is a great way to generate counter-arguments and make more accurate estimates. This is how prediction markets work - by combining many sources of information to provide forecasts that are often more reliable than individual experts. If you’d like help seeking out alternative hypotheses, our Belief Challenger tool does exactly this, encouraging you to question your existing perspectives on the world.
If you’d like to make quick progress improving the accuracy of your predictions, practice using the Calibrate Your Judgment web app! We’ve compiled thousands of facts to make the question sets you can practice on. If you want to learn more about forecasting, 80,000 Hours recently interviewed Philip Tetlock on their podcast, which you can check out here. The Open Philanthropy Project has also put together a helpful document outlining the process and difficulties of developing a calibration training app, which you see here. We also recommend checking out AI Impact’s excellent blog post on this topic and Philip Tetlock's book.
Furthermore, we have a full podcast episode about forecasting the things that matter that you may like:
Click here to access other streaming options and show notes.