Massachusetts Forecasting Tournament
I am a huge fan of forecasting tournaments, competitions in which people try to have the most accurate predictions on a variety of questions.
These were popularized by University of Pennsylvania psychologist Phillip Tetlock, who won an Intelligence Community competition with his “superforecasters”. Making a judgement about what is more or less likely to happen in politics has always been with us, right back to the betting markets in Gilded Age New York. What a forecasting tournament does is structure our predictions so lots of people can participate easily.
There are a few clear benefits to participating in these contests:
The wisdom of the crowds effect mean that average forecasts are usually more accurate than an individual’s.
They force us to think probabilistically and consider our own biases which makes us better political analysts.
We can prove that we’re better at understanding politics than our friends and colleagues.
There are a number of prediction markets and forecasting tournament platforms out there, but they are going for a national or global audience. Since Legislata is about politics at all levels, we’re going to step into that gap, and run a forecasting tournament for Massachusetts politics in 2022.
Here are the main points:
It’s free to play and you can play individually and as a team.
Teams are up to 10 people and you can be part of as many as you’d like. This is the form to create/join a team.
There’s $600 in prize money available.
You can jump directly to the questions here.
How the tournament works
We have 15 questions about the primary and general elections (and a 5 question bonus round - retweet, share, or like our tweet or LinkedIn post and we’ll send you the link).
You’re asked to predict what will happen in various elections, like who will win the governor’s race, the Second Suffolk District, and how turnout will be.
Importantly, your predictions should be the probability of each thing happening. We are not asking you to say who will the gubernatorial election - we are asking what is the chances of each candidate winning. So your answer will not be simply Maura Healey, but Maura Healey: 75%, Geoff Diehl: 15%, Chris Doughty: 10% (or whatever you think the odds are).
You score will be assessed at how close you are to what actually happened. If you say that Healey has a 90% chance of winning and she does, you were 10 percentage points off (which is pretty good). But if she loses, then you were 90 points off (which is pretty bad). Your prediction should reflect not only the odds but your confidence in the result. The best way to get consistently good scores is not to be overconfident.
You can make as many predictions as you’d like since your score will be calculated for each day. In fact, responding to events and revisiting your score has been shown to be one of the best ways to have a good score. You can access all the questions individually here for a quick prediction.
How your predictions should look. You’re saying the odds of each thing happening, not who you think will win.
How the prizes work
There is $500 available for the main part of the forecasting tournament.
$250 will go to the individual with the highest score.
$250 will go to the team with the highest average score.
For the bonus round, $100 will be awarded.
$50 for first place
$30 for second place
$20 for third place
How the scores are calculated
This part is technical so feel free to skip it and get to the tournament if you’d like.
We will calculate your accuracy scores based on how close you were to the right answer for every day that you had active forecasts. This is based on the idea of Brier scores, which came from meteorology.
Step 1. Calculate Brier score
A Brier score is the sum of the squared errors of your forecast. Let’s say you think that for Governor, Maura Healey has an 80% chance of winning, Geoff Diehl has a 15% chance, and Chris Doughty has a 5% chance. If Healey wins, then that result resolves as a 1 (it happened) and Diehl and Doughty as 0 (they did not happen). Your squared errors are therefore (1-0.8)^2 + (0-0.15)^2 + (0-0.05)^2 = 0.065. If you had the same forecast but Doughty pulled out a surprise victory, then your score would be (0-0.8)^2 + (0-0.15)^2 + (1+0.05)^2 = 1.565.
Like golf, a lower number is best. 0 is the theoretical lowest limit, which would mean that you predicted 100% for what happened. We don’t suggest being too bold, because predicting 100% and getting it wrong is very costly.
Step 2. Calculate daily relative Brier scores
Since some questions will be harder than others, we will compare you to the median Brier score for each question. So if the median score is 0.9 and your score was 0.5, then your relative score would be 0.4.
Since questions often get easier to predict as we get close to election day, we calculate this score on a daily basis. Your most recent forecast will be counted for every day until you make a new one. So if I say on September 1st that Healey has an 80% chance of winning and on September 7th update that to 90%, then I’ll be recorded has forecasting 80% for Sept 1st, 2nd, 3rd, 4th, 5th, and 6th.
Finally, we average your daily scores and multiply your score by the percentage of days on which you had a forecast. This means that it’s better to be slightly above average for the entirety of the tournament than to come in on the last day with a lucky guess.
Step 3. Add up all the questions
Once we have your score for each question, we add them up for all the questions and you have your final score.
After the primaries are concluded, we’ll send you your scores for those questions and your interim standings. When the general elections are over, we’ll send you your final scores.
Bonus round
The bonus round is scored much more simply. There’s 5 questions and you get a point for each question you get right. Question 5 serves as the tiebreaker.