Solvers will be invited to the GF Challenge Platform to compete. Each solver or team will be assigned a unique application programming interface (API) key and login account for the platform. A steady stream of questions (roughly 25 per month) will be released for solvers to produce probabilistic forecasts against (specific requirements will be described in the rules). Questions may be binary (yes/no), multiple choice, or ordered outcome (binned quantity/date). In addition to the questions, solvers will receive access to a continuously updated stream of forecast judgments produced by a crowd of human forecasters. Solvers are encouraged to create solutions that use this stream of human judgments alongside other publicly available data streams and information to create their forecasting solutions.
- Questions will be released on the platform and can be pulled through the API
- Each forecast question will include a set of exhaustive, and mutually exclusive response options, and will have a final resolution date that may range from a day to many months in the future. Challenge participants will be asked to produce forecasts until the question resolves.
- Examples of potential questions*:
- Will the WHO confirm more than 10 cases of Marburg in 2018? (Yes/No)
- Before March 2018, with South Korea file a World Trade Organization dispute against the United States related to solar panels? (Yes/No)
- *Additional sample questions and criteria for forecast submissions will be available in the challenge rules.
- All forecasts will need to be submitted through the API
- Forecasts may be updated and will be scored on a daily basis for each question that is open.
Scores will be calculated using a metric which incorporates forecast accuracy and confidence. This metric is based on the Brier score, a quadratic measure of the forecaster’s closeness to ground truth for the outcome that occurs. For example, an accurate forecast with 90% confidence for an event that occurs will score four times better than a forecast with 80% confidence. The score will be calculated on a daily basis over the duration of the question; those daily accuracy scores will then be averaged. The more accurate and timely the forecast, the better score.
As questions will range in duration, milestone check-ins for questions resolved during a certain period will be used to award progress-based incentives. This means solvers who start after the challenge has launched are still able to compete for some of the prizes. Finalists who wish to compete for monetary prizes will need to provide the solution package for review the GF Challenge Team at the end of the challenge, or as appropriate, if requested. Solutions will need to be documented and all data streams used in prize-winning solutions will need to be outlined and verified by the GF Challenge Team. A complete list of rules and eligibility will be available soon.
Solvers will be able to compete on their own or form teams to compete for a total prize purse of $200,000. Please see the complete list of rules and eligibility for participation requirements.