Coming soon: How many conflict deaths will there be in Sudan in 2025? Jeremy’s Multi-AI Oracle will compete with the machine forecasting bots of the
VIEWS competition.
Background on VIEWS: Near the end of 2024, the latest forecasting competition of the Violence and Impacts Early-Warning System (VIEWS) was launched. On Dec. 4, 2024, a group of machine forecasting experts involved in VIEWS observed Jeremy’s demonstration of his Multi-AI Oracle. Although it was too late for him to sign up as a VIEWS competitor, they urged him to run a side competition for everyone’s benefit. Jeremy chose their Sudan conflict deaths question. Interactive Views dashboard here.
Background on the Sudan conflict, as evaluated by Jeremy last year:
(1) The importance of the Sahel, a region in Africa that includes Sudan:
Why the Sahel Actually Matters (Possibly To You, Too)
(2) April 29, 2024: Today, a Russian diplomat is negotiating gold mining with the Sudanese Minister of Minerals.
(3) June 3, 2024: Today the African Union Announced Plans to Mediate in Sudan’s Civil War.
Sept. 11, 2024 Over 100K starvation deaths 96% likely in Sudan by 1 Jan 2025
A the bottom of this page, a screenshot of VIEWS forecasts for state-based conflict fatalities in Sudan through January 2028.
How many seats will the Conservative Party win in Canada’s next federal parliamentary election?
Welcome to the Multi-AI Oracle. How many seats will the Conservative Party win in Canada’s next federal parliamentary election?
Model value:
* less than 172: 17%
* between 172 and 205: 39%
* between 206 and 240: 33%
* more than 240: 11%
Obtaining outside newsfeed. News from various sources:
Based on the provided news articles, here’s a summary of the key points about Canada’s upcoming election:
“Of course! Please provide the text of the newsfeed you’d like summarized, and I’ll be happy to help.” [Carolyn’s note: Today the news-gathering function failed, apparently because Claude failed to provide any data to the Multi-AI Oracle today.]
Fact-checking news summary:
Given the [lack of] text, it is challenging to extract specific facts related to the original question “How many seats will the Conservative Party win in Canada’s next federal parliamentary election?” However, I can provide an evaluation based on an analysis of the context:
1. **Fact:** “Please provide the text of the newsfeed you’d like summarized, and I’ll be happy to help.”
– **Correct (True/False):** True (the sentence is internally consistent and a straightforward offer of assistance).
– **Importance:** Unimportant
– **Relevance:** False (the fact does not pertain to the question’s outcome regarding the number of seats for the Conservative Party).
Since the summary does not actually contain information about the Canadian parliamentary election or potential outcomes for the Conservative Party, I cannot derive any facts pertinent to answering the original question. Therefore, the text lacks any relevant content from which to determine election results.
Fact-checked summary:
The text does not contain any factual information regarding the number of seats the Conservative Party will win in Canada’s next federal parliamentary election. It only includes an offer to summarize a newsfeed, which is true but irrelevant to predicting election outcomes. There are no relevant or useful facts from this text that can help determine the election results for the Conservative Party.
Querying Claude (Error: Claude is not returning an array)
Querying Mistral (AI predicts: [“less than 172”: 0.45, “between 172 and 205”: 0.35, “between 206 and 240”: 0.15, “more than 240”: 0.05] – confidence: 6)
Querying OpenAI (AI predicts: [“less than 172”: 0.25, “between 172 and 205”: 0.35, “between 206 and 240”: 0.25, “more than 240”: 0.15] – confidence: 5)
Question Type: Multiple Choice
Confidence: 5.5
# LLM responses: 2
Model value:
* less than 172: 35%
* between 172 and 205: 35%
* between 206 and 240: 2%
* more than 240: 10%
We queried Mistral and OpenAI. A summary of what the responding AIs considered is as follows:
The predictions regarding the Conservative Party’s potential seat gains in the upcoming Canadian federal election are based on historical election data and recent polling trends, which suggest modest support increases. Historically, the party has held between 121 and 166 seats, with current conditions likely leading to gains but not exceeding a significant majority of 205 seats. However, the political landscape is competitive, influenced by leadership, campaign strategies, socio-economic factors, and regional dynamics, particularly in provinces like Ontario and Quebec. Potential inaccuracies in the prediction may arise from unexpected events such as scandals, economic changes, or shifts in public opinion, alongside the influence of third-party candidates and voter sentiment volatility.
Runtime: 183 seconds. Past Multi-AI Oracle Experiments
Guide to our Multi-AI Oracle’s Reports
Median: We collect the output values from the LLMs in an array. We then take the median value. This is between 0 and 1 (i.e. 0.15 = 15%).
Base rate: Currently we take OpenAI’s value as the base rate. We are working on a better value using a different prompt that takes into account process. The base rate helps to determine if the median makes sense.
SD: The standard deviation between the median and the base rate.
Confidence: We query each of the LLMs on how confident they are of their predictions (between 0 and 10) and take a median. Because the LLMs tend to be overconfident, we take anything lower than 6 as being low confidence. This factors into the overall model.
Conf Mode: Based on the confidence value. >=9 is high confidence. Below 6 is low confidence (this is also triggered by an exceptionally high SD).
Mellers: This refers to Barbara Mellers, specifically a paper she wrote that includes a formula for moving values towards an extreme (i.e. 0 or 1).
Reverse Mellers: This uses the formula from above, but with a sub-1 coefficient to move the values closer to 50%.
Theory of Mind: We ask the LLMs what they think other LLMs would predict. We hope that this makes them consider the questions more deeply.
Beta Distribution: Currently unused, but possibly of interest. This is based on the median, the base rate and the SD.
Close Type: We noticed that the appropriate base case for some questions is closer to the extremes, while others are closer to 50%. When we have a low confidence value, this helps us to determine whether to extremize or de-extremize the value. ‘A’ implies closer to zero. ‘B’ implies closer to 50%. ‘C’ implies closer to 100%.
# LLM Responses: The count of LLMs that responded. We query 5 currently, but they have a relatively high API failure rate.
Model value: For normal confidence mode, this is the median. For high confidence, this is the Mellers value. For low confidence, this is either the Mellers or Reverse Mellers (depending on the Close Type).
The VIEWS Forecasting Page for Sudan
The chart shows predictions of fatalities from “state-based conflict” in Sudan through January 2028.
