Botmaster Jeremy’s bestworldbot is competing in the Q4 AI Forecasting Benchmark Tournament for a prize pool of $30,000. It practiced on eight questions today.
Guide to the bestworldbot’s practice run outputs
Median from the bestworldbots: We collect the output values from the LLMs in an array. We then take the median value. This is between 0 and 1 (i.e. 0.15 = 15%).
Base rate: Currently we take OpenAI’s value as the base rate. We are working on a better value using a different prompt that takes into account process. The base rate helps to determine if the median makes sense.
SD: The standard deviation between the median and the base rate.
Confidence: We query each of the LLMs on how confident they are of their predictions (between 0 and 10) and take a median. Because the LLMs tend to be overconfident, we take anything lower than 6 as being low confidence. This factors into the overall model.
Conf Mode: Based on the confidence value. >=9 is high confidence. Below 6 is low confidence (this is also triggered by an exceptionally high SD).
Mellers: This refers to Barbara Mellers, specifically a paper she wrote that includes a formula for moving values towards an extreme (i.e. 0 or 1).
Reverse Mellers: This uses the formula from above, but with a sub-1 coefficient to move the values closer to 50%.
Theory of Mind: We ask the LLMs what they think other LLMs would predict. We hope that this makes them consider the questions more deeply.
Beta Distribution: Currently unused, but possibly of interest. This is based on the median, the base rate and the SD.
Close Type: We noticed that the appropriate base case for some questions is closer to the extremes, while others are closer to 50%. When we have a low confidence value, this helps us to determine
whether to extremize or de-extremize the value. ‘A’ implies closer to zero. ‘B’ implies closer to 50%. ‘C’ implies closer to 100%.
# LLM Responses: The count of LLMs that responded. We query 5 currently, but they have a relatively high API failure rate.
Model value: For normal confidence mode, this is the median. For high confidence, this is the Mellers value. For low confidence, this is either the Mellers or Reverse Mellers (depending on the Close Type).
[PRACTICE] Will Iran possess a nuclear weapon before 2030?
Obtaining outside newsfeed. Summarizing outside newsfeed:
“As of October 15, 2024, several reports have emerged concerning the advancement of Iran’s nuclear program, leading to heightened international apprehensions regarding the potential for Iran to develop nuclear weapons before 2030. The International Atomic Energy Agency (IAEA) suggested that Iran could have enough nuclear material for a weapon within a week and potentially for 13 weapons in four months. Some political figures, such as US Representative Mike Turner, have echoed these concerns by suggesting that Iran might declare it has nuclear weapons by the end of 2024. A former IAEA official, Olli Heinonen, speculated that Iran could possess ten warheads by April 2025, raising doubts about Iran’s claim that its nuclear activities are solely for civilian use. Public opinion within Iran is reportedly in favor of nuclear armament, with a recent poll indicating that 69% of respondents support the acquisition of nuclear weapons. These reports should be approached cautiously, as they may incorporate unverified information and reflect regional tensions and political agendas.”
Querying Perplexity (AI predicts: 0.85 – confidence: 8)
Querying Claude (AI predicts: 0.65 – confidence: 6)
Querying Mistral (AI predicts: 0.65 – confidence: 7)
Querying Cohere(Error: Cohere is not returning an array)
Querying OpenAI (AI predicts: 0.65 – confidence: 5)
Median from LLMs: 0.65
Base rate: 0.65 (from OpenAI)
SD: 0.09
Confidence: 6.5
Conf Mode: Normal
Mellers: 0.75
Reverse Mellers: 0.58
Theory of Mind: 0.525 (What did the LLMs think other LLMs predicted?)
Beta Distribution: 0.51
Close Type: B (B = cautious # closer to 50%; A/C = closer to extremes)
# LLM responses: 4
Submitted value: 0.65
We queried Perplexity, Claude, Mistral, OpenAI. A summary of what the responding AIs considered is as follows:
The various AI summaries highlight concerns about Iran’s rapid advancement in uranium enrichment and its potential to develop nuclear weapons in the near future, as reported by the IAEA and other sources. There is acknowledgment of the technical capability to produce enough nuclear material for weapons and public support within Iran for nuclear armament. Reports suggest Iran could potentially have nuclear weapons by 2024 or possess multiple warheads by 2025, raising alarms among international figures. However, the summaries also note the possibility of the situation being exaggerated due to political tensions, and recognize that international intervention, diplomatic efforts, or technical challenges could prevent or delay Iran’s nuclear weapons development. The reliability of the reports and the actual intentions of Iran’s leadership remain crucial factors in accurately assessing the situation.
Submitting to Metaculus..
[PRACTICE] Will someone born before 2001 live to be 150?
We queried Perplexity, Claude, Mistral, OpenAI. Obtaining outside newsfeed. Summarizing outside newsfeed:
“The newsfeed discusses the improbability of individuals born before 2001 living to 150 years, based on current research and expert opinions. Studies in fields like longevity and aging suggest that while medical advancements have increased life expectancy, the maximum lifespan remains around 115 years, with only a minor fraction of the current population projected to surpass 100 years. Experts such as gerontologist Jay Olshansky emphasize the inherent biological limits of aging and note that life expectancy is rising more slowly now than in the past. Meanwhile, some researchers, including Dr. Nir Barzilai, advocate for a focus on “healthspan,” which emphasizes maintaining good health throughout life rather than merely extending lifespan. Overall, the consensus appears to be that achieving a lifespan of 150 years is unlikely given present biological and medical constraints.”
Querying Perplexity (AI predicts: 0.01 – confidence: 8)
Querying Claude (AI predicts: 0.05 – confidence: 7)
Querying Mistral (AI predicts: 0.15 – confidence: 6)
Querying Cohere(Error: Cohere is not returning an array)
Querying OpenAI (AI predicts: 0.05 – confidence: 7)
Median from LLMs: 0.05
Base rate: 0.05 (from OpenAI)
SD: 0.05
Confidence: 7
Conf Mode: Normal
Mellers: 0.01
Reverse Mellers: 0.19
Theory of Mind: 0.025 (What did the LLMs think other LLMs predicted?)
Beta Distribution: 0.37
Close Type: A (B = cautious # closer to 50%; A/C = closer to extremes)
# LLM responses: 4
Submitted value: 0.05
We queried Perplexity, Claude, Mistral, OpenAI. A summary of what the responding AIs considered is as follows:
The consensus among the various AI analyses is that, based on current scientific research and expert opinion, the maximum human lifespan is believed to be around 115 to 125 years. Although medical advancements continue to improve life expectancy, these increases have slowed, and there is no substantial evidence to support the possibility of humans living to 150 years with current technology and biological understanding. The discussion highlights the inherent biological limits of aging, and while speculative breakthroughs in biotechnology, genetic engineering, or regenerative medicine could extend lifespans, such advancements remain theoretical and face significant challenges. The prediction of someone currently alive reaching the age of 150 is deemed improbable, although it could change if unforeseen major breakthroughs occur, altering the current limits of human longevity.
Submitting to Metaculus..
More about Botmaster Jeremy’s Activities
Botmaster Jeremy’s team includes BestWorld’s Brian LaBatte, Michał Dubrawski, Bruce Lawhorn, and Carolyn Meinel.
Practice run of Oct. 10, 2024.
Click here for the winners of the Q3 AI Forecasting Benchmark T# LLM Responses:ournament.
Metaculus’ analyses of the competition’s bots’ capabilities.
The final leaderboard for the Q3 AI Forecasting Benchmark Tournament.
Our Humans vs Multi-AI Panel Experiment of July 25 — Sept. 18, 2024 competition.
All of our bots vs humans experiments.
Our concluded FED rate increase competitio
[PRACTICE] Will Donald Trump be jailed or incarcerated before
2030?
Obtaining outside newsfeed. Summarizing outside newsfeed:
“As of October 15, 2024, Donald Trump is facing several legal challenges, including a conviction in a hush money case in New York. However, the judge has postponed the verdict until after the presidential election, suggesting that the outcome could influence potential incarceration. Experts assert that if Trump loses the election on November 5, 2024, he might face serious legal repercussions, including possible prison time, whereas a victory could lead to the cases being dropped. The situation remains uncertain and is closely tied to the election results and ongoing legal proceedings. It is important to note that the information in this newsfeed may include unverified claims or speculative statements.”
Querying Perplexity (AI predicts: 0.2 – confidence: 6)
Querying Claude (AI predicts: 0.6 – confidence: 5)
Querying Mistral (AI predicts: 0.65 – confidence: 6)
Querying Cohere (AI predicts: 0.2 – confidence: 4)
Querying OpenAI (AI predicts: 0.3 – confidence: 5)
Median from LLMs: 0.3
Base rate: 0.3 (from OpenAI)
SD: 0.2
Confidence: 5
Conf Mode: Low
Mellers: 0.18
Reverse Mellers: 0.4
Theory of Mind: 0.5 (What did the LLMs think other LLMs predicted?)
Beta Distribution: 0.46
Close Type: A (B = cautious # closer to 50%; A/C = closer to
extremes)
# LLM responses: 5
Submitted value: 0.18
We queried Perplexity, Claude, Mistral, Cohere, OpenAI. A summary of what the responding AIs considered is as follows:
The various AI analyses focus on Donald Trump’s ongoing legal challenges, including his conviction in a New York hush money case, and their potential outcomes based on the 2024 presidential election results. The postponement of sentencing until after the election suggests that the political landscape could heavily influence legal proceedings. If Trump loses the election, there’s speculation about potential incarceration, though this remains uncertain given the historical precedent of high-profile politicians avoiding prison and the possibility of parole or lighter sentences due to his status as a first-time offender. Furthermore, new legal developments, public opinion, and political maneuvers, including appeals and potential pardons, could affect the outcome, leading many to consider a fine more likely than imprisonment. The timeframe until 2030 allows for these legal processes to run their course, but the situation’s inherent unpredictability leaves room for various outcomes.
Submitting to Metaculus.
[PRACTICE] Will there be Human-machine intelligence parity
before 2040?
Obtaining outside newsfeed. Summarizing outside newsfeed:
“In reports from October 2024, discussions are ongoing about the potential for human-machine intelligence parity before 2040. Notably, futurist Ray Kurzweil has forecasted the emergence of general artificial intelligence (AGI) by 2029, with AI potentially surpassing human intelligence by 2045. He suggests that advancements in nanotechnology and brain-computer interfaces could facilitate a blending of human and AI capabilities, enhancing human cognition. These developments are expected to transform industries and introduce new opportunities, but they also spark ethical concerns regarding human identity and societal impacts. While such predictions highlight a promising trajectory, they should be considered cautiously, acknowledging the speculative nature of such technological forecasts.”
Querying Perplexity (AI predicts: 0.92 – confidence: 8)
Querying Claude (AI predicts: 0.65 – confidence: 6)
Querying Mistral (AI predicts: 0.75 – confidence: 7)
Querying Cohere(Error: Cohere is not returning an array)
Querying OpenAI (AI predicts: 0.25 – confidence: 6)
Median from LLMs: 0.7
Base rate: 0.25 (from OpenAI)
SD: 0.25
Confidence: 6.5
Conf Mode: Low
Mellers: 0.82
Reverse Mellers: 0.6
Theory of Mind: 0.25 (What did the LLMs think other LLMs predicted?)
Beta Distribution: 0.02
Close Type: A (B = cautious # closer to 50%; A/C = closer to extremes)
# LLM responses: 4
Submitted value: 0.82
We queried Perplexity, Claude, Mistral, OpenAI. A summary of what the responding AIs considered is as follows:
The various AI-driven analyses coalesce around the idea that recent advancements in AI, like GPT-4 and Gemini, have significantly narrowed the gap towards achieving human-machine intelligence parity. This progress is underscored by substantial increases in AI training resources, with a notable expansion over the past decade, and achievements in domains such as image recognition, speech transcription, and game playing. Predictions from experts, such as Ray Kurzweil, suggest the potential for achieving parity by around 2040. However, while the pace of technological advancement and increased funding appear promising, challenges remain. These include the complexity of human-level general intelligence, potential regulatory and ethical barriers, and the speculative nature of forecasting technological trends. Furthermore, achieving true parity would require proficiency in abstract reasoning and creativity, which are areas where AI is still developing. Overall, while optimistic projections exist, the multifaceted nature of human intelligence presents challenges that could slow down progress towards full parity by the forecasted timelines.
Submitting to Metaculus.
[PRACTICE] Will Joe Biden be elected US President in 2024? (Compare this with the other way the bot answered! this question!)
Obtaining outside newsfeed. Summarizing outside newsfeed:
“As of October 15, 2024, Joe Biden has officially announced his candidacy for re-election in the upcoming U.S. presidential election. His potential re-election could make him the oldest president to begin a second term at 81 years old. Recent polling suggests a competitive race, especially between Biden’s running mate, Kamala Harris, and former President Donald Trump, with Harris reportedly having a narrow lead. However, Biden’s approval ratings remain relatively low, with 41% of voters expressing approval, reflecting general voter dissatisfaction which could affect his re-election chances. Economic challenges and a desire for change are prominent themes influencing voter sentiment, adding to the uncertainty surrounding the election outcome scheduled for November 5, 2024. The newsfeed includes various articles that provide analysis and polling data, but readers should note that not all information may be verified, given the presence of rumors and propaganda.”
Querying Perplexity (AI predicts: 0 – confidence: 10)
Querying Claude (AI predicts: 0.45 – confidence: 5)
Querying Mistral (AI predicts: 0.65 – confidence: 6)
Querying Cohere (AI predicts: 0 – confidence: 10)
Querying OpenAI (AI predicts: 0.55 – confidence: 6)
Median from LLMs: 0.45
Base rate: 0.55 (from OpenAI)
SD: 0.28
Confidence: 6
Conf Mode: Low
Mellers: 0.41
Reverse Mellers: 0.47
Theory of Mind: 0 (What did the LLMs think other LLMs predicted?)
Beta Distribution: 0.68
Close Type: B (B = cautious # closer to 50%; A/C = closer to extremes)
# LLM responses: 5
Submitted value: 0.47
We queried Perplexity, Claude, Mistral, Cohere, OpenAI. A summary of what the responding AIs considered is as follows:
There is conflicting information regarding Joe Biden’s 2024 presidential candidacy. Some reports suggest that he has withdrawn from the race as of July 2024 and endorsed Kamala Harris as his successor, making it impossible for him to win the election. However, other sources claim he has announced his candidacy for re-election and is actively campaigning. Despite being the incumbent, Biden faces significant challenges, including low approval ratings, his age, and voter dissatisfaction. The race is described as competitive, particularly involving Kamala Harris and Donald Trump, with recent polls indicating a close contest. Economic challenges and a general desire for change among voters contribute to the uncertainty of the election’s outcome. The presence of rumors and unverified reports complicates the situation, and unexpected events could further impact the election before it takes place on November 5, 2024.
Submitting to Metaculus..
[PRACTICE] Will there be a US-China war before 2035?
Otaining outside newsfeed. Summarizing outside newsfeed:
“As of October 15, 2024, reports indicate escalating tensions between the United States and China, primarily over the issue of Taiwan. China has been conducting military exercises around Taiwan, a move that raises concerns due to the U.S.’s commitment to defend Taiwan if attacked. In response, the U.S. Navy is working to enhance its military readiness, with a focus on potential conflict by 2027. Experts speculate about the possibility of a war, including discussions on nuclear confrontation risks. Chinese President Xi Jinping has reiterated the goal of reunification with Taiwan, hinting at possible military action if diplomatic approaches fail. While a definitive timeline is unclear, the geopolitical climate suggests increasing chances of conflict before 2035. However, it’s important to note that these developments are based on various reports that may not be entirely reliable or free from bias.”
Querying Perplexity (AI predicts: 0.45 – confidence: 6)
Querying Claude (AI predicts: 0.15 – confidence: 5)
Querying Mistral (AI predicts: 0.2 – confidence: 6)
Querying Cohere(Error: Cohere is not returning an array)
Querying OpenAI (AI predicts: 0.3 – confidence: 5)
Median from LLMs: 0.25
Base rate: 0.3 (from OpenAI)
SD: 0.11
Confidence: 5.5
Conf Mode: Low
Mellers: 0.12
Reverse Mellers: 0.37
Theory of Mind: 0.375 (What did the LLMs think other LLMs predicted?)
Beta Distribution: 0.65
Close Type: A (B = cautious # closer to 50%; A/C = closer to
extremes)
# LLM responses: 4
Submitted value: 0.12
We queried Perplexity, Claude, Mistral, Cohere, OpenAI.A summary of what the responding AIs considered is as follows:
The collective reasoning suggests that while there are escalating tensions between the US and China, particularly over Taiwan, and ongoing military posturing enhances the risk of conflict, the probability of a US-China war before 2035 remains low. Several key factors contribute to this assessment: historical precedent of avoiding direct military conflict, the strong deterrent effect of nuclear capabilities, and significant economic interdependence between the two countries. The prediction acknowledges the potential for diplomatic resolutions given the considerable timeline till 2035. Nonetheless, there are caveats, such as the reliability of reports and the possibility of unforeseen geopolitical events, leadership changes, or miscalculations that could escalate tensions. Given these dynamics, the prediction remains cautious but optimistic about the avoidance of overt conflict.
Submitting to Metaculus.
[PRACTICE] Will humans go extinct before 2100?
Obtaining outside newsfeed. Summarizing outside newsfeed:
“A recent newsfeed from October 2024 highlights various studies and expert opinions expressing concern over humanity’s future, citing several existential risks that might lead to human extinction by 2100 if present trends persist. Key threats identified include the potential for a severe climate crisis, with projections indicating a global temperature rise potentially reaching 2.7°C above pre-industrial levels by the end of the century, which surpasses the Paris Agreement target and could have dire consequences. Additionally, experts warn of the dangers posed by natural disasters like supervolcanoes, asteroid impacts, and extreme climate events, all of which could threaten human survival. Moreover, human-made risks, particularly the misuse of technology and political obstacles, exacerbate these threats. The ongoing loss of biodiversity and depletion of natural resources contribute further to the crisis. Despite these alarming factors, there is no definitive prediction of human extinction by 2100, but the need for significant change is emphasized to mitigate these risks.”
Querying Perplexity (AI predicts: 0.06 – confidence: 6)
Querying Claude (AI predicts: 0.05 – confidence: 5)
Querying Mistral (AI predicts: 0.15 – confidence: 6)
Querying Cohere(Error: Cohere is not returning an array)
Querying OpenAI (AI predicts: 0.05 – confidence: 5)
Median from LLMs: 0.06
Base rate: 0.05 (from OpenAI)
SD: 0.04
Confidence: 5.5
Conf Mode: Low
Mellers: 0.01
Reverse Mellers: 0.19
Theory of Mind: 0.075 (What did the LLMs think other LLMs predicted?)
Beta Distribution: 0.35
Close Type: A (B = cautious # closer to 50%; A/C = closer to extremes)
# LLM responses: 4
Submitted value: 0.001
We queried Perplexity, Claude, Mistral, OpenAI. A summary of what the responding AIs considered is as follows:
The prediction regarding human extinction by 2100 is based on a variety of existential risks, including climate change, technological misuse, natural disasters, and biodiversity loss, as presented by expert opinions from conferences and studies. Experts acknowledged a low to moderate probability ranging from 1% to 19%, though higher hazard estimates might be skewed by selection biases at expert gatherings. Despite these identified risks, human resilience, adaptability, and potential technological advancements are believed to offset these threats to some extent. The prediction’s reliability is complicated by several factors: the inherent unpredictability of existential risks, potential cascading effects, unforeseen catastrophic events, and the long period until 2100, which could introduce new challenges or solutions. Overall, while existential risks are significant, the low probability forecasts and human potential for innovation offer a cautiously optimistic outlook against complete extinction by 2100.
Submitting to Metaculus.
[PRACTICE] Will Joe Biden be elected US President in 2024? (Same question but answered in a different way and got it right!)
Obtaining outside newsfeed. Summarizing outside newsfeed:
“As of October 15, 2024, Joe Biden will not be seeking re-election as President of the United States, with Vice President Kamala Harris stepping in as the Democratic candidate. Harris’s candidacy marks a historical attempt to become the first female US president. Former President Donald Trump is positioning himself as the Republican candidate, creating a competitive electoral race. Analyses suggest the election could be closely contested, with both candidates commanding significant support. Some speculate that Harris could have an edge due to her campaign efforts and financial backing, though these predictions are subject to the evolving nature of the campaign dynamics.”
Querying Perplexity (AI predicts: 0 – confidence: 10)
Querying Claude (AI predicts: 0 – confidence: 10)
Querying Mistral (AI predicts: 0 – confidence: 10)
Querying Cohere (AI predicts: 0 – confidence: 10)
Querying OpenAI (AI predicts: 0 – confidence: 10)
Median from LLMs: 0
Base rate: 0 (from OpenAI)
SD: 0
Confidence: 10
Conf Mode: High
Mellers: 0
Reverse Mellers: 0
Theory of Mind: 0 (What did the LLMs think other LLMs predicted?)
Beta Distribution: N/A
Close Type: A (B = cautious # closer to 50%; A/C = closer to extremes)
# LLM responses: 5
Submitted value: 0.001
We queried Perplexity, Claude, Mistral, Cohere, OpenAI. A summary of what the responding AIs considered is as follows:
The consensus among various AI analyses is that Joe Biden will not be elected as the US President in 2024 due to his official withdrawal from the race as confirmed by credible sources such as AP News and BBC. Biden announced his withdrawal on July 21, 2024, endorsing Vice President Kamala Harris as his successor, who is now the Democratic nominee facing likely Republican nominee Donald Trump. Based on these developments, there is effectively zero chance of Biden being elected, unless an extraordinary and highly improbable political event were to occur, such as Biden re-entering the race or an error in the current reports. However, all reliable indications affirm his withdrawal and endorsement of Harris, making any such scenario extremely unlikely.
Submitting to Metaculus.