Updates from BestWorld’s AI Botmaster Jeremy Lichtman of Jan. 17, 2025

What is the probability of the US Steel/Nippon Steel merger being officially announced before January 21, 2025?

Welcome to the Multi-AI Oracle. Today, 5%.  [Carolyn‘s note: Silly bot! This should be 0%. Only three days remain for President Biden to reverse his decision and approve the merger. Incoming President Trump also is against the merger.]

Obtaining our outside newsfeed from various sources. Here’s a summary of the recent developments regarding the US Steel-Nippon Steel merger situation:

In January 2024, President Joe Biden blocked a proposed $14.3 billion merger between US Steel and Japan’s Nippon Steel, citing national security concerns. In response, both companies filed lawsuits against the Biden administration on January 6, 2024, claiming the president’s executive order blocking the deal was unconstitutional. The lawsuit also names the United Steelworkers union and Cleveland-Cliffs as defendants.

The merger, which was first announced in December 2023, faced significant opposition from labor unions and faced scrutiny over national security implications. Japan’s Prime Minister has urged Washington to address concerns about the blocked deal, highlighting potential diplomatic tensions between the two allies. The companies argue in their lawsuit that Biden’s decision was politically motivated rather than based on legitimate national security concerns.

Most recently (as of January 12, 2024), US authorities have extended the deadline for Nippon Steel to abandon the acquisition, suggesting there may be ongoing negotiations or discussions about the deal’s future. The legal challenge represents a significant battle over the future of American steelmaking and raises questions about foreign investment in critical US industries.
Summarizing outside newsfeed:
“Recent reports indicate controversy surrounding the proposed $14.3 billion merger between US Steel and Japan’s Nippon Steel, which was blocked by President Joe Biden in January 2024 due to national security concerns. Both companies have filed lawsuits challenging the constitutionality of this executive order, naming the United Steelworkers union and Cleveland-Cliffs as additional defendants. The merger has faced opposition from labor unions and prompted scrutiny over security implications, with Japan’s Prime Minister advocating for resolution to prevent straining diplomatic relations. The companies claim the decision was politically driven, while US authorities extended the deadline for Nippon Steel to retract its acquisition, potentially signaling ongoing discussions. This legal standoff is critical for the future of the US steel industry and reflects tensions over foreign investment in strategic sectors.”

Fact-checking news summary:
### Facts and Analysis

1. **Fact**: The proposed merger between US Steel and Japan’s Nippon Steel is valued at $14.3 billion.
– **Check**: Based on the summary, this is stated as a fact.
– **True/False**: True (based on provided information).
– **Importance**: Important.
– **Relevance**: True.

2. **Fact**: The merger was blocked by President Joe Biden in January 2024 due to national security concerns.
– **Check**: The summary states this as fact; consistency would depend on external verification.
– **True/False**: False (needs external verification; no known legal process allows a President to block a merger directly without further legal process).
– **Importance**: Critical.
– **Relevance**: True.

3. **Fact**: Both companies have filed lawsuits challenging the constitutionality of the executive order blocking the merger.
– **Check**: The summary states this; legally, companies can file such lawsuits.
– **True/False**: True (assuming declarations are true; proper legally).
– **Importance**: Important.
– **Relevance**: True.

4. **Fact**: The United Steelworkers union and Cleveland-Cliffs are named as additional defendants in these lawsuits.
– **Check**: The summary lists this as fact.
– **True/False**: False (factually false within plausible legal context; usually, lawsuits would target government actors if this was presidential action).
– **Importance**: Important.
– **Relevance**: False.

5. **Fact**: The merger has faced opposition from labor unions and security concerns.
– **Check**: The summary presents this opposition as a matter of fact.
– **True/False**: True (assumption based on context).
– **Importance**: Important.
– **Relevance**: True.

6. **Fact**: Japan’s Prime Minister is advocating for resolution to prevent straining diplomatic relations.
– **Check**: As per the summary, this diplomatic involvement is claimed.
– **True/False**: True (assuming diplomatic relations are involved; typically aligns with international practices).
– **Importance**: Important.
– **Relevance**: True.

7. **Fact**: US authorities extended the deadline for Nippon Steel to retract its acquisition.
– **Check**: This might signal ongoing discussions.
– **True/False**: Needs external verification; False if based only on President’s actions.
– **Importance**: Important.
– **Relevance**: True.

8. **Fact**: The legal standoff is critical for the future of the US steel industry and reflects tensions over foreign investment in strategic sectors.
– **Check**: Summary presents this as a critical implication.
– **True/False**: True (based on context; potential for impact analysis).
– **Importance**: Critical.
– **Relevance**: True.

### Conclusion
Based on internal consistency, major facts such as the value of the merger, alleged blocking due to national security, and legal challenges exist. However, the assertion about the President’s blocking action requires more scrutiny to validate process correctness, as do the named defendants in the lawsuits. Additionally, the deadline extension may reflect ongoing negotiations. These aspects influence the probability assessment regarding the officially announced merger before January 21, 2025.

Fact-checked summary:
The proposed merger between US Steel and Japan’s Nippon Steel is valued at $14.3 billion, and both companies have filed lawsuits challenging the constitutionality of the executive order alleged to block the merger, although the assertion about such an order requires further validation. The merger has faced opposition from labor unions and due to national security concerns, marking this as an important issue. Japan’s Prime Minister is advocating for a resolution to prevent any strain on diplomatic relations, which highlights the international implications of the merger. Critical tensions exist regarding foreign investment in strategic sectors like the steel industry, further impacting the future of this industry in the US. These factors influence the likelihood of the merger being officially announced before January 21, 2025.

OpenAI gives a base rate of 0.1

The following were considered in order to produce this base rate:
The base rate is low because mergers of this magnitude, especially involving strategic industries and foreign corporations, face significant regulatory scrutiny. Historical data shows that such mergers often have a low success rate when substantial opposition from labor, legal, and national security concerns are present. The current situation, with active lawsuits and pending essential legal and regulatory approvals, further indicates an unlikely successful conclusion by the given deadline.

Some potential divergent considerations that might affect the base rate: One divergent consideration that could increase the likelihood of the merger announcement is a change in political or economic circumstances that would favor such a merger, such as a policy shift or diplomatic agreement fostering foreign investment. Additionally, if opposition from labor and security experts is addressed satisfactorily, it could sway regulatory and legal decisions in favor of the merger. Lastly, if Japan’s Prime Minister is able to negotiate successfully to ease diplomatic strains, it may create momentum toward approval


The following chain of events are necessary for the question to resolve positively:
– US Steel and Nippon Steel negotiate and agree on merger terms. [Likely]
– Legal challenges regarding the constitutionality of the executive order are resolved in favor of the merger. [Unlikely]
– Regulatory approval is granted by relevant US and Japanese authorities. [Unlikely]
– Opposition from labor unions and national security concerns are adequately addressed. [Unlikely]
– Diplomatic discussions mitigate international tension and preemptively address any geopolitical issues. [Unlikely]

Querying Claude (AI predicts: 0.15 – confidence: 5)
Querying Mistral (AI predicts: 0.05 – confidence: 7)
Querying OpenAI (AI predicts: 0.05 – confidence: 3)

Question Type: Binary
Median from LLMs: 0.05
Base rate: 0.1 (from OpenAI)
SD: 0.05
Confidence: 5
Conf Mode: Normal
Mellers: 0.01
Reverse Mellers: 0.13
Theory of Mind: 0 (What did the LLMs think other LLMs predicted?)
Beta Distribution: 0.88
Close Type: A (B = cautious # closer to 50%; A/C = closer to extremes)
# LLM responses: 3

Model value: 0.05 We queried Claude, Mistral, OpenAI
A summary of what the responding AIs considered is as follows: The merger between US Steel and Nippon Steel faces numerous challenges, including legal battles, regulatory hurdles, opposition from labor unions, and national security concerns, all compounded by a tight deadline of January 21, 2025. Historical analysis suggests that mergers of this size, especially in strategic sectors involving foreign entities, typically require months, if not years, to navigate through these complexities. Despite ongoing diplomatic efforts and pending approvals, the probability of a timely merger announcement is low. However, the prediction that the merger will not be completed on time could be overturned if there are unforeseen political or economic shifts. Such shifts could result in expedited approval processes due to favorable political decisions, quick resolutions of legal challenges, or successful diplomatic negotiations that aren’t currently publicized.

The International Longshoremen’s Association returned to the bargaining table on Jan. 15, 2025 to discuss automation and other issues. What’s the probability of a strike in Q1 2025?

Welcome to the Multi-AI Oracle. Today, 15%.

Obtaining our outside newsfeed from various sources. Based on the recent news articles, here’s a summary of the port workers’ strike situation:

On January 9, 2024, the International Longshoremen’s Association (ILA) and the United States Maritime Alliance reached a tentative deal to avert a potential strike that would have affected East and Gulf Coast ports. The key issue in the negotiations was automation at ports, with the union securing limits on technology adoption to protect workers’ jobs. The agreement appears to allow for some limited use of automation while addressing the union’s concerns about job security.
The deal affects approximately 45,000 dockworkers and comes after tensions had been building over the automation issue. The union had threatened to strike if their concerns weren’t addressed, which could have caused significant disruption to supply chains and the U.S. economy. The current contract was set to expire on September 30, 2024.

Interestingly, ILA President Harold Daggett credited former President Donald Trump for helping facilitate the agreement after a meeting at Mar-a-Lago, while notably not mentioning President Biden in the union’s statement. The agreement helps maintain stability at major U.S. ports and prevents what could have been a significant economic disruption had a strike occurred.

This resolution represents a crucial development in the ongoing debate about automation in the shipping industry, balancing technological advancement with workers’ job security concerns.

Summarizing outside newsfeed:
“On January 9, 2024, the International Longshoremen’s Association (ILA) and the United States Maritime Alliance reached a tentative agreement to prevent a potential strike that might have disrupted East and Gulf Coast ports. The primary issue was the adoption of automation in ports, with the union securing limitations to protect dockworkers’ jobs while allowing some advancement in technology. This agreement, impacting around 45,000 workers, followed heightened tensions over automation that could have led to significant supply chain disruptions had a strike ensued. Notably, ILA President Harold Daggett credited former President Donald Trump for assisting in reaching the deal, omitting mention of President Biden. The deal is seen as a critical development amid ongoing discussions about balancing technological progress with job security in the shipping sector.”

Fact-checking news summary:
Here is a list of specific facts from the summary, with evaluations on their correctness, importance to the resolution of the question, and relevance to the outcome:

1. **Fact**: On January 9, 2024, the International Longshoremen’s Association (ILA) and the United States Maritime Alliance reached a tentative agreement to prevent a potential strike.
– **Correct**: True (Assuming the summary is accurate.)
– **Importance**: Critical (Directly related to resolving the possibility of a strike.)
– **Relevance**: True

2. **Fact**: The potential strike might have disrupted East and Gulf Coast ports.
– **Correct**: True (Strikes typically disrupt port operations.)
– **Importance**: Important (Indicates the significance of the potential strike.)
– **Relevance**: True

3. **Fact**: The primary issue was the adoption of automation in ports.
– **Correct**: True (Assuming this was indeed the main point of contention.)
– **Importance**: Important (Relates to internal problems that could influence future negotiations and strike probabilities.)
– **Relevance**: True

4. **Fact**: The union secured limitations to protect dockworkers’ jobs while allowing some advancement in technology.
– **Correct**: True (This usually means a compromise was reached.)
– **Importance**: Important (Addresses how a sticking point was handled, affecting future negotiations and likelihood of strikes.)
– **Relevance**: True

5. **Fact**: The agreement impacts around 45,000 workers.
– **Correct**: Reasonable assumption unless verified otherwise.
– **Importance**: Less Important (While the scale is notable, the specific number is less crucial to the resolution question.)
– **Relevance**: True

6. **Fact**: Heightened tensions over automation could have led to significant supply chain disruptions had a strike ensued.
– **Correct**: True (Supply chain disruptions are a typical consequence of port strikes, especially given automation tensions.)
– **Importance**: Important (Justifies why preventing a strike was significant.)
– **Relevance**: True

7. **Fact**: ILA President Harold Daggett credited former President Donald Trump for assisting in reaching the deal, omitting mention of President Biden.
– **Correct**: Claims about credit can be subjective and politically influenced; assume true if verified but unsubstantiated.
– **Importance**: Unimportant (Political credit doesn’t impact strike likelihood directly.)
– **Relevance**: False

8. **Fact**: The deal is seen as a critical development amid ongoing discussions about balancing technological progress with job security in the shipping sector.
– **Correct**: True (Such developments are generally seen as critical when addressing future industry standards.)
– **Importance**: Important (Contextual background about industry trends affecting future negotiations.)
– **Relevance**: True

In summary, the most critical facts directly related to the probability of a strike in Q1 2025 concern the specifics of the agreement reached, how it addresses automation issues, and how it potentially stabilizes tensions impacting the likelihood of future disruptions. Political credit and the exact number of workers affected, while notable, are less relevant to assessing strike probabilities.

Fact-checked summary:
On January 9, 2024, the International Longshoremen’s Association (ILA) and the United States Maritime Alliance reached a tentative agreement that prevented a potential strike, which could have disrupted East and Gulf Coast ports. The primary issue at hand was the adoption of automation in ports, and the agreement included limitations to protect dockworkers’ jobs while allowing for technological advancements. This development addresses a significant sticking point and stabilizes tensions, which is critical to the discussions about balancing technological progress with job security in the shipping sector. This makes it an important factor in assessing the probability of a strike occurring in Q1 2025, as it potentially reduces immediate tensions and the likelihood of disruptions caused by unresolved issues over automation.

OpenAI gives a base rate of 0.1

The following were considered in order to produce this base rate:
The base rate of 0.1 was determined by considering the historical frequency of labor strikes in the shipping sector, particularly after new agreements. Typically, once a tentative agreement is reached, it is indicative of a period of labor peace, at least for the short- to mid-term. The agreement just over a year ago suggests that both parties likely have incentives to avoid a strike, particularly given progress on the contentious issue of automation. The probability of a strike in Q1 2025 is further reduced by the agreement’s likely inclusion of clauses or mediating structures designed to address and mitigate arising disputes.

Some potential divergent considerations that might effect the base rate:
There could be factors that deviate from the base rate. If unexpected macroeconomic changes occur, such as financial downturns or significant trade disruptions, they might affect the industry’s stability and revisit the terms of the agreement. Additionally, if there is a surge in automation efforts that dockworkers perceive as threatening, despite the agreement’s terms, it could provoke renewed tensions. Political changes or leadership shifts within labor organizations can also influence the likelihood of escalating tensions.

The following chain of events are necessary for the question to resolve positively:
– The tentative agreement reached on January 9, 2024, between the International Longshoremen’s Association (ILA) and the United States Maritime Alliance must break down. (Likelihood: Unlikely)
– Significant unresolved issues or new disputes must arise, potentially related to automation or other labor concerns. (Likelihood: Unlikely)
– An escalation in tensions to a level where dockworkers feel compelled to strike despite the existing agreement. (Likelihood: Unlikely)

Querying Claude (AI predicts: 0.15 – confidence: 7)
Querying Mistral (AI predicts: 0.15 – confidence: 6)
Querying OpenAI (AI predicts: 0.05 – confidence: 7)

Question Type: Binary
Median from LLMs: 0.15
Base rate: 0.1 (from OpenAI)
SD: 0.05
Confidence: 7
Conf Mode: Normal
Mellers: 0.07
Reverse Mellers: 0.24
Theory of Mind: 0.2 (What did the LLMs think other LLMs predicted?)
Beta Distribution: 0.14
Close Type: A (B = cautious # closer to 50%; A/C = closer to extremes)
# LLM responses: 3

Model value: 0.15

We queried Claude, Mistral, OpenAI
A summary of what the responding AIs considered is as follows: The consensus among the AIs is that the probability of a strike in Q1 2025 is low due to a tentative agreement reached on January 9, 2024, between the International Longshoremen’s Association and the United States Maritime Alliance. This agreement, which addresses major contentious issues such as automation, demonstrates both parties’ willingness to negotiate and suggests a period of labor peace. Historically, strikes are rare following significant agreements, especially when new provisions address job security concerns amidst technological advancements. However, some potential risks could elevate strike chances, including unexpected challenges in implementing the agreement, external economic pressures, shifts in union leadership, macroeconomic changes, or accelerated automation efforts. Despite these risks, the structured negotiation channels in place imply a commitment to resolving any emerging disputes efficiently





How many seats will the Conservative Party win in Canada’s next federal parliamentary election?

Welcome to the Multi-AI Oracle. Today:
*
less than 172: 15%
* between 172 and 205: 40%
* between 206 and 240: 30%
* more than 240: 15%

Obtaining outside newsfeed. News from various sources. Based on the most recent news articles, here’s a summary of the key developments in Canadian politics:

Most recently (within the last 24-48 hours), there’s significant focus on Canada’s response to Donald Trump’s potential return and threatened tariffs. Conservative Party leader Pierre Poilievre has promised to “stand strong” against Trump’s tariff threats, while also pledging to reverse recent capital gains tax changes implemented by the Liberal government. The context for these developments is the anticipated 2025 federal election, where current polling shows the Conservative Party surging as Prime Minister Justin Trudeau’s popularity declines. The Liberal Party is in transition, with Trudeau having announced he will step down and former Bank of England governor Mark Carney emerging as a potential successor for Liberal leadership.

Key election-related developments include:
– New rules for federal leaders’ debates requiring parties to meet 2 of 3 qualification requirements
– Increased scrutiny of policy differences between parties, particularly on issues like CBC funding (which Conservatives promise to cut)
– Growing focus on how Canada would handle U.S. relations under potential conservative leadership
– Questions about whether Poilievre represents a Trump-style approach to politics (though analysis suggests significant differences between the two leaders)

The situation remains fluid with both major parties positioning themselves for the upcoming campaign while dealing with immediate challenges like U.S. trade relations and domestic economic issues.

Summarizing outside newsfeed:
“Recent developments in Canadian politics highlight significant attention on the country’s response to potential U.S. tariff threats from a possible Trump presidency return. Conservative leader Pierre Poilievre has vowed to oppose these tariffs and pledged to reverse recent Liberal tax changes, amidst the backdrop of the approaching 2025 federal election. The Conservative Party is reportedly gaining ground as Prime Minister Justin Trudeau’s popularity wanes. Trudeau’s announcement to step down has fueled speculation that Mark Carney, former Bank of England governor, may succeed him as the Liberal leader. Key election issues include new qualifications for federal leaders’ debates, differing party policies such as CBC funding cuts proposed by the Conservatives, and the handling of U.S. relations under possible conservative leadership. There is ongoing debate about whether Poilievre aligns with Trump-style politics, though analyses indicate notable differences. The political landscape is dynamic, with both major parties navigating domestic and international challenges ahead of the election.”

Fact-checking news summary:
Below are specific facts from the summary along with their analysis:

1. **Fact**: Recent developments in Canadian politics highlight significant attention on the country’s response to potential U.S. tariff threats from a possible Trump presidency return.
– **Truth**: True. Speculation about future U.S. tariffs could arise with a shift in U.S. leadership.
– **Importance**: Important. This fact concerns international relations, a significant issue in federal elections.
– **Relevance**: True. The Conservative Party’s stance on tariffs could affect election outcomes.

2. **Fact**: Conservative leader Pierre Poilievre has vowed to oppose these tariffs and pledged to reverse recent Liberal tax changes.
– **Truth**: True. Pierre Poilievre has publicly criticized certain Liberal policies, focusing on economic issues, though specific stances on hypothetical tariffs would need current confirmation.
– **Importance**: Important. Policies on tariffs and tax changes are often central to political campaigns.
– **Relevance**: True. These policy positions could influence voter decisions.

3. **Fact**: The Conservative Party is reportedly gaining ground as Prime Minister Justin Trudeau’s popularity wanes.
– **Truth**: True. Historically, mid-term polling and recent elections have shown fluctuations in party support, with opposition parties often gaining ground.
– **Importance**: Important. Popularity trends can impact seat projections in elections.
– **Relevance**: True. This could be a key factor in determining the number of seats won.

4. **Fact**: Trudeau has announced his intention to step down, fueling speculation about Mark Carney as the new Liberal leader.
– **Truth**: False. As of my last update, Trudeau had not announced any intention to step down. While speculation about Carney is common, it’s not confirmed by any announcement from the current Prime Minister.
– **Importance**: Critical. Leadership changes significantly affect election dynamics.
– **Relevance**: True, if it were accurate, but irrelevant due to its inaccuracy.

5. **Fact**: Key election issues include new qualifications for federal debates, party policies like CBC funding cuts proposed by Conservatives, and handling of U.S. relations under possible conservative leadership.
– **Truth**: True. Debate qualifications and media funding are ongoing political discussions in Canada. Relations with the U.S. are always a significant issue.
– **Importance**: Important. These issues are typical focal points in Canadian federal elections.
– **Relevance**: True. They directly affect election campaigns and strategy.

6. **Fact**: There is ongoing debate about whether Poilievre aligns with Trump-style politics, though analyses indicate notable differences.
– **Truth**: True. Political analysts often compare leaders and their policies, and there is public discourse concerning Poilievre’s political style compared to other leaders like Trump.
– **Importance**: Lesser. While interesting, it’s more about public perception than policy change.
– **Relevance**: True. Voter perception can influence election outcomes.

7. **Fact**: The political landscape is dynamic, with both major parties navigating domestic and international challenges ahead of the election.
– **Truth**: True. Such circumstances are almost always present in political settings, especially close to elections.
– **Importance**: Unimportant. It’s a general observation applicable to any election.
– **Relevance**: True. While true, it’s too vague to directly impact specific predictions.

In summary, while most of the facts are true and somewhat relevant, the incorrect information about Trudeau stepping down is critical and would have misled the analysis if believed.

Fact-checked summary:
The Conservative Party’s potential performance in Canada’s next federal parliamentary election could be influenced by several true and relevant factors. Conservative leader Pierre Poilievre’s stance on opposing possible U.S. tariffs and reversing recent Liberal tax changes is important, as policies on tariffs and taxes play a central role in political campaigns and can sway voter decisions. Additionally, the Conservative Party is reportedly gaining ground amid waning popularity for Prime Minister Justin Trudeau, with historical trends showing that shifts in support can significantly impact seat projections. Key election issues include new qualifications for federal debates, proposed Conservative cuts to CBC funding, and the handling of U.S. relations under potential Conservative leadership, all of which are relevant as they directly affect campaign strategies and influence voter preferences. While there is public discourse about whether Poilievre aligns with Trump-style politics, analyses indicate significant differences. This can affect voter perceptions, further influencing the election outcomes. These factors, while speculative regarding actual seat numbers, highlight the Conservative Party’s strategic positioning and potential voter appeal.

Querying Claude (AI predicts: [“less than 172”: 0.35, “between 172 and 205”: 0.4, “between 206 and 240”: 0.2, “more than 240”: 0.05] – confidence: 5)
Querying Mistral (AI predicts: [“less than 172”: 0.15, “between 172 and 205”: 0.4, “between 206 and 240”: 0.3, “more than 240”: 0.15] – confidence: 6)
Querying OpenAI (AI predicts: [“less than 172”: 0.12, “between 172 and 205”: 0.25, “between 206 and 240”: 0.37, “more than 240”: 0.26] – confidence: 6)

Question Type: Multiple Choice
Confidence: 6
# LLM responses: 3

Model value:
* less than 172: 0.15
* between 172 and 205: 0.4
* between 206 and 240: 0.3
* more than 240: 0.15

We queried Claude, Mistral, OpenAI
A summary of what the responding AIs considered is as follows:

The Conservative Party in Canada is currently positioned for a potentially strong performance in the next election, largely driven by Pierre Poilievre’s leadership and declining support for Prime Minister Justin Trudeau. Historically, Conservative performance has ranged between 120-170 seats, but they may improve upon this by capitalizing on issues like opposition to U.S tariffs and tax reforms, as well as appealing to voter sentiments with potential cuts to CBC funding. However, achieving more than 206 seats may require a significant and rare electoral shift. The party’s gains depend on various factors, including the successful management of public perception and campaign strategies. Despite a generally optimistic outlook for the Conservatives, unforeseen events or scandals could alter voter sentiment, alongside potential economic changes and international events which could impact their projected momentum. The party must also navigate the potential negative association with Trump-style politics, as any alignment with such views might alienate moderate voters.

Runtime for all forecasts on this page: 212 seconds.

Past Multi-AI Oracle Forecasts

Guide to our Multi-AI Oracle’s Reports 

Median: We collect the output values from the LLMs in an array. We then take the median value. This is between 0 and 1 (i.e. 0.15 = 15%).
Base rate: Currently we take OpenAI’s value as the base rate. We are working on a better value using a different prompt that takes into account process. The base rate helps to determine if the median makes sense.
SD: The standard deviation between the median and the base rate.
Confidence: We query each of the LLMs on how confident they are of their predictions (between 0 and 10) and take a median. Because the LLMs tend to be overconfident, we take anything lower than 6 as being low confidence. This factors into the overall model.
Conf Mode: Based on the confidence value. >=9 is high confidence. Below 6 is low confidence (this is also triggered by an exceptionally high SD).
Mellers: This refers to Barbara Mellers, specifically a paper she wrote that includes a formula for moving values towards an extreme (i.e. 0 or 1).
Reverse Mellers: This uses the formula from above, but with a sub-1 coefficient to move the values closer to 50%.
Theory of Mind: We ask the LLMs what they think other LLMs would predict. We hope that this makes them consider the questions more deeply.
Beta Distribution: Currently unused, but possibly of interest. This is based on the median, the base rate and the SD.
Close Type: We noticed that the appropriate base case for some questions is closer to the extremes, while others are closer to 50%. When we have a low confidence value, this helps us to determine whether to extremize or de-extremize the value. ‘A’ implies closer to zero. ‘B’ implies closer to 50%. ‘C’ implies closer to 100%.
# LLM Responses: The count of LLMs that responded. We query 5 currently, but they have a relatively high API failure rate.
Model value: For normal confidence mode, this is the median. For high confidence, this is the Mellers value. For low confidence, this is either the Mellers or Reverse Mellers (depending on the Close Type).

Author