“What’s your p(doom)?” I can’t get away from that question these days - and unfortunately it is a distraction from the real debates we need to be having. Instead of debating whether we are rushing towards the possibility of an AI so powerful that it represents an extinction-level event for humanity, we should be spending our effort trying to make the best positive outcomes from AI more likely, and the negative ones less likely.

We can debate whether we are headed towards a superintelligence that acts like another species, whether that intelligence can be contained by humans who are less intelligent, and whether it might wish us ill or be indifferent to our demise. Putting a percentage on the possibility of impending human extinction sounds scientific and precise when in most cases it is neither. It is ironic that people in an industry built on using statistics from enormous data sets to teach machines are so quick to make specific predictions with so little data available.

Why do I say that? Well, there are two ways to make an estimate like this. The first is to look at historical analogs and reason by analogy. Look at the base case and adjust. This is great for estimating the probability that an incumbent will win a political race (very high), because there are lots of examples of both outcomes. We have exactly zero examples of doom occurring - so this method is of no help. The second is to lay out plausible chains of events that would lead to the outcome and put a probability on each step.

But most of the pessimistic predictions you hear aren’t based on either of these methodologies - they’re really just a gut feeling decorated with a percent sign. The most robust attempt at this approach comes from researcher Joe Carlsmith, who broke AI catastrophe into six steps - from “we build advanced, agentic AI” to “misaligned AI disempowers humanity” - and multiplied his estimates for each to get about 5% (he’s since raised that to over 10%). But that was by 2070, not the “next decade” timeframe most people inside the labs cite.

And I really do appreciate the more rigorous approach as it gives us something real to debate. For instance, I think his middle steps - that aligned AI is much harder to build than misaligned AI, that misaligned AI will seek power in ways that seriously harm us, and that this scales to full human disempowerment - are all far too high. And the answer is extremely sensitive to those individual guesses. Change them a bit (or add in a few more steps) and you can get a wildly different final answer. The real challenge is that we don’t have enough data to accurately predict any of these possibilities.

I will fully admit that there are some potential bad outcomes from AI acceleration - some very bad indeed. But when I think through the events required for things to go really wrong, I see three major reasons I believe our odds are better than the doomers think - and they point the way to the things we can do to improve the odds for a positive outcome from AI.

First, the likely harm to humans comes from impacts in the physical world - and I think the doomers dramatically overestimate how easy it will be for rogue AI agents to control physical infrastructure - and how quickly. Let’s take one of the earliest examples used by the safety crowd to talk about alignment - the rogue paperclip factory. The basic idea here is that a misaligned AI with the goal of optimizing paperclip output essentially takes over all of the world’s resources to do so - at a huge cost to humans.

I know this sounds a bit silly, but even Daniel Selsam, an OpenAI researcher who went public with his concerns this September (in the same wave of insider warnings as Jacob Coxon’s departure from Anthropic), described a doom scenario this way: “Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.” Not so far off from our runaway paperclip factory.

I can understand how and why an AI system might attempt such a thing - but there are so many bottlenecks where humans can exert control over these systems and prevent “full human disempowerment”. We’re talking power plants, steel mills, mines and chip fabs. The odds of each of those being taken over are small. Multiply them together and you get something dramatically smaller than the 40% Carlsmith assigns to the whole step.

Furthermore, for AIs to reach this level of capability, they will need more data centers - meaning gaining control over even more power generation capacity, land, construction crews, advanced chips. These resources are hard to come by (just ask anyone trying to get more Nvidia GPUs today or build a data center). It’s not believable that an AI, even a superintelligent one, will simply be able to take over control of all of these resources. And indeed, maintaining human control and authority over physical bottlenecks is an obvious way to limit the potential destructive downside of a rogue AI.

Second, there is an implicit assumption underlying these predictions that intelligence begets power and control. It doesn’t take a particularly in-depth survey of our current political landscape to find plenty of counterexamples. The line from the type of intelligence AIs are developing to real-world power is not clear at all. I’m sure many will claim that AIs will just develop whatever skills it is that allows people to gain power - but there is no evidence yet that we are training (or can train) AI systems to do this, and claims of exponential increases in intelligence rely on tests that involve almost exclusively tasks like science, math, and coding. How much authority and direct power we hand over to any AI system is up to us - and making smart decisions on this front can dramatically alter the likelihood of positive outcomes versus negative ones.

Third, it’s not clear whether AIs will be focused on improving human outcomes, fighting humans, or entirely indifferent to us. When asked for specifics on how AI doom occurs, one common refrain is to compare the question to chess: “If you were to play Garry Kasparov in chess, I’m confident you would lose even if I don’t know exactly how.” That’s fair - I’m sure I would lose to Garry Kasparov too! But, the real world doesn’t look much like a two-person game with one winner and one loser. Why are we playing chess against AIs?

Whether we end up with AI that is competing with us directly, indifferent to us, or actively aiding in human flourishing depends a lot on how we choose to approach model training, tuning, and more specific areas like model alignment.

I might not be so opposed to all the p(doom) talk if it was pushing forward reasonable discussion about how we protect against real AI risks. But so far it seems to be having the opposite effect. Outlandish claims about near-term “doom” seem unhinged to the general public (and many in the industry) and cause many to tune out the whole discussion.

Even more concerningly, AI safety is at risk of becoming a partisan issue - not a good recipe for progress. Indeed, this trend actually makes it harder to make the right choices and investments in areas like model alignment, safety research, or regulatory approach to push us in the positive direction by pushing people to either pick a side or tune out completely.

And we do need progress. While I worry less about rogue AIs causing human extinction, I do worry about bad human actors using AI to greatly amplify their impact (let’s be honest, we have as many human alignment issues as we do AI alignment issues). If you believe superintelligence is coming in the next five years, my concerns look like a distraction. I’ll admit my view assumes longer timelines, and a world where humans get time to respond.

But I don’t think the remedies are in competition. Whether you’re worried about a rogue AI or a human terrorist using a current AI, controlling access to bio precursors and sensitive lab equipment closes off one of the most plausible vectors of harm. Hardening our cyber infrastructure against potential attacks is useful whether those hacks are directed by an adversarial AI or a foreign government.

That’s defense in depth: it pays off whichever risk turns out to be real. I’m also all for the cheap insurance - alignment research, independent evaluators with real access, transparency about incidents. What I’m skeptical of are the expensive bets, like pauses and outright bans, that rest on a number nobody can actually estimate.