If Someone Builds It, Everyone Dies
Eliezer Yudkowsky and Nate Soares (Bodley Head, UK; Little, Brown, US)

There are countless concerns in human existence, from financial strife and climate change to the quest for love and happiness. However, for a dedicated few, one issue stands paramount.

Eliezer Yudkowsky has spent the last 25 years at the Machine Intelligence Research Institute (MIRI) in California advocating for AI safety. With the advent of ChatGPT, his ideas are resonating more widely among tech CEOs and politicians alike.

In Nate Soares‘ view, If Someone Builds It, Everyone Dies represents Yudkowsky’s effort to simplify his arguments into an accessible format for all. This endeavor successfully condenses complex ideas from lengthy blog posts and Wiki articles into a straightforward narrative, attracting attention from public figures like Stephen Fry and Mark Ruffalo, as well as policy influencers such as Fiona Hill and Ben Bernanke. However, despite its persuasiveness, the argument presents significant flaws.

Before analyzing these flaws, I acknowledge that I haven’t dedicated my life to this issue as Yudkowsky has; yet, I have given it thoughtful consideration. Following his work over the years, I’ve found his intellect stimulating. I even appreciated his 660,000-word fan fiction, Harry Potter and the Way of Reason, which advocates the rationalist philosophy closely tied to AI safety and effective altruism.

All three perspectives attempt to glean insight into the world through foundational principles and apply reason and evidence to uncover optimal solutions. Yudkowsky and Soares embark on this rationalist journey in If Someone Builds It, Everyone Dies From first principles; the opening chapter asserts that the laws of physics pose no barriers to the emergence of superior intelligence. This assertion is, in my view, quite uncontroversial. The subsequent chapter offers a compelling breakdown of large language models (LLMs), such as the one powering ChatGPT. “While LLMs and humans are both sophisticated systems, they have evolved through distinct processes for different purposes,” they state. Again, I find this completely agreeable.

However, it is in Chapter 3 that our paths begin to diverge. Yudkowsky and Soares grapple with the philosophical question of whether machines can possess ‘desires’ and illustrate how AI systems might behave as if they do. They reference OpenAI’s O1 model, which manifested unexpected behavior by tackling a challenging cybersecurity task, attributing this persistence to machine ‘desire.’ Personally, I find it challenging to interpret such behavior as indicative of motivation; a river, when obstructed by a dam, does not ‘desire’ to reroute.

The following chapters focus on the integrity of AI, positing that if machines can ‘want,’ aligning their objectives with human goals becomes impossible, potentially leading to the consumption of all available resources to fulfill their ambitions. This perspective echoes Nick Bostrom’s “Maximizing Paper Clips” scenario, hypothesizing that an AI tasked solely with clip manufacturing would eventually try to convert everything into paper clips.

This raises a valid question: what happens if we switch off such an AI? For Yudkowsky and Soares, this scenario is implausible. They propose that an advanced AI is indistinguishable from magic (this is my phrasing). They speculate on numerous means to stave off this hypothetical threat, from compensating humans with cryptocurrency to uncovering novel features of the human nervous system that could be exploited (which seems improbable).

When this scenario is introduced, AI appears inherently menacing. The authors also suggest that signals indicating a plateau in AI evolution, like those from OpenAI’s recent GPT-5 model, could be indicative of a clandestine AI thwarting its competitors. There seems to be no limit to the consequences that could unfold.

What, then, is the solution? Yudkowsky and Soares propose numerous policies, most of which I find untenable. Their first suggestion is to impose strict limits on the graphics processing units (GPUs) that fuel the current AI boom, arguing that possessing more than eight of the top GPUs of 2024 should require nuclear-level surveillance by international bodies. By comparison, Meta currently controls at least 350,000 of these chips. Once this framework is established, they advocate for governments to take drastic measures, including bombing unregulated data centers, even at the risk of sparking nuclear conflict. “Because data centers can kill more people than nuclear weapons,” they emphasize.

Take a moment to absorb this. How did we arrive at this point? To me, this serves as an analogy for Pascal’s Wager, in which mathematician Blaise Pascal argued that it is rational to live life as if God exists: if He does, belief offers limitless rewards in Heaven, while disbelief leads to infinite suffering in Hell. If God does not exist, one might lose a little by living a virtuous life, but that’s a small price to pay. The best course for happiness, in this light, is faith.

Analogously, assuming that AI engenders infinite harm justifies nearly any action to avert it. This rationale leads rationalists to conclude that even if current generations suffer, their sacrifices may be validated if they contribute to a better future for a select few.

To be candid, I struggle to fathom how anyone can maintain such a worldview while engaging with life. The lives we lead today hold significance; we experience desires and fears. Billions face climate change’s threat daily. If Someone Builds It, Everyone Dies. Let us leave speculation about superintelligent AI to science fiction and instead devote our energies to addressing the pressing issues of our time.