People often ask me: how could an AI kill everyone?
The heart of the question is how an AI could become intelligent and powerful enough that killing everyone becomes an option. The question of what it could then do with that power is a separate one. I would start with the first.
The best read for how an AI could get to that position is AI 2027: a multi-year scenario following increasingly capable systems into the economy and into the work of building more powerful AI. Its graphs track capabilities, compute and economic change. The supporting forecasts give you something to inspect when you disagree with a turn in the story.
AI 2027
The scenario I’d hand over to someone asking how we get from useful assistants to systems we might no longer control. Read the story first; follow the research supplements where you want to check its reasoning.
Read AI 2027 ↗- Format
- Web scenario
- Commitment
- A long sitting
- Expect
- Graphs + forecasts
If you’d rather watch
If you would rather watch, We’re Not Ready for Superintelligence by Aric Floyd covers the AI 2027 story in video form. It is the companion option here; the report is where I would go to examine the assumptions.
We’re Not Ready for Superintelligence
A way into the same scenario if you are more likely to finish a video than a long report.
The underlying recommendation is AI 2027. I haven’t reviewed the video separately, so I’m leaving that judgment open.
- Format
- YouTube video
- Source
- AI 2027
- Review status
- Not yet reviewed
The rest of the short list
Some ways AI could kill us all
If you are willing to grant an AI capabilities far beyond ours and want to understand possible mechanisms of harm, my recommendation is Some ways AI could kill us all, a short essay by my colleague Ruben Bloom. It considers broad possibilities including biological threats and autonomous weapons, and addresses common objections about how software could affect the physical world.
Read the essay ↗Ajeya Cotra’s account of misalignment
The original pick for this category. I would use it for the question about incentives and training, then return to Ruben’s essay for the question about means.
If Anyone Builds It, Everyone Dies
The book-length option from the shortlist. It makes more sense as a further commitment once you know which part of the argument you want to investigate.
It Looks Like You’re Trying To Take Over The World
Gwern’s story is another way to make the idea tangible. I would put an explanatory essay before it: a story can make something imaginable without establishing that it is likely.
Who is this for?
People who would rather not die, and would rather everyone else not die either. Especially people who keep hearing researchers warn about AI and are struggling to see the connection to the chatbot they actually use.
An analogy about losing at chess to a grandmaster can illustrate a gap in ability. It leaves a lot unexplained about the real world. The readings above are for the missing steps.