A long argument can hide disagreements by moving too quickly between claims. A report organized around separable premises gives each disagreement its own place.
I would pick this for a reader ready to do some work. The point of an explicit estimate is that you can ask which inputs drive it and substitute your own judgments. A useful reading ends with a more precise disagreement, even if it leaves your overall view unchanged.
Get an email when a guide is updated.
Picks change as better pieces appear. A confirmation link arrives first; unsubscribe from any email.
- Introduction
- Six premises
- Timelines
- Incentives
- Probability estimate
- Objections
Is Power-Seeking AI an Existential Risk?
Joe Carlsmith · 2022
Find the premise you reject, and see what follows when you change it.
Read Is Power-Seeking AI an Existential Risk? ↗- Format
- Report
- Commitment
- 3 hr
Where it falls short
Numbers can suggest more precision than the underlying judgments support. Pay attention to dependencies between premises as well as to each individual estimate.
The rest of the short list
Without specific countermeasures, the easiest path…
Easier to follow if your main uncertainty concerns training. It covers a narrower part of the overall risk argument.
AI 2027
A more approachable way to see how developments might connect. The narrative is less convenient for isolating a disagreement about a single premise.
Who is this for?
You want the full argument laid out so you can challenge individual steps.
Why you should trust me?
For the last nine years I have built LessWrong and the AI Alignment Forum, where professionals and independent researchers discuss this subject. Along with the pieces themselves, I have read rebuttals, failed ideas and more comment threads than I would recommend.
I built and ran the LessWrong Annual Review, which evaluates the writing each year. I have also worked alongside people in the field as an editor, web developer, event host and occasionally an argumentative commenter at 2am.
I have watched arguments and research areas take shape, and I have watched some of them fail.
Thanks for reading. Subscribe to hear when this guide, or any other, changes.
No schedule, no digest: only a short note when a category gets a new pick or a rewrite. Unsubscribe from any email.