A short introduction earns its place by leaving a useful distinction behind. This essay separates systems that push us toward what is easy to measure from systems that acquire influence in more deliberate ways.
I would choose it when time is the binding constraint. It gives the next conversation a structure without requiring a tour through every part of the field. You can return to either story later and ask which developments would make it more or less plausible.
Get an email when a guide is updated.
Picks change as better pieces appear. A confirmation link arrives first; unsubscribe from any email.
What failure looks like
Paul Christiano · 2019
Two failure stories you can carry away from a single short reading session.
Read What failure looks like ↗- Format
- Essay
- Commitment
- 15 min
Where it falls short
The compression is the tradeoff. You will need other readings to examine training methods, technical evidence or the feasibility of particular countermeasures.
The rest of the short list
Silicon Valley Is Turning Into Its Own Worst Fear
A useful counterweight, especially for readers interested in corporate incentives. It does a different job from explaining the principal failure stories.
Without specific countermeasures, the easiest path…
The extra detail is worth the time when you want to understand training incentives. It does not fit the brief of a short first sitting.
Who is this for?
You have one short sitting and want more than a list of reasons to feel concerned.
Why you should trust me?
For the last nine years I have built LessWrong and the AI Alignment Forum, where professionals and independent researchers discuss this subject. Along with the pieces themselves, I have read rebuttals, failed ideas and more comment threads than I would recommend.
I built and ran the LessWrong Annual Review, which evaluates the writing each year. I have also worked alongside people in the field as an editor, web developer, event host and occasionally an argumentative commenter at 2am.
I have watched arguments and research areas take shape, and I have watched some of them fail.
Thanks for reading. Subscribe to hear when this guide, or any other, changes.
No schedule, no digest: only a short note when a category gets a new pick or a rewrite. Unsubscribe from any email.