Superintelligence

Nick Bostrom, 2014

If machines someday surpass humans in general intelligence, our fate could depend on their goals. How superintelligence might arrive, why controlling it looks quite difficult, and what to do now.

Get the app Read on the web

16 parts, about 32 minutes to read.

Why read it

Argues that a machine smarter than us could seize decisive control, and that we may get only one chance to set its goals safely
Gives you the core concepts behind AI risk debates, from the orthogonality thesis to the treacherous turn and value loading
Reasons carefully through takeoff speeds, control methods and strategy, marking how confident it is in each claim

Who it is for

Readers who want the foundations of AI safety arguments
Technologists building ever more capable AI systems
Policy thinkers weighing long term technological risk

What the summary covers

The sparrows and the owl2 min
Past developments and present capabilities2 min
Paths to superintelligence2 min
Forms of superintelligence1 min
The kinetics of an intelligence explosion2 min
Decisive strategic advantage2 min
Cognitive superpowers2 min
The superintelligent will2 min
Is the default outcome doom?3 min
The control problem2 min
Oracles, genies, sovereigns, tools2 min
Multipolar scenarios2 min
Acquiring values2 min
Choosing the criteria for choosing2 min
The strategic picture2 min
Crunch time2 min

Recommended by

Stuart RussellProfessor of computer science at UC Berkeley
Nick Bostrom makes a persuasive case that the future impact of AI is perhaps the most important issue the human race has ever faced.
Martin ReesPast President of the Royal Society
Those disposed to dismiss an 'AI takeover' as science fiction may think again after reading this original and well-argued book.
Max TegmarkProfessor at MIT
This superb analysis by one of the world's clearest thinkers tackles one of humanity's greatest challenges: if future superhuman artificial intelligence becomes the biggest event in human history, then how can we ensure that it doesn't become the last?
Olle HäggströmProfessor of mathematical statistics at Chalmers University of Technology
If this book gets the reception that it deserves, it may turn out the most important alarm bell since Rachel Carson's Silent Spring from 1962, or ever.
Bill GatesMicrosoft cofounder and philanthropist
I highly recommend this book.
Elon MuskCEO of Tesla and SpaceX
We need to be super careful with AI. Potentially more dangerous than nukes
Sam AltmanCEO of OpenAI
Nate SilverStatistician and founder of FiveThirtyEight

About the author

Nick BostromPhilosopher and founding director of the Future of Humanity Institute at Oxford
- Born 10 March 1973 in Helsingborg, Sweden, as Niklas Bostrom
- Earned a PhD in philosophy from the London School of Economics in 2000
- Taught at Yale from 2000 to 2002, then held a British Academy postdoctoral fellowship at Oxford until 2005
- Professor at Oxford from 2008 to 2024 and founding director of the Future of Humanity Institute from 2005 until it closed in April 2024
- Now founder and principal researcher at the Macrostrategy Research Initiative
- His books include Anthropic Bias (2002), Global Catastrophic Risks (2008), Human Enhancement (2009) and Superintelligence (2014)
- His work introduced ideas such as existential risk, the simulation argument, the vulnerable world hypothesis and the unilateralist's curse
- Named to Foreign Policy's Top 100 Global Thinkers list twice and to Prospect's World Thinkers list (editorial lists, not awards)

Read it alongside

Pushes backMore Everything Forever
Nick Bostrom sketches an argument for fearing that existential catastrophe is a plausible default outcome of superintelligence, and from an impersonal, secular view ranks reducing existential risk as our principal moral priority; Adam Becker argues the doom case is a hypothetical with several premises and that measurable present harms should not lose moral weight.