This book is why people say orthogonality thesis, instrumental convergence and treacherous turn, and if you spend any time in discussions about AI risk you will encounter all three within a week. That alone makes it historically important, and importance and enjoyment are unrelated quantities. Let me start with what it genuinely achieves. The central argument about goals is the strongest part and it holds up.
The orthogonality claim is that intelligence and goals are independent axes, so a highly capable system can pursue an objective that strikes us as absurd, and being capable does not make it wise in any sense we would recognise. The instrumental convergence claim is that a wide range of final goals imply the same intermediate goals, because almost any objective is served by continuing to exist, by acquiring resources, and by not having your objective changed. Put those together and you get a specific worry that does not depend on the system being malicious, only on it being competent and pointed at the wrong thing. Whatever you make of the timelines, that argument is clean, and the book states it better than the many summaries of it.
He is also more careful than his reputation. The book that circulates in argument is a confident prophecy of doom. The actual text hedges constantly, offers multiple scenarios, gives explicit probability caveats and repeatedly says that the author does not know. People who cite it have often not read it.
And in 2014 almost nobody in a serious institutional position was treating this as a subject worth attention. This book, more than anything else, made it discussable in academic and policy settings. The safety research community that exists now traces a good deal of its funding and its framing to arguments that got their first careful statement here. Now the criticisms, and they are substantial.
The prose is punishing. It is written in an academic philosophy register with long qualified sentences, nested conditionals and a taxonomic instinct that produces list after list of scenario types, each subdivided further. There are chapters that read like a filing system for possible futures. Plenty of intelligent, motivated readers have started this book and put it down at page ninety, and that is a failure of the writing rather than of the reader.
Technically it is out of date in a way that matters. It was written before the transformer, before large language models, before anything resembling the current landscape. The paths to advanced capability it considers most seriously, including whole brain emulation and recursive self improvement by an agent rewriting its own source, are not the path the field actually took. Scaling a fairly simple objective over enormous quantities of data and compute is barely anticipated.
That does not invalidate the goal arguments, and it does mean that the concrete texture of the book, the how rather than the what if, describes a world that did not happen. Long portions are unfalsifiable speculation. Chapters on the dynamics of a takeoff, on multipolar outcomes, on the strategic situation between competing projects, are careful reasoning built on premises nobody can check. Read as philosophy that is legitimate.
Read as forecasting it is a very elaborate structure resting on assumptions chosen by the author. And the reception has been a mess through no fault of the text. It has been used as the definitive proof that AI will kill us all, and used as the archetype of overwrought science fiction dressed as scholarship, by people on both sides who have mostly read the same three summaries. The actual book is more tentative and more boring than either camp needs it to be.
My three point four is for a genuinely influential work whose core argument about goals remains worth understanding, dragged down by prose that repels most readers, by technical framing that the last decade has left behind, and by long speculative passages that cannot be evaluated. If you want the ideas without the slog, the chapters on the superintelligent will and on the control problem contain most of what people quote. If you want a book length treatment of the same worry that is better written and more current, Stuart Russell's Human Compatible does the job with a fraction of the pain.