Placing this book properly matters. It came out in 2013, which was before the deep learning wave had fully arrived in public consciousness and long before language models made any of this a mainstream conversation. At the time, the position that advanced AI might pose an existential risk was held by a small and somewhat isolated group, mostly outside the major research institutions, and was treated by most working researchers as science fiction. The author is a documentary maker rather than a technologist, and what he did was go and interview those people at length, seriously, and write down what they said.
That has real value as a record. What holds up best is the conceptual material about why a sufficiently capable system pursuing an objective might behave in ways its designers did not intend. The explanation of instrumental convergence, the argument that a wide range of goals imply similar sub-goals like acquiring resources and resisting being switched off, is presented accessibly and remains the clearest part of the argument. The point that intelligence and goals are separable, so that great capability does not imply benign objectives, is likewise still live and still worth understanding.
Someone reading this will come away understanding the shape of the concern. The journalism is decent. Interviews are used well, the researchers come across as people rather than as positions, and the account of how marginal this view was at the time is valuable perspective for anyone who assumes the current debate has always existed. Now the problems, and the largest one is simply time.
Twelve years is a long time in this field and the technical picture the book describes is not the one we have. The systems that have actually arrived were not built the way the book anticipated, the capability profile is strange in ways nobody predicted, and the specific concerns about self improving recursive systems have not been what the last decade turned out to be about. That does not settle the underlying question, and it does mean the technical content is essentially obsolete. A reader could be seriously misled about the state of the field.
The evidential base is narrow. The interviewees are mostly people who already held the view the book is presenting, and the effect is a chorus rather than an investigation. Researchers who disagreed appear briefly and mostly as foils. For a book presented as an inquiry, that is a real weakness, and it makes the conclusions feel arrived at rather than reached.
The register is a problem for me. The writing reaches consistently for the alarming phrasing, the ominous chapter ending, the analogy designed to unsettle. There is a version of this argument that is made calmly and is far more persuasive for it, and this is not that version. Readers inclined to dismiss the concern will find plenty here to confirm that dismissal, which is a shame given that the underlying argument deserves better.
Counterarguments are handled thinly. The serious objections, about whether capability gains generalise the way the argument requires, about whether the goal-directed framing describes real systems, about the difference between competence and agency, are either absent or presented in weak forms. A book that engaged with the strongest version of the opposition would have aged much better. My two point eight is for readable journalism that took an unfashionable position seriously and captured a moment worth having on record, with an accessible explanation of the core conceptual argument.
Marked down heavily for a technical picture that events have overtaken, for an interview base that only points one way, for a tone that works against its own case, and for not engaging with the best objections. If you want to understand this argument today, read something written in the last three years and treat this as a historical document.