Back to index
OtherAround 380 pages across 11 chapters, a few weeks of evening reading·Roughly $50 in print, or included with an O'Reilly subscription

Designing Machine Learning Systems (Chip Huyen)

4.4

Still the best single explanation of why machine learning projects fail after the notebook works. It predates the foundation model era in places, and you should read it knowing that.

What We Liked

  • Treats data engineering, deployment and monitoring as the real job rather than an appendix
  • The chapter on data distribution shifts is worth the cover price on its own
  • Written by someone who has actually shipped systems, so the trade offs are real rather than theoretical
  • Concept led rather than tool led, which is why it has aged far better than most technical books
  • Genuinely readable, with case studies and references instead of walls of code

What Could Be Better

  • Published in 2022, so the foundation model and LLM application material is thin or absent
  • Assumes you already know how to train a model, which makes it the wrong first book
  • Some tooling references have dated even though the principles have not
  • Light on hands on exercises, so you have to supply your own project to practise against
  • Her newer AI Engineering covers the LLM application layer that this book barely touches

Detailed review

There is a moment in most machine learning careers where you realise the modelling was never the hard part. You have a notebook that produces a good number on a held out set, and then somebody asks you to make it serve predictions to real users, keep working next quarter, and not fall over quietly when the upstream data changes. Nothing in the courses prepared you for that. Chip Huyen's book is the clearest map I know of that territory, and it has held up remarkably well for something written before the current wave.

The structure walks the actual lifecycle rather than the academic one. It opens with when you should and should not use machine learning at all, which is a question the field is bafflingly reluctant to ask, then moves into framing a problem as an ML problem, data engineering fundamentals, training data, feature engineering, model development and offline evaluation, deployment and prediction services, data distribution shifts and monitoring, continual learning and testing in production, the infrastructure and tooling around all of it, and finally the human and organisational side. That ordering is deliberate and it is correct. Most books put modelling at the centre and treat everything else as plumbing.

This one puts the system at the centre and treats the model as one component that happens to have a habit of degrading. The single best chapter is the one on data distribution shifts and monitoring. I have watched more projects die from this than from any modelling failure, and it usually happens silently. Covariate shift, label shift, concept drift, the difference between them and why your monitoring needs to catch each differently, what you can actually detect without labels, how long your detection window should be.

This material is scattered across a hundred blog posts of varying quality and it is assembled here properly, with the practical caveats attached. If you read one chapter in a bookshop, read that one. The training data and feature engineering chapters are similarly grounded. Sampling strategies, class imbalance, labelling under uncertainty, data leakage and the specific ways it sneaks in, when feature stores earn their complexity and when they are a solution looking for a problem.

The tone throughout is that of a senior engineer explaining why a tempting shortcut will hurt you in six months, which is the most valuable kind of technical writing and the rarest. There is also a chapter on the human side, covering how ML teams interact with the rest of an organisation and why the responsibility for a model in production is so often unclear. That material tends to get cut from technical books and it should not be. Now the honest limitation, and it matters.

This came out in 2022. It was written for a world where the default assumption was that you train your own model on your own data. An enormous amount of current work does not look like that at all, it looks like calling a foundation model through an API and building a system around it, and the specific problems of that world, evaluation without ground truth, retrieval quality, prompt versioning, inference cost, agent orchestration, are not in this book because they were barely problems yet. Huyen knows this, which is why she wrote AI Engineering, and that book is the right companion.

If your job is building on top of foundation models, read AI Engineering first and come to this one for the data and monitoring foundations underneath. If your job is training and serving your own models, read this one first. It is also not an introductory book. It assumes you know what a validation set is, that you have trained something, that terms like precision and recall do not need defining.

Somebody who has finished a beginner course and reads this will nod along without absorbing much, because the value here is in recognising problems you have already had. Read it after your first real project rather than before. And it is not a build along. There is code, but this is a book about decisions rather than implementations, so you need a project of your own running alongside it or the ideas will not stick.

My 4.4 reflects a book that is excellent at what it does and is now covering roughly two thirds of the modern job rather than all of it, through no fault of its own. What it gets right is the part that will still be true in ten years. Data is where your problems come from, production is where your model meets reality, and monitoring is the difference between a system that works and a system that appears to work. Nothing about foundation models has changed any of that.

Buy it, read it slowly, and stop being surprised when things break.

[ final ]

The verdict.

Read it if you are moving from building models to running them. Pair it with AI Engineering if your work is on top of foundation models rather than under them.