Most technical fields do not have a book that everybody in them learned from. Reinforcement learning does, and the remarkable thing is that after a couple of decades and a complete transformation in what the field can do, it is still the right recommendation. Sutton and Barto wrote something with genuine pedagogical intent rather than a reference dressed as a textbook, and the difference shows on every page. The build is the whole design.
It opens with multi armed bandits, which is reinforcement learning with the hardest part removed, and uses that stripped down setting to establish exploration and exploitation, action values, and why greedy is not enough. Then it introduces the full problem as a Markov decision process, and works through dynamic programming, Monte Carlo methods, and temporal difference learning in sequence. Each step adds one difficulty. Dynamic programming assumes you know the model.
Monte Carlo drops that assumption but needs complete episodes. Temporal difference drops that too. By the time you reach n step methods and eligibility traces you understand why each piece exists, because you saw the specific limitation it was invented to overcome. That is what good teaching looks like and it is much rarer in technical writing than it should be.
The second part moves to approximate solution methods, which is where the practical field lives. On policy prediction and control with function approximation, off policy methods and the deadly triad, policy gradient methods, and the actor critic framing. The discussion of why off policy learning with function approximation and bootstrapping is unstable is one of the most useful things in the book, because it names a failure mode that people otherwise rediscover painfully through debugging. The examples are chosen with unusual care.
Gridworld, the cliff walking problem that makes the difference between Sarsa and Q learning immediately visible, blackjack for Monte Carlo, the mountain car, the access control queuing task. None of them are decorative. Each is constructed to make a specific idea legible, and the cliff walking example in particular teaches the on policy versus off policy distinction better than any amount of prose could. The final chapters on psychology and neuroscience are unusual for a machine learning text and are worth reading.
Reinforcement learning came out of animal learning theory, and Sutton and Barto trace the connections explicitly, including the relationship between temporal difference errors and dopamine signalling. It gives the material an intellectual grounding that most machine learning books lack entirely, and it is written with obvious affection for the history. The book is free, in full, from Sutton's own site, alongside errata, solutions, code in several languages, and teaching slides. That is the model everybody should follow.
Now the limits, and they are about scope rather than quality. This is a book about reinforcement learning as a mathematical framework. Deep reinforcement learning appears mainly in chapter sixteen as a set of case studies, including the Atari work and AlphaGo, and those are well written summaries rather than instruction. You will not come out of this able to implement DQN or PPO.
The book gives you the theory that those methods rest on, which is genuinely the harder and more durable half, but you need a separate practical resource for implementation. Spinning Up, the Hugging Face deep RL course and various university lecture series all fill that gap. The second edition is from 2018, and the most consequential development since then, at least in terms of what reinforcement learning is now used for daily, is the application of preference based methods to language models. RLHF, direct preference optimisation and everything downstream of them are absent.
Somebody arriving here because they want to understand how instruction tuned models are trained will find the foundations they need and none of the specifics. The tabular first half also demands patience. It is long, and a reader who wants to get to neural networks will be tempted to skim it. That is a mistake, because almost every problem people have with deep RL is a tabular concept they never solidified, but the temptation is real and the book does not do much to counter it.
My 4.6 is high because I genuinely struggle to fault the execution against its own aims. It is clearly written, carefully sequenced, generous with its material, and it teaches a way of thinking about sequential decision making that transfers well beyond the specific algorithms. The deductions are for a scope that stops where a lot of current practical interest begins, and for the absence of anything in the way of implementation. Read it first, do the exercises, then go and build something with a library.