Deep reinforcement learning has a reputation for being the hardest subfield to learn, and the reputation is deserved. The mathematics is not the main obstacle. The obstacle is that the gap between the equations in a paper and code that actually trains is enormous, filled with implementation details that determine whether anything works and that papers routinely omit. Spinning Up exists to close that gap and it does it better than anything else I have used.
Josh Achiam wrote it while at OpenAI and the design decisions are consistently correct for learning. The reference implementations are single file, minimally abstracted, and readable top to bottom. There is no framework, no plugin architecture, no configuration system. You can sit down with the PPO implementation and understand the entire algorithm in an afternoon, because the whole thing is in front of you.
Production RL libraries are the opposite. They are well engineered, heavily abstracted, and almost impossible to learn from, because following the logic means chasing calls through six files. The explanations are the other half. Each algorithm gets a page that states the intuition, develops the mathematics, and then points at the specific lines of code where each term appears.
That correspondence between equation and implementation is the thing that textbooks cannot do and papers do not bother to do, and it is where understanding actually happens. Sutton and Barto tells you what policy gradients are. Spinning Up shows you what they look like in a training loop. The key papers list is quietly one of the most useful things on the site.
Deep RL has an enormous literature and most of it is not worth your time. Having a curated, ordered list of the papers that actually matter, with brief notes on why, saves an enormous amount of unfocused reading. I still refer back to it. The honesty deserves credit.
There is a section on the practical difficulties of the field that states plainly that results are hard to reproduce, that performance varies wildly across random seeds, that hyperparameter sensitivity is severe, and that a substantial part of published progress is fragile. Coming from OpenAI, at the height of RL enthusiasm, that was an unusually candid thing to publish and it is still the most useful expectation setting a newcomer can read. Now the age problem, which is real. This has not been substantially updated in years.
Dependencies have moved, the Gym API changed to Gymnasium with breaking differences, MuJoCo licensing and installation changed, and the specified versions of things no longer install cleanly on a modern setup. Expect to spend time on environment problems before you run anything. Community forks and updated instructions exist and finding them is part of the experience. The algorithmic coverage stops where it stopped.
Policy gradients, VPG, TRPO, PPO, DDPG, TD3, SAC. That is the correct foundational set and it is not what people use at the frontier now. Nothing on offline RL, nothing on model based methods beyond a mention, nothing on the distributional approaches, and critically nothing on RLHF or the reinforcement learning used in language model post training, which is what most people asking about RL in 2026 actually want. That is not a fair criticism of a resource written before that mattered, and you should know the gap exists.
The prerequisites are real. Comfortable Python, working knowledge of deep learning and PyTorch or TensorFlow, probability and expectation, and enough mathematical maturity to follow a derivation. This is not an introduction to machine learning. Someone arriving without those will be lost by the second algorithm.
My four point three is for the clearest and most honest introduction to a genuinely difficult subject, with implementations designed for understanding rather than performance, and a curated path through a sprawling literature, marked down because it is effectively frozen, because the environment setup is now a real obstacle, and because it stops well short of what practitioners currently do. Learn from it, then leave it. Nothing has replaced it as the starting point.