Most people who train models have a vague and wrong mental model of GPU memory. They think the model weights are the thing using it, and they are surprised when a model whose weights obviously fit will not train. The material here is the best public explanation of why. When you train with a common adaptive optimiser in mixed precision, the weights are a minority of your memory.
You are also holding gradients, two optimiser moment estimates per parameter, a master copy of the weights in higher precision, and activations from the forward pass waiting to be used in the backward pass. Add that up and the optimiser state alone can be several times the size of the model. Once you have internalised this, a great deal of otherwise mysterious behaviour becomes obvious, and you will make better decisions about batch sizes, checkpointing and hardware for the rest of your career. The staged partitioning approach is the central idea and it is taught well.
Stage one splits the optimiser state across your devices, so each holds only its share. Stage two splits the gradients too. Stage three splits the parameters themselves, gathering them just in time for each layer's computation and releasing them afterwards. Each stage saves more memory and adds more communication, and the documentation is clear about that progression and about what it will do to your throughput.
Offloading extends the same logic to slower storage, moving state to CPU memory or to disk, and the pages are honest that this buys you capacity at a real cost in speed. That honesty matters, because the marketing version of this technology tends to emphasise the models you can now fit and skip past how slowly they will train. The conceptual value survives leaving the library. Everything here about partitioning, communication overlap and activation checkpointing applies to the other distributed training frameworks, which implement variations on the same ideas.
Learn it once and you can read the others. Now the practical experience, which is rougher. The tutorials and the code drift apart. This is an actively developed research-adjacent project and the documentation has not kept pace.
You will find tutorials referencing configuration keys that have moved, examples built against older versions of the training libraries they integrate with, and instructions that fail in ways that require you to go and read the source to resolve. Budget time for this and expect to spend part of your week on GitHub issues rather than on your actual work. Configuration is a sprawling JSON object with a great many keys, some of which interact, and the reference material is not good enough. There are options whose effects are described in one sentence, options that are only meaningfully documented in a blog post, and combinations that fail at runtime rather than being rejected up front.
Getting a working configuration frequently means starting from someone else's and changing one thing at a time. The prerequisite gap is significant. The documentation assumes you know how distributed training works, what a collective communication operation is, what the launcher is doing, and how your cluster's interconnect affects performance. If you arrive without that, the pages will not build it for you, and you should go and learn distributed training basics first.
Debugging support is the biggest omission. This library fails in unfriendly ways. Processes hang. Memory errors appear on one rank and not others.
A misconfigured setting produces a stack trace deep in a communication library that tells you nothing about what you did wrong. Documentation that walked you through the common failure signatures would be enormously valuable and it does not exist. My three point seven is for the best available public teaching on the memory mechanics of large model training, with a clear progressive explanation of partitioning and honest treatment of what offloading costs. Marked down for tutorials that do not match the code, for configuration reference material that is not equal to the size of the configuration surface, for assuming a distributed background it does not supply, and for abandoning you completely when things break.
Read it for the concepts and be ready to fight the tooling.