There is a small set of documents that most working machine learning engineers have read and refer back to, and a disproportionate number of them were written by the same person. Karpathy ran computer vision at Tesla, was a founding member of OpenAI, taught the Stanford course that trained a large fraction of the field, and writes clearly, which is a combination that does not come up often. A Recipe for Training Neural Networks is the essential one and I would put it ahead of any course on this site for practical value per hour. It is about the process of getting a model to work, which is not what any textbook teaches and is what the job actually consists of.
Start with the data and look at it properly. Get a dumb baseline running end to end before adding anything. Overfit a single batch to prove the machinery works at all. Add complexity one piece at a time so you know what caused what.
Do not trust a result you have not tried to break. This is craft knowledge, the kind that normally transfers only by sitting next to someone experienced for a year, and it is written down in a few thousand words. Every engineer I know who read it recognised several of their own mistakes in it. The Unreasonable Effectiveness of Recurrent Neural Networks is the piece that got a lot of people into this field, and it is now partly a historical document because recurrent architectures have been displaced.
It remains a masterclass in explaining a mechanism. Build the simplest thing that demonstrates the idea, train it on something entertaining, show the output, and let the reader work out for themselves why it is remarkable. That approach to explanation is worth studying independently of the architecture it explains. Software 2.0 has aged into being obviously right in a way that makes it easy to underrate.
The argument was that a large class of programs would stop being written as explicit instructions and start being specified through data and optimisation, with the consequence that version control, testing, debugging and deployment all need rethinking. That was contentious when it was written. It is now roughly the situation everyone works in, and the practical implications are still being worked out. The writing style deserves attention on its own terms.
Posts are short, examples are minimal, code is included and readable, and there is no attempt to sound impressive. Where something is uncertain or where the author was wrong before, it is said. That combination of authority and honesty is why these pieces get recommended a decade after publication. Now the limitations.
This is not really a live blog. Posts appear rarely, sometimes years apart, and the archive is the point rather than the stream. That is fine for a resource and it means anyone subscribing expecting regular material will be disappointed. Some posts are historical now.
The recurrent network material describes architectures largely superseded, and a couple of the older practical posts assume tooling that has moved on. The underlying reasoning transfers and the specifics sometimes do not, and nothing on the page tells you which is which. Reading with a sense of when each piece was written is necessary. The material is scattered.
There is the original site, there is newer writing elsewhere, there are talks and the video series and various long posts on other platforms, and no single index brings it together. Anyone wanting to read all of it has to go looking, which is a small annoyance that a lot of people never bother to overcome. It assumes background. These are not beginner documents.
The training recipe post is written for someone who has already trained models and got confusing results, and a person who has never built one will read it and not understand what problem is being solved. Come back to it after your first project fails, at which point it will read completely differently. And there is no ordering. No suggested path, no beginner to advanced structure, just a list of posts.
For a resource this valuable that is a genuine shame, and the fix is easy enough to state. Read the training recipe first, read Software 2.0 second, and read the rest in any order you like. My four point seven is for a small collection of writing that has taught more working engineers more useful things than most institutions manage, held back only by being an archive rather than a living resource and by being scattered across the internet in a way that stops people finding all of it. The training recipe post is the single best free thing in this field.
Go and read it.