There is a gap in machine learning education that almost nobody fills. Courses teach you to train a model and evaluate it on a held out test set, and then they stop. What they do not teach is that your model will be wrong in six months, that nothing will alert you, that the accuracy metric you were watching will look fine because you have no labels in production, and that by the time somebody notices the business has been making bad decisions for a quarter. Evidently's courses are about exactly this and I have not found better free material on it.
The ML observability course is forty lessons across six modules, each with a video, written notes and code. The structure moves from why monitoring differs from ordinary software monitoring, through what to measure, into data quality, data drift, model quality without ground truth, and finally unstructured data including text and embeddings. The order is sensible and each module builds on the previous one. The data drift material is the standout and the reason I rate this highly.
Drift is a concept everyone in machine learning can name and very few can handle properly. The course does the actual work. It explains the difference between drift in your inputs, drift in the relationship between inputs and outputs, and drift in your target distribution, which have different causes and different responses. It goes through the statistical tests, population stability index, Kolmogorov Smirnov, Wasserstein distance, chi squared, and explains what each one is sensitive to and when it misleads.
Crucially it addresses the practical problem that if you test enough features frequently enough you will get drift alerts constantly and start ignoring them, which is how monitoring systems die. That warning alone is worth the time. The section on evaluating quality without labels solves the problem that defines production machine learning. In training you have labels.
In production you usually do not, or you get them weeks late, and you still need to know today whether something has broken. The course covers proxy signals, prediction distribution monitoring and delayed feedback loops. This is the hardest part of the job and it is barely taught anywhere. The language model evaluation material is newer and reflects where attention has moved.
Designing custom judges, evaluating retrieval systems, adversarial testing. The judge design content is the useful bit, because using a language model to score outputs is now standard practice and doing it badly, with a vague rubric and no validation of the judge itself, is also standard practice. The course treats the judge as something you have to evaluate before you trust it, which is the right stance and an uncommon one. The obvious caveat is that this is content marketing for a library.
The examples use Evidently, the workflows assume Evidently, and the intent is to make you an Evidently user. I think that is a fair trade. The concepts are library agnostic and would transfer to any monitoring stack, and the tool is open source rather than a sales funnel to something expensive. The bias shows in framing more than in substance.
It assumes prerequisites. You need to already be able to train a model, deploy it somehow, and read Python comfortably. That is correct scoping for the topic and it means beginners should go elsewhere first. The certificate needs cohort participation, which is a mild bait.
The material is fully free and self paced, and the paper requires signing up for a scheduled run with assignments. Reasonable, and it catches people who assumed otherwise. Production examples are cleaner than production. Real monitoring involves data arriving late, pipelines failing silently, schema changing without notice and labels appearing from three different systems at three different times.
The course teaches the concepts on tidy examples. That is a defensible teaching choice and worth knowing before you assume you are ready. Some of the older modules predate the current focus on language models, so a few things feel like they belong to a different era of the field. The fundamentals have not changed and drift is drift.
My four point two is for teaching the most neglected part of applied machine learning, doing it with real statistical depth rather than API calls, and giving it away, marked down for vendor framing, simplified production scenarios, and a certificate gate that the free positioning does not quite prepare you for. If you have ever put a model in production and had no idea whether it was still working, take this.