There are two useful things in this documentation and they serve different needs. The first is straightforward evaluation. You write a configuration file listing your prompts, the models you want to test, the inputs to run through them, and the assertions each output must satisfy. Run the command and you get a grid showing every prompt and model combination against every test case, with pass and fail marked.
It sounds mundane. It replaces the enormous amount of time teams waste arguing about whether the new prompt is better, or whether they should switch models, with a table anyone can read. I have watched a twenty minute meeting end because someone ran this and put the grid on screen. The assertion documentation is better than it needs to be.
There are deterministic checks, contains this string, matches this pattern, valid JSON, under this latency, below this cost. There are model graded checks, is this answer factually consistent with the reference, does it stay on topic, is the tone appropriate. The docs are clear that the deterministic ones are cheap and reliable and the model graded ones are expensive and approximate, and they encourage you to use the cheap ones wherever the check can be expressed that way. That is correct advice and most evaluation tooling does not give it.
The command line and configuration file approach is the right architecture for this problem. It means evaluation lives in your repository, runs in continuous integration, and blocks a merge when quality regresses. That is how it should work and how it almost never does, because the alternative tools want you to click things in a dashboard. The docs cover the continuous integration setup properly.
The second half is red teaming and it is the reason I rate this above the other evaluation tools. Promptfoo will generate adversarial inputs against your specific application and report what got through. Prompt injection, jailbreaks, attempts to extract the system prompt, attempts to make it produce harmful content, attempts to get it to reveal data from other users, tests for excessive agency where a tool calling agent can be talked into actions it should refuse. The plugin structure means you pick the risk categories relevant to what you built rather than running a generic scan.
This material is the most accessible treatment of language model security I know of. The threat descriptions explain the mechanism of each attack, not just its name, so you finish with an actual model of how these systems get broken. Anyone who has deployed a customer facing assistant should run this before their users do. It is genuinely uncomfortable to watch, which is the point.
Now the complaints. YAML stops being pleasant at scale. A configuration with several providers, many test cases and varied assertions becomes a long file that is hard to review. There is support for splitting it and for generating tests programmatically, and the default path leads you into a large file first.
The balance of the documentation has tilted heavily towards red teaming and enterprise security. Someone arriving to learn basic evaluation now walks past a lot of material about compliance frameworks and vulnerability databases to reach it. Related, the line between open source and paid enterprise is not always obvious while reading. You can find yourself reading about a feature and then discovering it belongs to the commercial product.
Model graded assertions cost real money and the docs mention it in passing. A red team run with many plugins against a frontier model produces a bill that surprises people. Worth understanding your provider costs before running the full suite. It assumes command line fluency.
That is reasonable for the developer audience and it excludes the product and quality people who would benefit most from seeing these results, and who currently have no way to run this themselves. My four point zero is for evaluation tooling that fits how software is actually built, assertion guidance that steers you towards cheap checks, and red teaming documentation that makes security testing something an ordinary team can do, marked down for configuration that scales awkwardly, a shifted centre of gravity that buries the basics, and blurred boundaries around the commercial tier. Run the red team quickstart. Today.