This, however, changes the entire optimization problem when writing documentation. Good documentation now becomes more objective - you are solving the very concrete problem: can a dumb harness running the dumbest model implement this reliably?
Humans can typically compensate for inconsistent terminology or scattered context across pages, but for agents, this often will waste time (or even just completely confuse the agent).
We’ve been building a small project around this called dari-docs: users can upload their documentation via website or CLI and run agents across different providers to see where they falter. You can upload your documentation, feed a list of tasks, and ask agents with varying intelligence / cost levels to complete those tasks in parallel. When a run is complete, you get back a list feedback markdown files from each agent run and can apply changes based on agent feedback.
Managed service: https://optimize.dari.dev/, repo link: https://github.com/mupt-ai/dari-docs
The agents actually try to use the product end-to-end. They search through the docs, follow instructions, run commands, try examples, and attempt to debug failures. Importantly, this is not a static LLM review of the documentation. The agents are actually attempting the integration.
You can also enable live verification with test credentials so the agents can actually verify workflows against real APIs:
dari-docs check . --live-verify --secret-env DARI_TEST_API_KEY --task "Create a checkout session"
Discussion (7 Comments)Read Original on HackerNews
What exactly is the advantage of doing this vs just running a prompt in my existing coding agent?
I don't understand why this is a harness/project vs just for example, a skill?
I'm confident there's a good reason, I just don't understand.
We actually aren’t rebuilding a harness here, it’s Pi with several LLM options to select from. The reason this is a project is that the useful workflow is more like a docs test suite: run realistic user tasks across multiple models, isolate each run in a greenfield sandbox, keep the transcripts/results, and make failures reproducible in CI.
You could ask an existing coding agent to spawn subagents for every task/model pair, but once that matrix grows, running hundreds of subagents on your computer gets messy. It’s also the wrong isolation boundary: for docs testing, you usually want the agent to start from a clean environment with access only to the docs/product surface you’re testing, not your whole working tree or local setup.