doc-verdict
Task-based comparisons of original and rewritten documentation.
A documentation rewrite can read better without helping an agent complete a task. Assessing the prose alone does not show what changed in the output.
I built doc-verdict to compare task results under the original documentation and a rewritten specification.
Change the document and hold the setup fixed
Each benchmark supplies a task, both document versions, and a checklist for the expected output. Both conditions use the same model settings and trial count.
This setup reduces differences caused by the test configuration. Repeated trials help assess whether a score difference persists across generated outputs.
Code scores the outputs
Weighted checks evaluate output shape, exact values, headers, and probes. The scoring scale totals 200 points. A language model does not assign the grade.
The report includes the average score, the proportion of perfect outputs, and named failure categories. Configured thresholds determine whether the process exits successfully.
The documented benchmarks cover a Kubernetes deployment task and a Notion page-creation task. Each benchmark defines its own checks.
Dry runs test the evaluation code
Dry-run mode scores bundled outputs without calling a model. That makes it possible to exercise the evaluation and CI decision without API credentials.
Stored results can be rendered again without new model calls. Each run records the commit, worktree state, input hashes, and an implementation hash.
Those records help connect a result to its inputs and code. A dry run does not measure how a live model responds to a document.
The checklist defines the result
The checks encode my judgment about a correct output. A document can score well on those checks and still omit information or be difficult to read.
Two tasks provide limited coverage. A result from either benchmark should not be treated as a general documentation-quality score.