The docs-kit post-mortem — design docs did not save tokens
About a week after its July 30 release, we called docs-kit a failure. The effect we aimed for — letting an AI read design docs efficiently and saving tokens — never arrived; instead, 63% of all commits went into keeping the documents in sync. Here is what we got wrong, and where we could have noticed. We publish the record as primary material for anyone trying to solve the same problem.
About a week after publishing docs-kit on Labs, we concluded it was a failure. The effect we aimed for — letting an AI coding agent read design documents efficiently, and saving tokens — never arrived.
The world is full of success announcements; records of failure are rare. This article is a summary of the post-mortem we left in the docs-kit repository: what we aimed at, what happened, and where we went wrong. If it saves a week for someone trying to solve the same problem, it is worth publishing.
What we aimed for
Two problems. An agent implements without knowing the design intent, and it believes a stale design doc without questioning it.
docs-kit tried to solve both with Markdown + front matter as the single source of truth. Each document carries the code it is responsible for (scope) and the commit it was last reconciled against (verified_at); freshness is judged mechanically from git history. And a reverse lookup, which, takes the files you are about to touch and returns only the documents that matter.
The flow we imagined:
What actually happened
Two rounds of field reports from a real project — a greenfield macOS app — broke this mapping in both directions.
Round one. Seventeen documents were written before a single line of code existed. which starts by glob-matching scope against real files, so it matched nothing and returned nothing. The agent, following the convention, started implementing without reading any of the seventeen — the entire implementation plan among them. At one point the agent fed deliberately invalid values to lint and read the error messages to reverse-engineer the front matter schema. The schema was fully documented in the README; there was simply no path to it from inside a session.
Round two was worse. A reverse lookup for one file returned 19 ADRs — 141KB — while the status: current requirements document and the roadmap, 40KB that the convention explicitly says to trust as specification, could never be returned at all: they carry no scope. The result: features remained unimplemented after every roadmap item had been closed.
And with roughly 4,800 lines of code belonging to no document's scope, lint kept saying OK and stale kept reporting all green. From inside a session, a healthy board and a half-ownerless codebase looked exactly the same.
What the numbers were saying
In hindsight, the repository's own statistics told the story.
The tool itself is 4,009 lines of Go. The documentation explaining how to use it: roughly 150KB. The explanation outweighed the thing it explained. And two thirds of the commits were not code — they were reconciling documents against the implementation.
What went wrong
The documents never substituted for the code
There is exactly one condition under which documents save tokens: reading the document must spare you from reading the code. In implementation work it never holds. However good the spec, you still read the code for the signature of the function you are changing, its existing call sites, the local error-handling idiom. The documents were read in addition to the code, not instead of it — pure added cost.
The spec layer in particular — a transcription of current behaviour — was a degraded copy of information derivable from the code. The code is the truth; a copy always lags. We built freshness verification to detect the lag, and a reconcile routine to close it, and that routine consumed 63% of all commits.
A convention cannot beat an agent's natural behaviour
The convention — run which before working, read only what it returns — sat in every session's context. But it is a weaker pull than what an agent naturally wants to do: read the target file directly and grep as needed. Worse, in round two the failure happened because the agent followed the convention — reading the irrelevant 141KB and missing the essential 40KB. And there was no way to observe from outside whether the convention had been followed at all. The failures were silent.
"Return the relevant documents" was not solvable with globs
The core of which is a mapping from paths-about-to-change to documents-worth-reading, implemented as scope glob matching plus one hop of the dependency graph. But whether something is relevant is fundamentally a question of semantics, and a heuristic of this size cannot approximate it. Every attempt to improve precision grew the return payload — colliding head-on with the goal of saving tokens. The clearest warning sign: every fix only had exits in one direction. More documents, bigger returns.
Freshness checking converted a problem into an obligation
Detecting stale documents mechanically worked exactly as designed. But that did not solve the rot problem — it merely converted it into a standing duty to keep tending the documents. The tending fell to the AI, which spent the very tokens the system was meant to save. Same wallet.
And we never measured the goal
A design record written for the final fix contains this sentence:
Every check docs-kit has runs in one direction: it starts from a document and asks whether the code still agrees with it. Nothing ever starts from the code and asks whether any document claims it.
The same asymmetry ran through how we evaluated the project itself. We verified, every time, that the tool worked correctly — and never once measured whether using it actually reduced tokens. No baseline, ever. We mistook the correctness of the means for evidence of the end.
At the first field report — which returning nothing on a greenfield project — the right question was not "how do we fix it" but "is this mapping solvable at all". Instead we fixed the tool, shipped two more rounds of design, and arrived back at the same place.
What survived
Not everything was wasted.
- ADRs — decisions, and the reasons behind them. Why a design was chosen, and what was considered and rejected, cannot be derived from the code. Reading them is worth the tokens, and past decisions stay past decisions, so they never rot. They need no tool either: a directory of plain Markdown is enough. Recording rejected alternatives with their reasons demonstrably stopped the same debates from reigniting.
- A short CLAUDE.md carrying only the constraints the code cannot show. "Three direct dependencies at most", "lean on the standard library" — these worked.
- Separating what loads always from what loads at the moment of need. Moving document-writing guidance out of the every-session convention block and into templates was the right call, and carries over.
The spec layer — transcribing what the code already says — was the core of the failure, and must not be carried over.
Six questions for a successor
At the end of the post-mortem we left the questions to answer before designing the next mechanism. If you cannot answer them, do not build it.
- Which lines of code, concretely, does reading this document spare you from reading?
- Can the information be derived from the code? If so, do not write it down.
- When it rots, who repairs it — and does that cost come out of the same wallet as the cost you are trying to cut?
- What, other than a convention, guarantees the agent will read it?
- How will success be measured? Was a baseline taken?
- When the first field deployment surfaces a fundamental flaw, have you decided in advance to question the premise once before fixing the tool?
Feature work on docs-kit has stopped. The habit of writing ADRs continues, without the tool. It took about a week from release to knowing it had failed — and reaching that verdict as fast as we did may be the one thing this attempt can be proud of.
Judged a failure about a week after release, across two field reports from a real project. 4,009 lines of tool; roughly 150KB of documentation explaining how to use it. Two thirds of all commits were document synchronisation.
Do not write down what the code can already tell you. Shape an agent's behaviour with paths, not prose conventions. And if the goal is token reduction, decide how to measure it and take a baseline — first.