The checks I ran after an AI agent deleted 474 bookmark files all passed. They missed 196 URLs.
On July 17, I'd sent an agent to clean up a folder of personal notes. It consolidated bookmarks into a CSV and deleted 474 files from the working tree. Git's version history still held the deleted files.
The agent's report was thorough. It explained what the files had contained, why their content belonged in the new format, and which handful it had deliberately kept. Reading it, I believed it. I also pulled the actual commit, counted the deletions myself, opened four of the kept files to confirm they were still there, and checked about eight rows of the new CSV against their source notes.
The deletion count matched. The four retained files existed. The sampled rows matched the material I'd compared them with. The CSV contained 723 rows.
Later that night, a separately written extraction found URLs the first conversion had missed. The extractor had taken only the first URL from each note. Ninety source files contained additional URLs; recovering them added 196 rows, taking the CSV from 723 to 919 without removing any of the original rows. The extractor was then corrected.
Git history made that recovery possible. The initial checks hadn't demonstrated that all useful information survived the cleanup, and recovering the missing URLs didn't establish that every useful idea in the notes had been preserved either. My earlier conclusion that the 474 deletions were simply correct went beyond what those checks showed.
A row can be right while another row is missing
The rows I'd checked came from single-link notes. A first-URL-only extractor handles those correctly. Reading those rows more carefully would not have exposed its treatment of a note containing two links.
There's also a problem with choosing the sample from the output. An omitted URL has no row to select. A CSV can contain accurate rows and still be incomplete.
The check needs to begin with the source inventory: the original notes and the destinations intended for their contents. For each note, identify every URL that should have a corresponding row, whether the note contains prose worth retaining, and where that prose will live. Record deliberate omissions as decisions to review, rather than letting the extractor make them implicitly.
Sampling remains useful for judging whether a destination preserves meaning. A completeness claim needs a comparison covering the entire agreed inventory. In this case, the missing comparison was between all the source URLs and their CSV relationships.
Try a conservation check without deleting anything
The notes-to-CSV workbook uses five fictional notes. Each URL occupies its own line. The examples use example.test and make no network requests.
| Note | Contents | Required result |
|---|---|---|
a.md | One URL: /one | A row linking source a to that URL |
b.md | Two URLs: /two and /three | Both source-to-URL rows |
c.md | The same /one URL as a.md | A separate row preserving source c |
d.md | A workshop instruction, with no URL | The complete note retained |
e.md | URL /four and an entrance instruction | The URL row and the complete note retained |
The expected result has five source-to-URL relationships, four distinct URLs, and two retained notes. Counting unique URLs alone would miss the loss of source c: its URL would still be present through source a.
Note d says, “For the workshop, bring the printed route map.” Note e includes, “The covered entrance faces the river; use it when it rains.” The exercise treats both instructions as worth keeping. A URL-only export would lose information someone could act on.
The download includes the notes, a converter, a checker, tests and instructions. The expected relationships and complete retained-note contents are written literally in expected.json. The checker uses that fixed contract without calling the converter to decide what the answer should be.
After extracting the ZIP, run this from its directory with Python:
python3 -B run_workbook.py --workdir ./workbook-run
Use a new output directory. The workbook creates a CSV and copies the retained notes; it leaves every source note in place. It refuses an existing output directory and preserves the generated results for inspection.
One constructed wrong output drops b's second URL. The checker names the missing relationship, b → https://example.test/three. Another substitutes the wrong URL for e while keeping the row count, distinct-URL count and retained-note count unchanged. A third keeps those counts and e's URL but strips the entrance instruction. The full relationship and note-content checks reject both equal-count results.
In local runs, the source and a clean extraction of the download each passed seventeen tests and rejected six constructed wrong outputs. Each copy was also tested with four deliberately faulty converters: one took only the first URL, one deduplicated URLs across notes, one omitted prose-only notes, and one stripped prose from a mixed note. The expectations and tests stayed unchanged; each converter failed for the intended loss.
These results belong to the fictional workbook. It accepts whole-line URLs, not arbitrary Markdown links, and checks an agreed retention decision rather than deciding which prose matters. It demonstrates a way to test a preservation claim; it does not authorize deleting the source files.
Eight agents finished, but the shared record didn't
The same night, I'd dispatched eight agents in parallel, each converting a different repository into the same file format. Each finished its assigned work and reported success.
None had reconciled the shared tracking document with all eight results. Each had been careful to leave the other agents' work alone. That local caution left the project's own account of the work stale.
Reading the reports together exposed the missing responsibility. The overall job included reconciling the shared record after the individual conversions. Eight successful reports didn't establish that this last step had happened.
A process can share an assumption with the report it produces, so a detailed explanation can repeat the same omission. Checking the source inventory, or checking the combined result of several assignments, asks something the individual success report may never have tested.
The paused project's fourteen repositories
I'd also resumed a project that had been sitting untouched for about a month. Its notes said fourteen repositories still needed a conversion pass.
A roughly two-minute check found that count was wrong in both directions. Some of the listed repositories had already emptied out through unrelated work. Other repositories had been created after the project went quiet and had never appeared in its notes.
The old note had been accurate when it was written. Work elsewhere had changed the material it described. Before assigning the next round of conversions, I needed a fresh inventory.
The multi-agent tracking gap appeared in one night; this one accumulated while a project was paused. In both cases, the project record needed to be reconciled with the work it described.
Check the consequence of believing the summary
A task that only reads can still produce an expensive mistake if its report determines what happens next. A stale repository count could send agents toward completed work and leave new work unassigned. A convincing cleanup report could lead someone to accept a conversion before checking what it omitted.
The depth of verification should reflect the consequence of being wrong and the cost of a check. For a consequential conversion, inspect the source-to-destination relationships and the retained material. For a scope estimate, recount the current sources. For a claim about a commit, inspect the change itself.
The question I keep coming back to is: If this summary is subtly wrong, how would I actually find out, and have I done that specific thing?
Sometimes the answer is thirty seconds with git log. Sometimes it's re-deriving a count instead of reading one from an old document. Here, it required looking for information that never reached the output. The initial checks earned confidence in what they examined. Accepting the cleanup required a check of what they could not see.