Turning a Failed Code Review into Tests and Repairs

On September 17, 2026, an AI reviewer rejected a five-file change to my session-recording tools. The change added support for another AI client. It could also assign a pending session to the wrong client and accept a transcript through a symbolic link that a path check had missed.

For the first, the test needed to preserve the declared client's pending state even when another client had a record with the same identifier. For the second, it needed to distinguish a real directory path from one that passed through a link. A general instruction to “improve validation” would have left both decisions unresolved.

The tool creates a session receipt: a record connecting an AI client's session to the transcript that supports it. That connection supplies provenance, meaning evidence of where the record came from. The existing implementation recognized Claude Code and Codex; the change added Muse. The implementation already required Muse's identity to come from a structured stream.id field. A session identifier mentioned in ordinary message text did not qualify.

Keep a missing transcript with its declared client

The first error involved an absolute transcript path that did not yet exist. That was a pending Claude case: the client had supplied a location, but its transcript was not available yet.

The new branch checked whether the path was both absolute and an existing file. If either condition failed, it tried the Muse store. A Muse record with the same session identifier could therefore displace the pending Claude case. Finding a matching identifier elsewhere was insufficient reason to change the client.

The repaired branch handled an absolute path first. When that path was missing, it returned without assigning provenance and left the pending Claude handling in control. Muse lookup remained available when no absolute transcript path had been supplied.

This is the kind of review finding I want expressed as an expected result before anyone changes the code:

An explicit, missing transcript path stays pending for its declared client. A same-identifier record belonging to another client cannot satisfy it.

Test it by putting a plausible record in the other client's store and checking that the declared client still owns the pending case.

Check the directories leading to the file

The second finding concerned a symbolic link, a filesystem entry that points to another location. The path appeared to fit the expected layout:

session-store/
  year/month/day/
    session-id/
      session.jsonl

Checking the final session.jsonl for a link missed a linked session-id directory above it. The final file could be ordinary while the path leading to it redirected elsewhere. Python's Path.is_symlink() documentation describes a check on the named path; it does not establish that every ancestor is a real directory.

The review found that some callers already used a validator that checked the path components, while the session recorder bypassed it. The repair brought that caller through the same validation and strengthened the store lookup. It rejected linked roots, date directories, session directories and files. Date directory names had to contain ASCII digits, and the lookup had to find exactly one matching transcript.

Requiring one match also gave ambiguity a defined result. Two real date directories containing the same session identifier meant the resolver could not choose one as authoritative. Taking the first filesystem result would have hidden the conflict.

The search still had a fixed number of directory levels without a numerical limit on how many directories it could inspect. ASCII filtering removed unrelated names; it did not cap directory work. Nor did these checks make a changing filesystem race-free. A hostile process could alter a path between validation and reading it. Those are separate design problems from the missing component check.

Preserve the successful behavior while adding rejection tests

A test that expects rejection can pass when an implementation rejects everything. The initial review identified that weakness in some negative tests. It also recognized that an existing successful test exercised the new behavior and checked the recorded client, provenance, binding and transcript path.

I want both kinds of tests. Invalid evidence must fail, and valid evidence must still produce the right result. Otherwise a repair can satisfy a collection of rejection assertions by disabling the feature.

Four tests were added in the historical repair:

InputAssertion added at the time
Two real date directories hold transcripts for the same session identifierNo receipt is created.
A valid transcript sits beside a latest/ decoy and a linked session directoryA receipt is created and identifies Muse as the client.
The identifier appears only in message text, with stream containing a stringNo receipt is created.
An absolute Claude transcript path is missing, while Muse holds the same identifierThe provenance lookup returns no assignment, preserving the pending case.

The mixed valid-and-decoy test checked successful receipt creation and the client name. It did not independently assert the literal full transcript path. The older successful test had stronger path assertions. That difference is useful when deciding what to add next: keep the existing successful control and make the new mixed-input case precise about which file it selected.

The focused test output displayed 45 passed in 0.92s. A second review returned PASS for the two blocking findings, and the changes were committed. The saved test command piped its output through tail, so the evidence is the displayed pytest result, not a separately captured pytest process exit code.

The rereview also named missing dedicated tests for Unicode-digit directory names and links at the year, month, day and file levels. Its broad statement about unchanged Claude and Codex behavior needed narrowing: adding stream to the JSON fields examined by the parser could reject duplicate keys that had previously been ignored. Preserving the pending-path behavior did not establish identical behavior for every input.

Work through a fictional locator

The downloadable receipt-locator exercise turns those distinctions into cases you can inspect and run. Alpha and beta are fictional clients, and every transcript is generated beneath a new local fixture directory. The exercise does not connect to an AI client or read a real session store.

Start with fixture-plan.json, expected.json and expected-observation.json. They contain the inputs and literal expected results, written before the implementation. The checker compares the complete result, including the client, session identifier and selected relative path. It also checks which transcript files the locator read.

For example, the pending-path case puts the same fictional identifier in beta's store while alpha's explicit transcript is missing. Its expected result is:

{
  "status": "PENDING",
  "client": "alpha",
  "session_id": "00000000-0000-4000-8000-000000000001",
  "relative_path": null
}

The expected read list is empty. Selecting beta's transcript is wrong even if that transcript contains a matching identifier.

The positive mixed-input case includes one valid beta transcript, a latest directory, a Unicode-digit directory and a linked decoy. Its expected successful result names the selected file exactly:

{
  "status": "BOUND",
  "client": "beta",
  "session_id": "00000000-0000-4000-8000-000000000001",
  "relative_path": "beta/sessions/2030/01/02/00000000-0000-4000-8000-000000000001/session.jsonl"
}

There are fifteen cases, including six variants that place a link at different points in the path. The linked targets remain inside the isolated fixture, and their bytes are checked afterward. The observer records calls to the exercise's file reader, allowing the expected result to require that none of those target bytes was read. It does not monitor arbitrary operating-system activity.

After extracting the ZIP, run these commands from its receipt-locator directory:

python3 -B demo.py --workspace ./demo-fixture
python3 -B check.py --workspace ./demo-fixture

The first command requires a fresh directory and leaves the generated files in place for inspection. The second compares the observations with the fixed expectations. The README explains the individual cases and how to supply an alternative locator without changing the checker.

Source and clean-ZIP runs on Python 3.12.3 passed those checks. The same expectations rejected seven deliberately faulty variants: rejecting everything, switching clients for the missing path, following links, taking the first match, using message text as identity, accepting Unicode digits and overwriting duplicate JSON keys. A harmless comment and an equivalent ASCII-digit check both continued to pass.

The exercise assumes a stable directory owned by the caller. Unsupported symbolic-link creation is reported as unavailable, not counted as a pass. It tests these constructed files and decisions; it does not establish native-client compatibility or fill the gaps in the historical test run.

Record the finding and the rereview

For the pending-path finding, the review record can now be concrete: a missing absolute path belongs to its declared client; the test adds a same-identifier record elsewhere; the corrected branch leaves the case pending; and the faulty fallback fails the unchanged expectation. The successful lookup cases show that preserving pending state did not require disabling valid receipt creation.

For another finding, I would use the same sequence: specify the good result, construct the conflicting input, repair the deciding branch, and ask the rereviewer to check both. Keep the remaining uncertainty beside that finding. A source commit, a passing test result and a rereview each answer useful questions. Installation and behavior in a running client require their own checks.