My release on September 18, 2026, updated a shared skill package for Claude Code and Codex while Muse remained on an older version. Both the refresh loop and the verification loop named only the first two applications. The script could complete every check it knew about and still miss a client I intended to support.
I was investigating this from Muse, the newly added client missing from the release process.
Here, a client is an AI application consuming the shared package of skills and instructions. A staged bundle is the prepared set of files for distribution. An installed cache is the package directory recorded by a client. A running session may need a separate operation to show that it can use the expected material.
Start with the clients you intend to support
The release log for version 4.103.0 showed successful updates for Claude Code and Codex, followed by reported-version, on-disk-manifest and content checks. It compared 500 selected source files for each. The later Muse inspection still showed version 4.102.1, including that version in its cached manifest.
The two-client checks had done useful work. They had also made the release's coverage depend on the same incomplete list used to distribute it. Checking that every attempted update succeeded would never reveal the missing attempt.
I want the required-client inventory to be an input to release acceptance. In this case, it contained Claude Code, Codex and Muse. For each client, the check needed to retain the update result, the reported version, the installed location and the file comparison. An absent client's row had to remain visible.
Muse's earlier onboarding had already shown why these observations needed to stay separate. Selected calls to read the thinking-framework and autopilot skills failed as unknown skills even though the catalog listed 63. Inspection found an enabled but inactive, invalid package with a digest mismatch: its content fingerprint disagreed with the recorded value. An update reported a valid, active package, but the selected reads still failed in that session. After I restarted the session, both named skills were read successfully. Those were two successful reads, not a test of every catalog entry.
The release omission explained why the automated process left Muse behind. The write that caused the earlier digest mismatch remained unidentified. Updating the consumer list could repair the release's coverage without answering that separate question.
Read the failed step before trusting the final status
Updating Muse's recorded source and then updating the package brought its reported and cached version to 4.103.0. Automating that path took another repair.
The first added Muse stage prepared the bundle correctly, then tried an installation that the client refused because the plugin was already installed from a different source. Its output included:
FAIL install.muse.install: plugin `synthesis-skills` is already installed from a different source
REFRESH: False
PASS verify.muse.content: 501 files byte-equal to source
VERIFY: True
failed steps: ['install.muse.install']
EXIT=0
The content check in that attempt selected the staged bundle. Fresh prepared files could pass even though the attempted installation had failed. The command pipeline's printed zero status did not erase the failed step or REFRESH: False.
This is why I keep the operation result alongside the evidence it was supposed to produce. A final status can summarize a run, but it cannot resolve a contradiction between a failed update and a check of files the client may never have received.
Compare the installed files, including same-version changes
The next attempt used Muse's synchronization and update path. The version records agreed on 4.103.0. Comparing the installed content then found two differing files among the 501 selected source files: the release implementation, release.py, and its tests, test_release.py.
The repair itself had changed those files without changing the version label. Reusing a staged bundle merely because it already carried 4.103.0 could therefore distribute an older copy of the repair.
The revised staging step checked for content drift under the same version and exported the bundle again when needed. The content verifier also put the client's reported cache first among its candidate directories. In the recorded repaired run, that cache's manifest matched 4.103.0, so the comparison used it. All 501 selected files matched, and the failed-step list was empty.
The implementation could still choose staging when the reported cache did not have the expected manifest version. For a check specifically claiming to verify installed content, I would require the reported installed directory and return incomplete evidence if it could not be checked. The exercise below follows that rule.
The final historical release, 4.104.0, included all three clients in its refresh and verification steps. Its log recorded matching reported versions, matching disk manifests and 506 selected source files for each client. The selection had grown since the earlier runs; 506 describes what that release compared.
The log did not include a new skill-read operation in every running client after that final release. The two successful Muse reads after my earlier restart remain useful evidence of that earlier session. A running-session result belongs to the release and session actually observed.
Work through a three-client release
The release-evidence workbook uses three fictional clients (cedar, maple and willow) and two small files per installation. It creates local records and files, then classifies them. It never installs, updates, restarts or contacts an application. Its running-session observations are explicitly constructed inputs.
Read contract.json, fixture-plan.json and expected.json before the implementation. They fix the required client list, version 2.0.0, the two expected file digests and the complete expected reports. A digest is a content fingerprint; the workbook uses Python's hashlib.sha256 to compare the named files. A matching fingerprint says nothing about whether a running process loaded them.
Case R06 supplies matching version and installed-file evidence for all three clients. Willow has no running observation. Its expected row is:
{
"reported_version": "MATCH",
"installed_bytes": "MATCH",
"running_observation": "UNKNOWN"
}
The complete report must say INCOMPLETE. Filling the last field with MATCH would invent an observation. Conversely, R09 supplies an explicit failed observation, which makes the result EVIDENCE_FAIL. Missing evidence and observed failure call for different next actions.
The nine cases hold cedar and maple's evidence constant while changing willow's:
| Case | Willow's condition | Expected workbook result |
|---|---|---|
| R01 | Version, installed files and supplied running observation all match | MATCHES_CONTRACT |
| R02 | No client record was supplied | INCOMPLETE |
| R03 | The reported version is older, despite the other supplied evidence | EVIDENCE_FAIL |
| R04 | The version matches; installed content differs | EVIDENCE_FAIL |
| R05 | Staging matches; installed content differs | EVIDENCE_FAIL |
| R06 | Installed evidence matches; the running observation is absent | INCOMPLETE |
| R07 | The client is explicitly disabled and invalid | EVIDENCE_FAIL |
| R08 | The refresh failed, while the aggregate pipeline status is zero | EVIDENCE_FAIL |
| R09 | The supplied running observation explicitly failed | EVIDENCE_FAIL |
R04 establishes the version-versus-content mismatch. R05 asks you to inspect where those bytes came from. In both cases, choosing the prepared staging directory would hide the stale installed file. The workbook reads the directory named by installed_directory; the separate staging field cannot satisfy that comparison.
After extracting the ZIP, run these commands from its release-evidence directory:
python3 -B make_fixtures.py --workspace ./release-fixture
python3 -B check.py --workspace ./release-fixture
python3 -B inspect_release.py --root ./release-fixture/R01
python3 -B inspect_release.py --root ./release-fixture/R06
The first command requires a fresh directory and leaves the files available for inspection. The checker compares the complete report produced by the command-line program against the fixed expectations. R01 exits zero. The individual R06 command intentionally exits 1 because its evidence is incomplete; the checker succeeds when it observes that expected result.
The checker also exercises eleven invalid inputs, including duplicate and unknown clients, malformed UTF-8 and JSON, wrong field types, an escaping path and a linked directory. Invalid inputs produce a structured INVALID_INPUT response with exit 2. The README explains the directory layout and how to test an alternative implementation without changing the expected reports.
Source and clean-ZIP checks passed on Python 3.12.3. The unchanged expectations rejected ten faulty implementations, including always accepting, always rejecting, deriving the client inventory from the supplied rows, trusting version labels, reading staging, inventing a missing running observation, ignoring a failed child command and corrupting only the printed report. A harmless comment and an equivalent iteration over the required clients continued to pass.
These are checks of constructed records in a stable, caller-owned directory. They do not protect against every concurrent filesystem change, and the supplied running observations do not stand in for native-client tests.
Keep a result for each consumer
For a real release, begin with the clients you mean to support and the specific source revision or artifact being distributed. Retain each update's result and the client-reported installed location. Compare the intended files at that location, and keep the selected file set with the result so its coverage is inspectable.
Then choose an operation that can distinguish the new material in the running client. Where a client provides a skill reader, that could be a read of a changed skill through the session itself, with the request and returned content retained. Record which release and session the observation belongs to. If it has not been run, leave that entry unknown; the next engineer can see exactly which check remains.