An agent plugin can be valid, installed, enabled, and still absent from the work that matters.
I have encountered an agent hook that passed when called directly but never reached a normal session. Two plugin caches contained the same number of skills but different content. A project summary existed, yet the receiving agent recovered an old branch or the wrong next action. A connector was configured but no longer authenticated.
Each check answered a narrower question than the readiness claim. I use five separate checks, or evidence planes, in synthesis engineering: source, installed state, live behavior, continuity and authenticated capability. The model is useful beyond Synthesis because it turns a vague compatibility claim into observations another person or runtime can reproduce.
Conformance means meeting a declared support contract. That contract should name the work, client surfaces and outcomes being promised; “ready” without those details gives an engineer little to verify.
1. Is the source internally valid?
The source check starts with the release the team intends to install. Does its manifest parse? Do the declared dependencies exist? Are client-specific adapters generated from one canonical source? Do tests cover failure paths? Does the version agree across the native packages, and does the skill catalog meet its metadata contract?
A test can establish that a script emits the expected JSON for a given input. That result says nothing about whether a client discovers or trusts the script, calls it at the intended event, or delivers its output to the model.
A lifecycle hook is code a client invokes at a named event, such as session start or a tool call. Testing the code directly exercises only part of that arrangement. The source result belongs in the report with the tested input and expected output; the client-dependent claims need their own observations.
2. What is actually installed?
Compare the intended release with the files and registries the client currently uses. A directory's presence or a newer version elsewhere in the cache does not identify the active consumer.
For a plugin, I check:
- the active version and installation root;
- the marketplace or source that supplied it;
- actual content, including generated instructions, against the intended release;
- whether the plugin and relevant behavior are enabled;
- local changes that have diverged from source; and
- obsolete duplicate copies that could still win discovery.
Two installations can have identical file counts while one contains an older instruction or executable. Content comparison catches that difference. A useful report names both the expected source and the active installed copy, so a reviewer can repeat the comparison.
Versioned caches also create a session-lifetime question. An active session may retain a path or loaded behavior from its startup. Updating the cache can replace or remove what that session expects to use. For an update that affects those retained paths or behavior, my operating policy is to checkpoint the work, close the affected sessions, update through the native client and start fresh sessions. I check the product's documented reload behavior before deciding which sessions need that boundary.
3. Did a normal session receive and apply the behavior?
A hook can be defined but untrusted, trusted but disabled, or enabled on a tool route that the current surface does not invoke. Those conditions call for different repairs. Record the client and surface, event, installed version and root, genuine session and tool-call identifiers where available, and time. Bind that record to the client's own session evidence. A file with plausible field names can otherwise be written by any process that knows the format.
For context delivery, establish what the client delivered as well as what the script emitted. For a protective rule, inspect what happened to the attempted action. Invocation alone cannot establish either outcome.
Prepared September 28, 2026: the following documentation check and proposed test design supplement the August account.
The Codex hook reference, checked September 28, ties non-managed hook trust to the definition's hash. PreToolUse supports a specific deny decision for covered calls. Returning an unsupported field such as continue: false at that event produces a hook error while the tool call continues. A PostToolUse hook runs after the tool and cannot undo its side effects. Hosted tools and some specialized paths also fall outside the ordinary tool-hook coverage.
Claude Code's hook reference, checked the same day, likewise makes behavior event-dependent. Exit code 2 can block a PreToolUse call; at PostToolUse, the tool has already run. A timed-out command hook at PreToolUse does not itself block the call. A checker that reports an error therefore needs a tested failure policy if it is supposed to prevent an action.
Consider a constructed policy that permits writing draft.txt and prohibits writing protected.txt. I would test it in an authorized disposable environment with the same client tool route for both files:
| Control | State before the attempt | Expected result under the stated policy | Observation needed |
|---|---|---|---|
| Permitted write | draft.txt contains draft-0 | The write succeeds | Native tool outcome and final contents draft-1 |
| Disallowed write | protected.txt contains protected-0 | The write is rejected before it occurs | Native rejection and unchanged contents protected-0 |
| Checker failure | Restore protected-0, then induce the specified checker failure | A fail-closed contract requires prevention | The actual error, tool outcome and final file state |
The permitted control helps reveal a tool route that never executes anything. For the rejected case, an unchanged file alone leaves other explanations open: the attempt might not have reached the tool, or a separate permission layer might have stopped it. Preserve enough evidence to distinguish those cases. A warning after the file changes demonstrates observation of the write, not prevention.
This table is a test design. It reports no native execution. If the client cannot safely exercise one of the required cases, retain that gap. The report should state the policy's tested boundary and keep any broader enforcement claim unresolved.
4. Can another session continue the work?
A receiving session needs more than a transcript of what the departing session said. It needs the current phase and status, stable reference facts, controlling plan, recent session history, decisions and exact next actions. For repository work, that includes repository, branch and upstream identity. When writers share resources, it also includes active ownership. Check these records against Git and the services they describe.
The handoff test I want starts a fresh session in another client without access to the first conversation. Ask it to reconstruct the project from durable files and identify the next action. It should recover the same controlling decisions and notice conflicts with current state. If the plan says to review a prepared patch while the summary says to start implementing, the receiving session needs to resolve the discrepancy before acting.
Concurrency changes the ownership requirement. Writers sharing mutable files need an agreed claim-before-write mechanism; separate Git worktrees can isolate their edits. Across machines, a lease or other ownership service must make an atomic decision when exclusive access is required. Heartbeats help identify stale ownership. File synchronization by itself does not provide that exclusion, and worktrees do not serialize writes to a shared database or service.
A read-only review or a single writer does not inherently need a cross-machine lease. Match the coordination mechanism to the resource that could be changed and the writers that could change it. My tool-handoff procedure develops the departing record and receiving checks in more detail.
5. Can the client perform the required action now?
A configured connector can be signed out. A supported browser tool can lack operating-system permission. An installed repository client can lack access to the target organization. A capability available in one product surface may be absent in another.
Probe the required capability through an authoritative, read-only operation where possible. Record the target, surface, authorization scope and time. A successful identity lookup may establish authentication without establishing access to the repository the task needs; the probe must answer the actual support question.
I keep three outcomes alongside pass and fail:
| Outcome | Meaning in the readiness report |
|---|---|
UNKNOWN | The check did not establish the answer: a probe was unperformed, a network request failed, or the authoritative response could not be read. |
UNSUPPORTED | The product surface intentionally does not provide the capability under test. |
WARN | An observed issue is outside the required support contract but should remain visible. |
A required unknown holds the readiness decision. An unsupported required feature also prevents that support claim; it may call for a different client or an explicitly narrower contract. Relabeling either as a warning would conceal the missing requirement. An optional unknown can remain visible without blocking work that does not depend on it.
A report with four unresolved requirements
Exercise prepared September 27, 2026.
The downloadable readiness-report exercise contains a constructed project-check plugin, a stale installed copy, conflicting recovery records and a Python evaluator. Its source format is deliberately a toy example, not an installable client plugin. Every release, time and observation in this exercise is synthetic.
All five checks are required by the example's contract:
| Plane | Supplied evidence | Result | What would change the result |
|---|---|---|---|
| Source | The version 2 toy manifest names an existing check file | PASS | A broken manifest or missing dependency would fail this check |
| Installed | The installed directory still contains version 1 and different check text | FAIL | Compare the active copy with the intended source after updating it |
| Live | No normal-session receipt, context-delivery evidence or enforcement result | UNKNOWN | Obtain genuine observations for the named client, event and required effect |
| Continuity | The project record says to review the prepared patch; the constructed receiving record says to start a new implementation | FAIL | Resolve the conflict and repeat the receiving-session check |
| Capability | No authenticated probe for the required connector | UNKNOWN | Perform the authorized probe against the required target |
The decision is HOLD. Replacing the two installed fixture files with their source counterparts changes only the installed result. The two unknowns and the recovery conflict remain. A passing source row cannot supply missing live evidence.
After extracting the archive, run python3 -B evaluate.py and python3 -B test_example.py from its directory. The evaluator prints the table's result as JSON. Its process exits successfully when it computes a valid report, including a report that says HOLD. The fourteen tests cover the supplied report, the installation-only repair, a fully passing synthetic control, missing and mismatched evidence, stale observations and required unknown or unsupported outcomes. The archive also contains the separate native-control worksheet described above.
The executed example tests report semantics and fixture comparisons. It cannot authenticate a receipt, establish what a model received, or prove a client's safety behavior. Even its READY_IN_FIXTURE control means only that the supplied constructed records satisfy the exercise's rules. A real report needs independently inspectable observations behind those records.
For the constructed report above, I would hold the stated release, repair the installed copy and recovery conflict, then collect the missing live and capability observations.
Releasing against a declared contract
Claude Code and Codex should not be forced into identical file layouts or hook implementations. Their native systems are part of their value.
They can meet a common outcome through different native mechanisms. One may use a different registry, installation layout or hook envelope. Preserve those differences while testing whether the method is discoverable, a live session receives current context, the relevant safety decision is enforced within its stated boundary, and project state survives the handoff. Each observation should name the surface that produced it. A known unsupported feature is acceptable only if the required outcome remains achievable through a tested path or the contract explicitly excludes it.
The least capable common subset is a poor target. Native adapters with shared outcome tests are a better one.
I would use these eight steps for a release:
- Validate canonical source and dependency structure, including failure-path tests.
- Install from the reviewed, merged source.
- Compare active client registries and installed content with that source.
- Review executable hooks through the client's trust mechanism, including hashes where that mechanism uses them.
- Start fresh sessions in every required client; record genuine invocation, delivery and action-effect evidence for the promised behavior.
- Recover a project in both directions from durable files and check the receiving state.
- Probe required connectors and permissions through authoritative read-only interfaces.
- Produce a structured report that preserves every plane, exact outcome and unresolved requirement.
A receipt from an old plugin does not establish behavior for a new release. A newly installed plugin does not retroactively validate an old session. Keep the source revision, installed content, session and observation time connected so that a change makes the affected evidence visibly stale.
A green result should mean the required evidence chain is complete. That is the standard agent infrastructure needs if people are going to trust work that moves across sessions, computers, models and competing clients.