Two vendors, one standard: running a real upgrade with synthesis engineering

Earlier this week I planned a significant upgrade to synthesis-skills, my open-source collection of skills for AI coding agents, and used the work as a live test of synthesis engineering itself. I did the designing and deciding, then assigned models from two vendors to deliberately different roles: one executed my plan; the other attacked the result.

The setup: separation of powers, by design

The upgrade added a review process that records every finding and how it was resolved, staleness checks that flag project notes nobody has kept current, and release checks that verify a release contains exactly the files it claims to. Each release's acceptance manifest is committed beside the code.

I wrote the plan with Claude Fable 5 because that was where I wanted design depth. The plan fixed the release order and four constraints: reproduce a defect in a failing test before fixing it; record any omission where the next reader will see it; never let "the checker passed" stand in for "the change is correct"; and drive every review finding to a terminal state.

Execution I handed to OpenAI's Codex agent running GPT-5.6 Sol at maximum reasoning effort, unattended against that written plan. It worked release by release from mid-afternoon until past midnight, then picked the thread back up the next morning. It read the project files to reconstruct where things stood, opened pull requests, ran the checks I'd specified and wrote progress notes to a shared record file that I and both models could read. I supervised through short reports and kept the reserved decisions. One I approved from a train, in two words: "I approve." The tooling paused at that gate until those words were captured verbatim, because an approval someone paraphrased is not an approval I gave.

The executor couldn't see the designer's conversation, so my written plan had to contain the plan. Different vendors also brought different questions to the work. That was a deliberate engineering choice: use the difference in approach to examine assumptions the executor might otherwise leave untested. An agent can still ask for clarification; the written plan should make it possible to proceed without guessing my intent.

When execution finished, I turned Claude Fable 5 around into the reviewer role and used the review protocol the upgrade had just shipped to review the upgrade itself. I enjoy that kind of recursion. The upgrade held up, and the review surfaced two serious gaps.

Follow the invocation

A control that shipped but never switched on. A new commit-time gate had a complete test suite, all green, but its activation setting was missing on the machine it was meant to protect. Commits on that machine had passed while the control announced its own absence. The tests proved that the gate worked when configured; they had not checked the configuration used by the real workflow. I had written before that a green config is not a working agent. This was that lesson catching my own tooling.

A unit test can call a guard directly, provide the configuration it expects and confirm that it rejects an invalid input. That establishes behavior under the test's conditions. The operational workflow also needs to find the right version, invoke it at the right event and supply the intended configuration.

For a commit gate, I would ask for evidence at each step:

StepEvidence to inspect
InstallationThe gate's actual location and version in the environment being protected.
InvocationA trace showing the normal commit workflow called that gate.
ConfigurationThe settings the running gate read, including its enabled state and applicable rule.
EnforcementThe attempted action and resulting repository state for both an allowed and a rejected case.

These checks distinguish repairs that otherwise look alike. Installing a missing file won't fix a workflow that never calls it. Wiring the invocation won't fix a rule disabled by configuration. A rejection message won't protect the repository if the commit still goes through.

The newly activated gate's first production action was to refuse a commit from the session that had switched it on. I enjoyed that result: the control enforced its rule even against its author. An allowed commit followed once the missing attribution was supplied.

Test the boundary with an allowed and a rejected case

Consider an illustrative gate whose rule requires a valid review record before a commit. Use two disposable copies of the same repository state and candidate change, with identical workflow configuration. In the first, supply a fixture review record that satisfies the documented rule. Attempt the commit through the same entry point the protected workflow uses. Preserve the invocation trace, the decision and the resulting commit identifier.

In the second copy, leave the required review record absent and attempt the same candidate commit. Confirm that the gate was invoked, identify the rule responsible for rejection and inspect repository state to confirm that no commit was created. An unrelated failure, such as an invalid author configuration, would not establish that this gate enforced its rule.

Both cases matter. A guard that rejects every action can pass the rejection example while preventing legitimate work. A guard that prints a warning and returns success can appear in the trace while allowing the forbidden action. Checking the final state makes the expected effect explicit.

Bind the results to the code version, configuration, environment and entry point used. A successful test through a terminal command does not establish coverage for an application that takes another route. Test each route the control is meant to cover, or record the untested one as an unresolved acceptance condition.

For a control required to protect a workflow, specify what happens when its configuration is missing or the check cannot run. Test that failure path too. Otherwise a later environment change can turn a required check into an optional one without changing its implementation.

Read the record from the next session's position

A record that was safely stored and quietly invisible. The upgrade's project history, including decisions, findings and evidence, had been committed and pushed to a side branch but never merged into the main branch that other sessions read. The backup was complete. The working record was missing.

The branch problem calls for a different check. Git's branching model lets a pushed branch preserve a decision while leaving it absent from the branch checked out by the next contributor.

Start with the consumer: which repository, branch or revision will the next session open? Locate the decision there, check its version and follow the documented entry point to it. If the record is available only on another branch, integrate it through the project's normal review process or give the consumer an explicit, durable reference it can use.

Then start a fresh session with the intended access. Ask it to identify the decision and its remaining conditions, and inspect the source it used. This tests transfer, discovery and interpretation together. Supplying the missing decision in the test prompt would skip the discovery step.

Both gaps were repaired and re-verified the same day: the commit gate was activated and exercised, and the project history was merged into the branch later sessions would read.

I call this boundary the Built/Deployed Gap. Once you have the name, you see it in familiar forms: an alert rule defined but not enabled, a backup never restore-tested, a feature flag whose call site moved. The checks in this upgrade stopped at build and branch state. Deployment probes, restore tests, live controls and independently derived review extend the surface.

Delegate implementation and name the reserved decisions

A useful assignment identifies the choices an assistant can make within the plan and the decisions reserved for a human. Selecting a test fixture may be delegated; changing a release's approved scope or taking an action outside the granted permissions may require a decision. Specify that boundary and retain the actual approval with the work it covers.

Documentation checked September 29, 2026: The goal guidance in the next paragraph describes later product documentation.

OpenAI guidance on long-running work describes goals with outcomes, constraints and verification criteria, plus follow-up messages that can steer the work. Starting a goal retains the existing access and approval boundaries. The ability to continue working is compatible with human review at defined points.

Jeremy Lewi's August 25, 2026 account of using Codex at OpenAI describes reviewing a plan, monitoring progress, intervening when needed and retaining commands, results and decisions in a notebook. That is a useful comparison for delegated work with human decisions at defined points.

What the pattern gives you

Across August 26–28, the work produced eight minor releases, v4.50.0 through v4.57.0, plus two same-day patch fixes. Some of those releases followed the review described here. At the reported findings checkpoint, fifty-three of fifty-four review findings had reached verified repair; the last remained recorded as an open design decision. It had not disappeared into a chat thread.

Write the plan so the executor cannot depend on your unwritten intent. An agent without access to the author's chat tests whether the written plan can stand on its own. Documentation gaps surface before execution makes them expensive.

Rotate in a reviewer with different context, and give it artifacts before explanations. Starting from the same chat and assumptions risks reproducing the author's blind spots. A reviewer that derives its own inventory adds questions the executor did not ask.

Keep the record in files, not in chat. Chat history is a scratchpad tied to one tool's context. Repository files let a second model, a colleague or future you pick the work up, verify it and challenge it. The record is the project.

I designed this exercise, set its constraints, made the judgment calls reserved for me and own its results. What the models multiplied was how much of my own standard I could enforce at once. The full practice, from the Direction Dynamic to the review protocol this upgrade shipped, is documented at synthesisengineering.org. The synthesis-skills repository makes the methodology and content available under CC0 and its executable scripts under Apache 2.0.

Pick one control that passed in CI and verify, from the running system, that it enforces what you think it does. Keep the installed version, invoking event, configuration, allowed and rejected outcomes, and next session's record together. That is the evidence a reviewer can use to challenge the result.