Testing a Codex skill: when it runs and what it produces

A constructed API review identifies a timeout limit changing from 120 seconds to 60. It offers a request with a 30-second timeout as evidence that an existing caller will be rejected. The finding names a change in the contracts, and the report is valid JSON. Its example does not demonstrate rejection: 30 seconds is accepted under both limits.

That is an output defect you can test without running a model. If you maintain a Codex skill, you also need to know whether the skill was selected for the task. I would record those results separately. Otherwise, a missed skill and a skill that produced an incorrect review can look like the same failure.

The contracts and review outputs below are synthetic, authored examples. The downloadable grader runs offline. No Codex run produced these outputs, and the proposed skill has not been installed or activated. Product guidance is based on official documentation checked on September 28, 2026.

Establish the answer before testing the skill

A skill is a package of reusable instructions and optional resources. In this exercise, request-contract-review compares the old and proposed rules for an API request and identifies requests that would cease to be accepted.

The fictional create-job API accepts a JSON object. Its contract uses a flat exercise format with strings, integers, required fields, permitted string values and inclusive numeric bounds. It rejects extra fields. This format has its own validator; it does not implement OpenAPI or general JSON Schema.

FieldOld contractProposed contractEffect on previously accepted requests
priorityOptional; normal or urgentRequired; same permitted valuesRequests omitting it are rejected
timeout_secondsOptional integer from 1 through 120Optional integer from 1 through 60Requests using 61 through 120 are rejected
client_referenceAbsentOptional stringExisting accepted requests can still omit it
job_nameRequired stringSame rule; description rewordedAcceptance is unchanged

Here, “breaking” means the new contract rejects a request that the old contract accepts. It does not decide whether the product should make the change. The maintainer still decides whether that change is intended and what callers should do.

The answer key therefore requires two findings, each supported by a request that isolates the relevant change. To demonstrate the newly required priority, omit it while keeping the timeout within both limits:

{"job_name": "nightly-report", "timeout_seconds": 20}

To demonstrate the lowered timeout limit, include a valid priority and use 90 seconds:

{"job_name": "nightly-report", "priority": "normal", "timeout_seconds": 90}

The second request is accepted by the old rules and rejected by the proposed rules for the timeout alone. Omitting priority from that request would introduce another rejection cause. That distinction matters when judging whether a review has explained a particular change.

The optional field and description edit are controls: a reviewer should acknowledge them without inventing additional breaking changes. The answer key is written from these input rules, independently of the review output being graded.

Test when the skill should be selected

OpenAI's January 22, 2026 article on evaluating skills recommends explicit, implicit, contextual and negative prompt cases. I would use all four to test the proposed review skill.

CategoryExample requestExpected selection
ExplicitUse $request-contract-review to compare before.json and after.json and return the requested review JSON.Yes
ImplicitCompare these two flat request contracts. Report breaking changes with requests the old contract accepts and the new one rejects.Yes
ContextualCheck the proposed change described in change-request.md before we accept it. Which existing requests would stop working?Yes
NegativeList the request-field names inside fields in before.json alphabetically for a reference index.No

Explicit invocation names the skill; the examples use the $skill form documented for Codex CLI and the IDE extension. Implicit selection asks for work matching its description without naming it. The contextual case makes the need clear through the task file. It remains a test category, not a third native invocation mechanism. A negative case is nearby work the skill should leave alone.

Current skills documentation describes loading skill metadata first and full instructions when the skill is used. It also documents explicit and implicit invocation. Repository skills use .agents/skills; the older evaluation article's .codex/skills example should not determine a new installation path.

The download includes two cases per category. Its proposed skill description names the comparison task, requires before-and-after inputs and excludes field listing or explanation of one unchanged contract. Those exclusions give the negative cases a reason to exist.

Before a native evaluation, check that implicit invocation is enabled if you expect the implicit and contextual cases to select the skill. OpenAI documents an allow_implicit_invocation policy that can disable that behavior while allowing explicit invocation. A deliberately disabled setting changes the test's expectation. Skill invocation policy is part of the setup.

Keep selection evidence with each run

A useful selection test needs a record of what the runtime exposes. For a native evaluation, I would:

  1. Record the Codex surface and version, model/settings, effective permission policy, exact skill bytes and exact input files. Check that the intended skill is available on that surface. Availability is a setup check; the individual run still needs a selection observation.
  2. Give each case a fresh task with the same declared setup. Supply only its task inputs. Keep the grader, answer key and authored good/bad reviews outside the task's accessible evaluation workspace. If that separation cannot be established, record possible answer-key exposure.
  3. Retain the full emitted run record and the final output. For CLI automation, the current non-interactive guide documents --json event output and a separate --output-last-message file. These capture options do not promise a dedicated skill-selection event.
  4. Assign a selection result from the evidence actually available. Preserve the exact record and skill identity supporting it. Grade the produced work separately, even if selection remains unknown.
StateEvidence required
SelectedThe runtime explicitly records selection or use of the identified skill for that run
Not selectedA documented, complete selection record for the run establishes that the skill was not used
UnknownThe surface exposes no adequate record, the trace is incomplete, or the evidence is ambiguous

A prompt requesting the skill establishes intent. A catalog entry establishes availability. An incidental read of a skill file establishes that its bytes were read. None of those observations alone settles whether the runtime selected it for the task. Keep such observations alongside the selection field instead of upgrading them.

The download's selection-results.template.json has one row per prompt case. Every observed selection starts as unknown, and every output result starts unset. These are templates for future observations, not recorded Codex results.

Grade the meaning of the review

The offline skill-test exercise contains the contracts, answer key, proposed skill source, prompt cases, review fixtures and a Python grader. Extract it and open the codex-skill-test-example directory. Python 3.10 or later is sufficient; the program uses the standard library.

The proposed skill is stored as skill-source.md, outside an active discovery directory. Its output instructions specify the exact category labels and properties the grader expects. This avoids failing a correct review merely because the test never told it the required report format.

python3 -B grade.py fixtures/good-review.json
python3 -B grade.py fixtures/invalid-counterexample.json
python3 -B grade.py --self-test

The good fixture passes 26 mechanical checks and exits with status 0. The defective fixture exits with status 1: its timeout counterexample uses 30 seconds, so it remains valid under the new contract. The self-test checks 16 positive, defective and boundary controls.

The grader checks more than field presence. It compares the finding set with the independent answer key, rejects missing or spurious findings, checks the exact before-and-after values, and applies both contracts to every counterexample. It requires each new rejection to have the cause claimed by that finding.

For example, a report that correctly identifies the timeout change but cites a limit of 90 instead of 60 fails. So does a request already invalid under the old contract. Reordering equivalent findings passes. These controls test whether the grader responds to meaning while tolerating an irrelevant ordering difference.

Structured output can make those checks easier to automate. OpenAI documents --output-schema for constraining a final response's shape. The schema still needs a separate correctness test against task evidence. Structured output documentation describes the formatting facility.

The distinction also applies to execution traces. The evaluation article's starter command check matches a command string in started or completed events. That check alone doesn't establish a successful exit or correct result; the article later recommends build and runtime checks. For a skill that runs a program, check the completion status and inspect the resulting artifact.

Include a rubric for the explanation

The grader checks that client_impact and decision contain text. It does not establish that the text is true. Run this deliberate control:

python3 -B grade.py fixtures/false-prose.json

It passes the mechanical checks while claiming that an authentication token expires immediately. Neither input contract says anything about authentication. A reviewer should reject that explanation.

The human rubric asks whether each finding:

  • Names the affected caller and explains the exact cause of rejection.
  • Makes a decision actionable by identifying the changed field or limit.
  • Distinguishes the contract's consequences from the maintainer's choice about accepting them.
  • Avoids invented implementation behavior, rollout history or business intent.

A valid timeout explanation says a caller requesting 90 seconds exceeds the proposed 60-second limit. A useful decision asks whether rejecting those existing requests is intended and how affected callers should change. Neither requires guessing what the production service does.

Negative prompt cases need their own output expectations. A request for field names should produce job_name, priority, timeout_seconds in that order. A request to explain the old priority field should say it is optional and permits normal or urgent. Requiring a full change review for either case would reward unnecessary skill use.

Use the results to choose the revision

Keep the prompt, observed selection, mechanical result and human rubric result together. Their combinations suggest different investigations:

Observed resultNext investigation
Explicit case selects the skill and produces a correct review; an implicit case demonstrably misses itInspect the skill description, availability and invocation policy
Selection is established, but the timeout counterexample failsInspect the skill's review steps and the task evidence
A negative case demonstrably selects the skillNarrow the stated applicability and rerun both matching and neighboring requests
Output passes, but selection is unknownRetain the output result; improve observation before claiming selection worked
Mechanical checks pass, but the explanation invents authentication behaviorRevise the grounding instructions and apply the human rubric again

Change one declared part of the skill, retain both versions, and compare the same cases under the same setup. Report unknown selection separately from measured selection, and show any regressions in cases that previously worked.

The final report should let a maintainer point to the missed request, the wrong finding or the unsupported explanation that motivated the next edit.