I watched a panel of four AI agents disprove a sentence I had written, checked to my own satisfaction and shipped into system documentation. The sentence asserted that a set of terms appeared nowhere in a large published archive. I had searched the places where I expected to find them, found nothing, and written “verified absent.”
I asked the panel to try to prove the work wrong. One agent treated that absence claim as a target and produced counterexamples in minutes. My verification had been a search for agreement wearing a lab coat: I had let the places I expected to find the terms define the search, then stated a conclusion about the whole archive.
That experience changed what I ask for in a review. For consequential work, I want a refutation assignment: try to prove the claim wrong and show the attack. Asking for a more critical tone leaves too much of the task unchanged. The useful output is an input that breaks the function, a source that contradicts the claim, or a scenario in which the design corrupts data.
Give each reviewer a claim to test
“Review this carefully” leaves a lot of judgment about where to look. For consequential work, I assign distinct failure classes. A correctness reviewer tests whether the result meets the stated contract. A security reviewer examines the trust and authority boundaries. A failure-handling reviewer asks what happens when a required dependency cannot run. A provenance reviewer checks whether the evidence supports the factual claims.
Those assignments produce different questions about the same protective check. Does it detect the content it is meant to detect? Can an input avoid that check? What happens if the checker exits with an error? Does a successful test show the installed version's behavior, or only a function called directly by the test?
In my own work, a failure-handling lens exposed a problem in a pre-commit guard, the program that checks changes before Git records them. It brought attention to what happened when the checking mechanism itself failed. That is why I keep failure handling separate from whether the detection rule matches the expected text.
Several reviewers can share assumptions and repeat the same finding. I choose lenses according to the claims and risks, then ask each reviewer to name the bounds of the attempt. “No defect found in these inputs” is more useful than an unexplained approval, and more accurate than a general assurance of safety.
I would use this prompt for the initial review:
Your assigned lens is [correctness, security, failure handling or provenance]. Test the stated claims in the supplied work. Try to construct a concrete counterexample: a valid input with the wrong result, an unsafe permitted action, a failure path or source evidence that contradicts a claim. Preserve the exact version, input, expected behavior and observation. Do not manufacture an objection if you cannot demonstrate one. If a plausible concern remains undecided, label it unresolved and name the missing input, next check and responsible owner. If you find no defect, state what you examined and what remains untested.
The owner and next check matter. A finding that says “this might be a race condition” has not yet told the team what to do. A finding that specifies two overlapping writes, the state each writer reads and the invariant they could violate gives a verifier a test to construct. If the environment needed for that test is unavailable, that condition belongs in the record.
Keep the finding's status separate from the release decision
A candidate finding can end in three states:
| Status | What supports it | What to do with it |
|---|---|---|
| Demonstrated | An observed counterexample or other decisive evidence shows the claimed defect under stated conditions. | Repair the defect, preserve the case and verify the repair against it. |
| Refuted within stated bounds | Evidence contradicts the finding for the version, inputs and conditions examined. | Retire or narrow that finding, retaining the bounds and evidence. |
| Unresolved | Missing evidence, an unavailable environment or an inconclusive test prevents a decision. | Name the missing input, next check and owner. Keep the question open. |
Suppose a release requires evidence that a protective check stops an unauthorized write. If the test cannot be run, I can withhold release without declaring that the check failed. The acceptance condition is unmet; the behavior is still unknown. Conflating those two statements makes it harder to diagnose the next result.
The same distinction applies when a verifier cannot reproduce a reported defect. Perhaps the original report was wrong. Perhaps the verifier used a different version or omitted a prerequisite. Failure to reproduce alone does not choose between those explanations. A bounded refutation needs evidence for its bounds: the relevant version and input, a working test and a reason the observation contradicts the reported failure.
I use a separate brief for that verification:
Test this candidate finding independently against the specified version and contract. Check the input and prerequisites before accepting its conclusion. Record whether the defect is demonstrated, refuted within stated bounds, or unresolved. Include the exact command or source passage, expected and observed results, and relevant controls. A missing fixture or unsuccessful setup is unresolved evidence. Keep any release hold separate from the finding's truth status. Propose a repair only for a substantiated defect; for an unresolved question, specify the next discriminating check and its owner.
Independence here means doing the check, not restating the first reviewer's explanation. A second agent can repeat the same misunderstanding. A verifier may need to read the code path, inspect a source document or execute a reproduction that would distinguish two plausible explanations.
What the reviews caught in my work
In one run, a panel surfaced twenty-nine candidate defects. Verification separated out five real, exploitable holes in protective automation that had passed conventional review, mine and my agents' both, over months of use.
The demonstrated cases included a filename containing a quote character, a change arriving as a rename, and an invalid exclusion pattern. Each forced attention to a different assumption in protective automation. Checking ordinary paths was insufficient if the parser mishandled a valid filename; checking additions and modifications was insufficient if a rename escaped the same rule; a scanner's failure could not safely be interpreted as a clean result.
The five holes became regression tests. I described them and the corresponding fail-closed behavior in A warning that doesn't block is a bug.
Independent verification also rejected plausible-sounding defect reports before I spent a morning fixing problems that did not exist. I value that negative catch nearly as much as finding the holes.
For work with consequences like these, a single “please review this critically” feels like asking the work's admirer to officiate its trial. I argued for the same structural humility in treating a green test suite as a warning rather than a clearance: actively challenge the signal you were hoping to receive.
A counterexample outside the search folder
The downloadable review exercise contains two synthetic archive files, a positive-control file, supplied change records and a Python checker. It needs Python 3.9 or newer and no extra packages. All text and identifiers are constructed for this exercise; it calls no AI service or native protective system.
The marker ORCHID_REVIEW_SENTINEL appears in an archived historical file. The initial search looks only in the current guide and returns no matches. A separate control file contains the same marker, so the checker can first establish that its literal search finds a known occurrence. The complete inventory for the exercise is exactly two archive files, and searching both exposes the counterexample.
The change records demonstrate a different failure. The stated contract requires inspecting added, modified and renamed text. The seeded checker inspects additions and modifications but returns ALLOW for a rename without examining its content. A sensitive rename therefore supplies a valid failing input. A sensitive modification supplies a counterexample to the separate allegation that the checker skips modifications, too.
After extracting the archive, run these commands from its review-findings-exercise directory:
python3 -B review_example.py
python3 -B test_example.py
The first command prints the evidence and dispositions. The second runs eighteen tests against the worked expectations and controls.
| Case | Observation | Finding or decision |
|---|---|---|
| Initial archive search | No match in one of the two archive files. | Absence across the full archive remains unresolved. |
| Complete archive search | Marker found in archive/history/retired.txt. | The finding against the broad absence claim is demonstrated. |
| Sensitive rename | Seeded checker returns ALLOW; the contract requires BLOCK. | Rename defect demonstrated. |
| Sensitive modification | Seeded checker returns BLOCK, as required. | The claim that it skips this modification is refuted for the supplied case. |
| Missing environment fixture | No test observation is available. | Finding unresolved; acceptance stays on hold if this is a required condition. |
The corrected checker blocks the sensitive rename and allows clean records. That clean-input control matters: a checker that blocks everything has not met the stated contract. Other controls deliberately break the search, omit an archive file and supply an incomplete inventory. Those failures must not become evidence of absence.
The README works through the expected results independently of the checker.
For the archive example, the next action is to correct the absence claim and preserve the counterexample. For the missing environment, it is to obtain that fixture and run the specified check.
Choosing the review effort
A refutation panel can spend several times the tokens and minutes of a single review. I reserve that expense for work where being wrong is costly: protective systems, factual claims headed for publication, architectural decisions and difficult-to-reverse changes. For a routine refactor, a focused linting issue or a draft I will rewrite anyway, one reviewer with a specific brief is cheaper and usually suffices.
A panel can manufacture objections. It can miss the failure class that mattered because I assigned the wrong lenses. Several agents can agree for the same mistaken reason. These are reasons to require concrete evidence from the panel and its verifiers. Silence tells me which commissioned attempts found nothing; it does not certify the work.
The topic list for my writing goes through hostile lenses: a jaded reader who assumes every AI-adjacent post is empty, and a skeptic who rejects anything without a transferable pattern. Article drafts face the same panel structure before publication. A prose finding needs the relevant passage, the reader harm or factual defect, and a repair that improves it. A reproducible code failure and an editorial judgment require different evidence. The reviewer should preserve a strong, distinctive sentence when the proposed edit fixes nothing.
The choice of what is worth building or publishing remains mine. Thinking through a strategic decision with AI addresses that earlier judgment. I want the work to earn its way past well-aimed attacks, and I want every alleged defect to do the same.
Commission refutations, give them distinct ways to attack the work, and test the findings before acting on them.