A missing Python dependency stopped my commit scanner. It did not stop the commit. The pre-commit hook that scans my repositories for credentials and confidential patterns had announced that it could not run, then allowed the action it was supposed to check.
Revision prepared September 28, 2026. The incident and repair accounts below are historical; the downloadable exercise was constructed for this revision.
The diagnostic was right there in the commit output. Nobody saw it when it printed. A required check that proceeds after announcing its own failure depends on somebody noticing the announcement quickly enough to undo what just happened.
The hook’s wrapper collected its scanning patterns from a Python helper. That helper needed PyYAML. When the import failed, it produced no policy output. The wrapper treated the empty output as an empty pattern list: no patterns, nothing to check, exit zero.
Which Python ran mattered. My machine carried three interpreters, and the daily default had the dependency. That made the guard appear healthy in ordinary sessions. A session that reordered PATH could select an interpreter without it. A control whose effectiveness depends on interpreter order is a coin flip wearing a uniform.
Git invoked the shell wrapper; the wrapper invoked Python; Python loaded the policy; the scanner used the resulting patterns. The error at the Python step disappeared before the wrapper returned its verdict to Git.
I wrote earlier about why a green test suite is a warning, not a clearance. Tests can pass while checking the wrong thing. Here, the detector was dead and nobody had scheduled the funeral: a successful exit survived the loss of the check itself. Both deserve investigation before anyone reads silence as safety.
Property one: a required check fails closed
A required guard fails closed when it stops the guarded action if it cannot produce a valid verdict. A commit scanner, release gate or other mandatory check needs an error path that its caller actually enforces. An advisory message can remain advisory; a failed prerequisite cannot count as permission to proceed.
Investigating a blocked commit costs time. An unscanned commit can expose a credential or a confidential name. That risk is the reason to stop on a missing verdict, while giving legitimate work a precise way through the policy.
The original wrapper used this shape:
# Historical failure shape; do not use as a guard.
eval "$(python3 helper.py)"
The command substitution supplied the helper’s standard output as an argument to eval. When the helper failed without producing that output, eval received an empty string. Bash treats that empty evaluation as successful. The helper’s diagnostic and failed status did not become a failed status for the wrapper. The Bash manual documents this behavior.
Nearly every hand-rolled wrapper I had inspected had some version of this problem, including snippets copied between projects. Search wrapper code for command substitution fed into eval, warn-and-continue branches and error masking with || true. For each, check the outer command’s status when the inner check fails.
The repair captured the helper’s exit status before evaluating its output, refused a nonzero result, and required an explicit completion sentinel. It also refused an empty active pattern set. Those checks answer different questions: did the helper finish successfully, did it finish emitting the expected result, and is there a policy to enforce?
A scanner’s protocol should distinguish at least these outcomes:
| Scanner outcome | What the caller knows | Required action |
|---|---|---|
| Complete, valid clean verdict | The specified check finished without findings | Permit this gate to pass |
| Complete verdict with a finding | The check finished and found a policy violation | Block and identify the finding |
| Missing, malformed or failed verdict | A clean result has not been established | Block and identify the failed check |
A zero-findings result needs to be an affirmative result. An empty file, absent error message or successful outer shell command is insufficient. A deploy gate can use a structured verdict artifact; a local scanner can return a small result object. The caller must validate the result and enforce it. Git, for example, aborts a commit when an invoked pre-commit hook returns nonzero. That only helps if the wrapper preserves the failure. Git’s hook documentation also makes clear that hooks without the executable bit are ignored.
The later regex hole
The dependency fix did not close every failure path. The July 29 wrapper already captured the helper’s status and required its sentinel. It still filtered detected matches through this allowlist branch:
STAGE1=$(echo "$STAGE1" | grep -ivE "$ALLOWLIST_REGEX" || true)
The inverted grep was supposed to remove permitted matches. A valid expression could legitimately remove every line. But an invalid expression could also produce no standard output, this time because grep had failed. The || true turned that failure into success, and the assignment replaced the detected matches with an empty string. The later match check saw nothing to block.
This was a particular pipeline failure, not a universal property of broken allowlists. The old helper assembled the allowlist without validating it on the normal policy-emission path. Error masking also appeared around path exclusions. The July 30 repair added explicit validation of those expressions, alongside other hardening.
The distinction between no selected lines and an error is part of grep’s documented exit-status contract. GNU grep normally returns 0 when it selects a line, 1 when it selects none and 2 on error; other implementations may use a higher error status. A wrapper needs to preserve that distinction rather than convert every nonzero status into success.
Policy validation belongs on the path that actually makes the decision. A doctor that rejects a malformed pattern is useful, but it cannot compensate for a live scan that accepts the same pattern without validation. Missing policy, invalid expressions and an incompatible policy version must stop a required check. The engine must enforce its declared version contract rather than guess how to interpret the file.
I also removed the helper’s third-party dependency and compared hashes of the emitted policy under the three interpreters on my machine. The outputs matched. Keeping the helper in the standard library removed a package-installation step from the protective chain. If a guard needs an external package, installation must be controlled, health checked, and a missing package must block the action.
Property two: the guard needs its own doctor
My hook had been silently weakened for a stretch I could establish only afterward, by audit. Its normal quiet output had looked the same as its failure to protect anything.
Every protective layer I run needs a doctor: an explicit health check for the protective chain. It should answer questions that a clean scan of one change cannot answer: is the intended policy available, is the expected engine installed, and will the caller invoke it?
My doctor checked the configuration and its version contract, validated patterns in both regex dialects used by the chain, checked hook-path wiring, and compared the installed engine with its source. The two regex checks had a concrete purpose: a pattern accepted by the authoring-time parser could still fail in the engine that executed it. A source/installed comparison could expose a hot fix that existed only in the installed copy.
Comparing files or hashes establishes whether the copies match; it does not establish that the code is correct. Checking a global Git setting does not establish the effective configuration of every repository. A repository may also depend on a chained hook, so its presence, executable state and actual invocation need checking. If a required delegate cannot run, a wrapper that merely skips it has recreated the same failure one step later.
An absent entry-point hook is different again: code Git never invokes cannot refuse a commit. A doctor should report that protection is unavailable, not promise that commits will be blocked.
I made the doctor part of installation, the start of the working day and new-machine bootstrap before the first commit. Treat an unhealthy installation check as a failed installation, with a repair path before first use. The goal is to catch silent decay by the next working day. Meeting that goal requires noticing a missed check or a stale result as well as a failed one.
Monitoring therefore needs a record of the last completed check and an explicit freshness rule. A missing result, an old result and a recent failed result require different diagnoses. The workbook below makes those states visible at a fixed supplied time. It does not run a scheduler or deliver an alert.
For each protective control, I ask: if this broke right now, would the action stop? Who would notice if the check never ran? What evidence supports the promised detection interval? If nothing would stop the action and nobody would ever notice, the control is theater.
Property three: a false-alarm budget near zero
A guard that repeatedly blocks legitimate work makes bypassing it attractive. A tool that emits benign warnings on every successful run makes its real warnings harder to notice. A guard’s false-positive rate is a security property and deserves the same design attention as its detection rate.
While rebuilding this system, my own hook blocked my own commits twice. The work was about sensitive patterns, so the pattern definitions matched the scanner. There was a legitimate reason for those lines to exist and a sanctioned way to express it: a narrowly scoped, version-controlled exception. The bypass flag never got typed during those repairs.
My path exclusions originally matched anywhere in a path. A decoy directory with the right name could exempt a file the rule was never intended to cover. Anchoring exclusions to explicit roots narrowed both accidental overreach and that bypass opportunity. An exception for policy/catalog should not silently cover other/policy/catalog or policy/catalog-copy.
Credential patterns belonged to a mandatory tier that scanned every path, including excluded paths. An exception for writing about sensitive vocabulary was not permission to place a credential in the same file. In a policy with that requirement, the mandatory scan must happen before exemptions can discard its input.
The rebuilt health check also initially printed harmless interpreter warnings on every run. I fixed that noise the same night. Useful diagnostics identify a failed check or actionable condition; routine implementation chatter should not compete with them.
A false-positive budget is not a reason to weaken a true-positive test. Keep a legitimate input that must pass next to a forbidden input that must block. When tuning an exception, run both. Otherwise, removing the annoyance can quietly remove the protection.
Exercise the wrapper without making a commit
Download the fictional guard workbook. It contains a Python scanner, a subprocess wrapper, fixed input fixtures, a heartbeat classifier and harmless Bash/grep probes. The scanner reads virtual files supplied as JSON. It does not inspect a repository, install hooks, use credentials or perform the action it labels PERMIT.
The ordinary marker DEMO_POLICY_MATCH represents a finding that can have a narrowly scoped exception. DEMO_MANDATORY_MATCH represents the tier that must survive every exclusion and allowance. Neither is a credential. A canary here is a harmless input with a known expected result.
Run the commands in the workbook’s README from its extracted directory. The central command is:
python3 -B exercise.py
The expectations were fixed before the scanner and wrapper were implemented. They require a clean input to pass, a finding to block, and an engine error to block for a different reason. They also check a missing helper, absent completion sentinel, incompatible policy version, malformed allowance, misleading path names and a mandatory marker inside an excluded path.
The workbook checks the caller’s decision and exit status, not just whether the scanner printed something reassuring. It then deliberately introduces seven wrong behaviors, including accepting empty failed-helper output and skipping the mandatory scan under an exclusion. Each must contradict the same fixed expectation that the correct implementation satisfies.
The shell’s exit status
Probe on DEMO_POLICY_MATCH | Observed exit | Observed standard output |
|---|---|---|
| Valid inverted grep that retains the marker | 0 | Marker retained |
| Valid inverted grep that removes the allowed marker | 1 | Empty |
Invalid [ expression | 2 | Empty |
The same invalid expression followed by || true | 0 | Empty |
The last two probes both printed a regex diagnostic to standard error. The final status changed anyway.
The supplied source and a clean extraction of the download were checked against all 42 fixed cases and seven deliberate faults. The heartbeat cases distinguish MISSING, STALE, FAILED and HEALTHY, and reject malformed or future-dated records. These are finite checks of a teaching example, not evidence about a deployed guard’s detection rate or today’s installed configuration.
Test the actual protective boundary, with harmless inputs
During the repair, I staged test changes in real repositories and checked both refusal and permitted work. Credential-pattern matches blocked in ordinary and excluded paths, legitimate edits passed through their narrow exception, and a confidential-pattern match in a public repository blocked with the offending line identified. The adversarial AI review had found five fail-open holes in the earlier engine; I added regression tests for all five and checked the actual commit boundary.
For a guard you maintain, use an authorized isolated environment and harmless fixtures that exercise the actual caller. Do not put a live credential into a repository to prove the scanner works. Keep the expected refusal separate from the test harness’s own success status.
- Break the required guard deliberately. In the isolated test environment, make its dependency or policy unavailable. Confirm that the guarded action is refused, with a diagnostic that identifies the failed prerequisite.
- Distinguish broken from clean. Check the result format, completion evidence and caller’s exit status. An empty result must not mean both “no findings” and “the engine failed.”
- Use a known positive canary. Feed an inert marker through the actual protective boundary, with a rule that is expected to catch it. Verify the refusal; a unit test alone does not establish that the caller enforces the result.
- Exercise the exceptions. Test intended legitimate work and nearby inputs that must still block. Send the mandatory-tier canary through every exclusion and allowance path.
- Check health and freshness. Run the doctor, inspect when it last completed unattended, and verify how missing or stale results reach a responsible person. A daily schedule is one part of that chain.
- Count false alarms. Review the legitimate work blocked during the last month and narrow the responsible rules without losing the paired forbidden cases. Each needless block spends some of the attention the next real alarm needs.
AI agents raise the stakes in both directions: more commits, more drafts and more surface for a guard to cover. They also change the economics of compliance. In my workflow, agents routed legitimate blocks through the sanctioned exception path and ran the doctor as part of the daily ritual without getting bored. That repetition can make dependable guardrails easier to sustain as the work grows. The refusal still belongs in the mechanism that controls the action, with tests as the mechanism changes.
Protective automation is production code with an adversary. Make a required guard refuse a missing verdict, give its doctor a monitored execution path, and spend false alarms like the currency they are.