We had finished the review, generated the PDFs and sent them to leadership when we learned that we had reviewed the wrong branch.
I had built an open-source code review runbook, a structured set of questions with review depth chosen by project complexity. The tier guide called for roughly 150 checks at Tier 2, Standard, and 900-plus at the highest tier. The checklist did its job on an advisory engagement reviewing a security startup's Python/FastAPI backend and Next.js/TypeScript frontend. Most of what I learned was about how we conducted the review. Some of those lessons seem obvious in retrospect. None was obvious in the moment.
September 2026 update: The February runbook is now available as the synthesis-codebase-review skill.
We reviewed the wrong branch
We assumed main was the right target. We used the Standard tier for our fifteen-category review, produced detailed reports and shared the PDFs with the CEO and head of engineering.
Then the engineering lead told us they wanted develop reviewed. It was hundreds of commits ahead of main in both repositories. The team used a git-flow model in which main changed on releases. We'd reviewed an older product state than the one they wanted assessed.
Ask the owner which branch or release represents the intended review target before starting, then record the exact commit. Commit dates and branch differences help expose a mistaken assumption. They don't choose the target for you: an older release may be exactly what a release audit needs to examine.
That question became a pre-flight step in the runbook. It should have been there from the start.
The unintended baseline
We initially saw the wrong-branch review as wasted work. We had to run the review again against develop. Having the first report beside the second, though, made the comparison much more useful than another standalone list of findings.
Nearly half the backend findings from the main review were addressed in the develop state. We saw CSRF middleware, structured audit logging and a token-management overhaul. In the frontend comparison, none of the earlier findings was fully addressed, a few were partially addressed, and the largest component was larger.
Those were differences between the reviewed branch states. They didn't tell us that the team made those changes after receiving our first report. Calling a difference a response to a review requires evidence of when the change happened and why. Even Git ancestry establishes a relationship between commits, not the cause of the work.
A comparison gives leadership more to work with: which concerns remain, which have changed, and where further investigation is needed. I use five familiar labels, with the supporting evidence beside each one:
| Label | What the comparison needs to show |
|---|---|
| Fixed | The finding's repair criterion is satisfied in the compared state. |
| Partially fixed | Some of that criterion is satisfied; the remaining gap is named. |
| Still present | The same finding remains in the reviewed state. |
| Worse | A specific consequence, exposure or structural problem has increased. |
| New | The finding is newly observed; the report separately establishes whether it is also new in the code. |
If the relevant code wasn't assessed, say not assessed. A missing finding in a report is too easily mistaken for a fix. Keep a stable finding ID when a file moves, and record scope and ruleset changes so a newly added check doesn't manufacture a deterioration in the product.
The label also doesn't set the priority. A still-present privileged credential exposure can need immediate attention while worsening formatting can wait. “Still present” describes the observed state; it doesn't explain the team's intent or give permission to leave a serious risk unresolved.
If I could offer one recommendation about code reviews, it would be to retain a usable baseline. Keep dated reports with the source commits, requirements, scope, exclusions and rules used. When no earlier review exists, today's review becomes the first baseline.
A comparison you can inspect
Consider this fictional comparison. Both snapshots are reviewed on the same day with the same scope and rules. Their ancestry is unknown.
| Field | Record |
|---|---|
| Finding | F-credential |
| First source | snapshot-A, branch main, scope scope-1, rules rules-v1 |
| Compared source | snapshot-B, branch develop, scope scope-1, rules rules-v1 |
| First observation | Credential finding present |
| Compared observation | Finding verified absent in the supplied comparison record |
| Supported conclusion | Absent in the compared snapshot |
| Timing and cause | Post-review remediation and review causation are not established |
That row supports a difference between A and B. To describe a later repair, establish the chronological relationship and test the repair criterion. To credit the first review with prompting it, obtain evidence connecting the review to the change.
The comparison workbook includes this example among nine core cases: a chronological repair with no causal attribution, changed rules, unexamined code, equal totals hiding turnover, a renamed file, severity versus status, incomplete identities and a first review. Further cases examine unknown prior states and repairs already present in both snapshots. An unexamined first snapshot cannot establish that a finding stayed present; a repair criterion met in both snapshots cannot count as a newly completed repair. Its local Python checker evaluates supplied fictional records. It doesn't inspect a repository or authenticate the observations recorded in one.
What the systematic approach catches
The runbook kept concerns in view that were easy to miss while concentrating on the architecture or application logic.
random.choices() for MFA codes. The backend used Python's random module to generate multi-factor authentication codes. For that randomness choice, the appropriate replacement is secrets.choice(): Python's documentation reserves secrets for security-sensitive uses and describes random as intended for simulation. A reviewer concentrating on architecture can easily overlook a randomness choice. A named security check keeps it in view. The rest of the authentication flow still needs its own review.
Sensitive material in AI coding context. Both repositories tracked .cursorrules files containing an infrastructure inventory: server IP addresses across three environments, EC2 instance IDs, security group IDs, CloudFront distribution IDs, SSH key paths and deployment directory structures. One included a developer's full local filesystem path.
These files help an assistant understand a codebase. Their contents deserve the same scrutiny as configuration and documentation. A tracked instruction file can be useful; an unnecessary infrastructure inventory can give someone with repository access a detailed reconnaissance map. I added an explicit check for coding-assistant context files so they wouldn't disappear between the code and documentation portions of a review.
Tests that verified imports instead of behavior. Both repositories had test suites that looked comprehensive at a glance: dozens of files, organized directories. Reading them changed the picture. Many frontend tests only verified that modules were importable. API route tests mocked fetch globally and exercised the mock infrastructure instead of the route handler. Backend tests relied on pre-seeded database data and silently skipped when it was missing.
“Tests exist” and “tests pass” leave a lot unanswered. Read the assertions and trace the call into the behavior supposedly being tested. Would changing that behavior make a test fail? The runbook's starting sample became three to five test files. Expand the inspection when the sample exposes a pattern; that sample alone doesn't establish coverage of an application.
Cross-repository patterns tell a bigger story
Several findings appeared in both the backend and frontend: hardcoded JWT secret fallbacks, infrastructure IP addresses in source, default database credentials in docker-compose.production.yaml, missing LICENSE files and deployment workflows that skipped CI checks.
Repeated problems across repositories call for a process response as well as individual code fixes. Shared templates and team practices deserve review alongside each affected file. We made those process recommendations in the combined executive report, where they landed differently from isolated repository findings.
The backend/frontend asymmetry also helped leadership consider resource allocation. The backend state contained substantial security work while frontend gaps remained. The team may have deliberately prioritized backend security. Making the difference visible allowed that choice to be discussed with evidence, without treating a comparison label as a diagnosis of the team's motivation.
Security debt plays whack-a-mole
The favorable backend changes were substantial. Yet the second review also exposed critical concerns: plaintext API keys in workflow comments and the context file's infrastructure inventory. The net number of critical findings barely changed.
An unchanged total can hide both valuable repairs and different serious problems. Keep the identities and consequences of the findings visible. A count of one critical finding before and one afterward doesn't establish that the risk stayed the same.
The team had addressed known problems but had no systematic process for keeping new secrets out of the codebase. Every new feature was another chance for a credential to land in a config file, a comment or a CI workflow.
The production keys in comments surprised me. A commented-out part of a deployment workflow described a planned migration and included actual API keys. Git tracks a comment just as it tracks executable code. Comments, TODOs and disabled blocks belong in credential scanning.
For a potentially exposed credential, start with the owner's response: revoke or rotate it and assess possible exposure or use. Removing the string, moving it into an environment variable or adding a scanner doesn't invalidate an old key.
Then examine prevention. A local pre-commit scanner, using tools such as gitleaks or TruffleHog, can check changes before a commit. GitHub's secret scanning detects supported secrets in repository content and history; push protection can block detected secrets during a push. Coverage, configuration and bypass behavior matter. Review the actual controls alongside the findings instead of assuming that the absence of a detected secret proves prevention works.
Deliver findings people can use
We led the feedback with the team's strengths: the multi-tenant architecture, assessment pipeline design and CI/CD discipline. Then we covered critical issues, high-priority findings and structural improvements. Showing that we understood the good engineering helped the critical findings arrive as constructive guidance.
I've seen that pattern in advisory work whenever I've been able to choose the report's structure. I want a team to recognize its product in the assessment, including the parts worth protecting. Earned praise makes the recommendations more credible; vague compliments don't.
We initially produced Markdown files. The CEO immediately asked for "nicely formatted PDFs I can share via Slack." For this advisory audience, a polished PDF communicated professional work and was easy to circulate. When the audience includes non-engineers, invest in presentation that helps them use the findings.
What changed in the runbook
The next runbook version incorporated eight practical improvements:
- A branch-selection pre-flight check.
- Delta review mode with the five finding labels.
- A check for sensitive material in AI coding configuration files.
- Test-quality verification that requires reading actual tests.
- Credential scanning that includes comments and disabled blocks.
- Strengths-first reporting at every tier.
- A question about pre-commit secret-scanning prevention.
- Dated deliverable folders for longitudinal comparison.
Using the runbook on a real product improved both the questions and the way we used the answers. I expect it to keep improving through engagements and feedback. That's why I made it open source: a team that reports a missing check can improve the next team's review.
This article is part of the code review series. See also: Code review that scales (choosing review depth) and When a code review runbook becomes an upgrade playbook (turning findings into work).