Code Review That Scales: Choosing Review Depth by Risk

A short checklist can miss the risks in a production system; an undifferentiated enterprise checklist can bury them. I use tiers to organize review depth, then check the project's consequential risks before deciding which work applies.

Revision prepared September 28, 2026: adds risk overrides, updated checklist counts and a fictional coverage workbook.

A team of three building an internal command-line tool needs a different review from a team of fifty building a financial trading platform. Requiring both to work through the same enterprise checklist wastes attention. Yet the internal tool may hold a credential that can change production data. Its small size gives us no reason to leave that credential path unexamined.

I wrote the Enterprise-Grade Codebase Review runbook for engineers and engineering leads who make these choices regularly, including those directing coding assistants such as Claude Code, Cursor or Copilot. It became the synthesis-codebase-review Agent Skill, a reusable instruction package. The tier organizes the baseline review; the consequences of a failure determine which additional checks belong in it.

Why one checklist does not fit every project

A short list that says to check for bugs, write tests and use meaningful names can help establish basic habits. A payment system, a service storing health data, or an application serving millions of requests also needs attention to matters such as secrets, dependency licensing, tenant isolation, observability and recovery. Those questions cannot be answered by a general instruction to write clean code.

An indiscriminate enterprise review creates a different problem. SOC 2 controls, HIPAA safeguards, PCI-DSS requirements, distributed tracing and single sign-on federation do not all apply equally to every project. A reviewer needs to establish what the system actually does before deciding which controls matter. A checklist can organize that investigation; it cannot establish regulatory compliance by accumulating ticks.

AI coding agents can produce substantial changes without the reviewer having followed every decision. The review process needs to keep pace: choose the right work, investigate it systematically, and leave findings the next engineer can act on.

Five dimensions, then the risk overrides

The codebase review skill starts with five dimensions. Its assessment assigns one point for each listed characteristic that applies:

  • Scale and users: developer count and user-count thresholds.
  • Business criticality: production use, the cost of downtime, contractual service levels and the consequences of a breach.
  • Data sensitivity: accounts, personal information, financial or health data, and regulated data.
  • Architecture complexity: services, databases, data stores and third-party integrations.
  • Operational requirements: uptime targets, an operations team and on-call coverage.

The score selects one of four baseline tiers:

ScoreTierStarting review depth
0–4EssentialBasic structure, build, tests, input handling and credential hygiene, including prototypes and internal tools
5–10StandardAdds depth in testing, performance and reliability for team projects and production applications
11–18EnterpriseAdds concerns such as tenant isolation, identity integration, observability and applicable organizational obligations
19+Mission-CriticalIncludes all tier-labeled rows in the source checklist for high-consequence systems

I add an applicability pass to the runbook's tier selection: identify consequential data and actions, then include the checks they require regardless of tier. The source places Git-history credential checks at Enterprise and tenant isolation at Tier 3 or above; a low-scoring tool can need either today. A tier should stop irrelevant work from crowding the review, without letting an important path escape examination.

Review depth includes how a question is answered. Finding an authorization function is an initial inspection. Following the protected operation through its callers, checking where permissions are enforced, and testing an unauthorized attempt provide different evidence. A reviewer should record which of those happened. Likewise, finding a timeout setting does not establish that every network path uses it.

The breadth to choose from

The source skill has 16 review categories, including architecture, code quality, testing, security and deployment. At the source revision linked here, its detailed checklist contains 273 tier-labeled checkbox rows, plus prose addenda.

  • Secrets and credentials cover repository history, CI/CD configuration and preventive controls as well as hardcoded passwords.
  • AI and human maintainability asks whether both engineers and coding assistants can understand and modify the code.
  • Multi-tenancy covers data isolation, configuration and the tenant lifecycle.
  • Licensing and legal checks cover dependency compatibility, attribution and code provenance.
  • The open-source addendum covers community documentation, security policies, contribution workflows and project health.
  • Industry addenda identify concerns for financial services, healthcare, e-commerce and government work.

The quick-review section offers a fifteen-minute triage timebox for security, code health and operations, using 14 listed items. Use it to decide where to investigate next. Consequential questions left unanswered still belong in the review.

A small tool with a consequential credential

Consider a fictional internal tool whose assessment selects Essential. It uses a credential to call a service that can change stored records. In the teaching workbook, the baseline requires BUILD and TESTS; declaring credential_use adds AUTH_BOUNDARY and SECRET_HISTORY. These identifiers represent review tasks for the exercise, not the full runbook.

The reviewer now needs to determine which operations the credential permits, where the application limits their use, and what the repository history contains. If access to the relevant history is unavailable, a clean search of today's files cannot answer the historical question. The report must retain the gap:

Required taskFictional reported stateEvidence boundary
BuildpassA fixture build-result label is supplied
TestspassA fixture test-result label is supplied
Authorization boundarypassA fixture review label is supplied
Secret historyunknownNo history-review evidence is supplied

The workbook returns coverage_incomplete, with SECRET_HISTORY as the blocking task. Recording a pass or fail for that row, with a nonempty evidence label, completes this exercise's accounting, but the program cannot authenticate that label or determine whether the review was sound. Every result keeps real_security_verified false. A recorded failure can also be fully accounted for while still requiring a fix.

A second fictional case adds tenant data. It requires TENANT_ISOLATION even at the same baseline tier. In a real system, that review could need to follow tenant identity through queries, caches and background jobs and test an attempt to cross the boundary. Leaving the task unreviewed remains visible; it does not become a pass because the other rows are complete.

An optional single sign-on check can instead be marked not applicable, with a reason such as the absence of any federated-login path. Exclusions belong in their own count. The workbook rejects duplicate control IDs and pass rows with empty evidence, and refuses to exclude a required control merely because the baseline tier is low. It cannot detect a risk the human left out of the initial inventory.

Download the fictional review-selection workbook. It contains the fixed requirements, Python validator, tests and a review worksheet. From the extracted directory:

python3 -B -m unittest -v test_workbook.py
python3 -B workbook.py coverage.json

The second command deliberately exits 2 for the incomplete example above. The README documents the selection, complete-accounting, exclusion and invalid-record cases. Tests also run deliberately faulty variants that omit risk additions, count unknowns or exclusions as passes, ignore missing rows, accept empty evidence, confuse selection with review, or overwrite duplicate IDs. This is a local accounting exercise; no coding agent or real repository is reviewed by it.

Designed for agentic review

I built the original runbook working with Claude Code. It fits synthesis coding: software work directed and judged by a human with AI assistance. The wider synthesis engineering practice carries those decisions into team and project work.

Clear, evaluable checks give an agent concrete work. “No O(n^2) in hot paths” directs it to inspect actual data sizes and execution paths. “All network calls have timeouts configured” calls for an inventory of calls and their effective behavior. Give the agent the codebase and the agreed scope, and have it work through those questions systematically. Where source inspection leaves a question open, ask for the test or runtime evidence that would answer it.

I want findings with severity, file paths, line numbers and concrete remediation, along with the strengths of what the team built. For each required check, the report should distinguish pass, fail, unknown and not reviewed. Non-applicability needs its own reason. A report can then show the engineer where to act and the lead which consequential questions remain open.

That direction dynamic is central to synthesis coding: the human decides what matters, the agent does the systematic checking, and the human interprets and prioritizes the findings. The agentic approach makes it practical to cover hundreds of checks against a real codebase in minutes instead of days. The checklist also works without AI: print it, work through it yourself, or use it in a team review session.

  1. Assess the project. Confirm the branch and commit being reviewed, score the five dimensions, and record the baseline tier and risk-driven additions. State which directories or evidence sources are excluded and why.
  2. Run the selected review. Attach evidence to findings and completed checks. Keep unavailable evidence and unexamined paths visible.
  3. Act on findings. Use the skill's concise Tier 1–2 report or its more detailed Tier 3–4 format. Prioritize consequential defects and assign the remaining investigation as well as the fixes.

Four practical prompts

These are four prompt patterns I have used with Claude Code, revised here to include risk overrides and coverage records.

Assess the tier before choosing the work

Read the codebase review runbook at https://rajiv.com/open-source/codebase-review-runbook. Assess this project against its Project Complexity Assessment. Score each dimension, cite the project evidence and recommend a baseline tier. Identify consequential data, credentials and actions that require checks above that tier. State unknowns and proposed exclusions. I'll confirm or override the scope before the review proceeds.

The assistant may not know the roadmap. A prototype might handle personally identifiable information next quarter. The human also needs to identify consequential uses already present, even if they affect only a few users.

Review at a specified baseline tier

Using the codebase review runbook, review this project at Tier 2 (Standard), plus the agreed risk-driven additions. For each category, report findings with severity, file paths, line numbers, evidence and remediation. Account for every selected check as pass, fail, unknown or not reviewed. List exclusions separately with reasons. Do not omit an applicable higher-tier check solely because its tier is above 2.

Explicit scope keeps irrelevant enterprise requirements from crowding the report while preserving the checks the project's actual risks require.

Follow one category deeply

Using the security section of the codebase review runbook, review this project's selected Tier 2 controls and security risk overrides. Pay special attention to hardcoded secrets, .env files tracked in Git, and AI tool configuration files such as .cursorrules, .cursor/ and .aider* that might expose infrastructure details. State which files, history and runtime evidence you examined. Keep secrets out of the report; identify their locations and needed remediation without reproducing their values. Mark inaccessible evidence and unreviewed paths explicitly.

When the concern is known, a focused review can follow it through the relevant code and configuration instead of treating breadth as the objective.

Compare findings over time

Use review-findings-jan.md as the prior findings record. Repeat the agreed review at Tier 2 plus its risk overrides, recording any change in scope. Keep each prior finding's ID and classify it as fixed, partially fixed, still present or worse, citing current evidence. If evidence is insufficient, mark the state unknown. Give new findings new IDs and label them new. Summarize resolved risks, regressions and remaining unknowns at the top.

Engineering leaders should make delta reviews a regular practice. They show whether consequential problems are being fixed, recurring or getting worse. Stable identities keep a renamed finding from appearing to be a fix, while unknowns show where more investigation is needed.

What I learned using it

February 2026 update.

After releasing the runbook, I used it on real engagements: reviewing a security startup's full-stack platform and upgrading three prototype repositories to production quality. The latter meant working builds, documentation, better structure and meaningful tests so engineers could take over; integration testing with the domain expert remained the next step. The tiering system held up in both cases. The reviews surfaced subtle cryptographic choices, AI tool configuration files exposing infrastructure details, and tests that looked comprehensive but only verified imports. The skill caught things that would have slipped past a manual review.

On codebases I owned, the review categories unexpectedly became an upgrade playbook: build, document, restructure and test, ordered around the work's dependencies. In the prototype upgrades, I wrote tests that captured existing behavior before restructuring. The assessment had already identified the work; its findings became the execution plan.

I described the advisory-review lessons in What reviewing real codebases taught me about code review and the upgrade approach in When a code review runbook becomes an upgrade playbook. These experiences informed the runbook's v2.1 update, which added branch checks, behavioral test review and delta reporting.

Use and improve the runbook

The Enterprise-Grade Codebase Review Skill is also available through the runbook page on rajiv.com. Its source declares CC0 1.0; that license applies to the skill. It is part of the synthesis-skills project.

Use it on a codebase you know, and suggest improvements through issues or pull requests. If you find a missing or mistiered check, include the system characteristic that makes it necessary and the evidence a reviewer should obtain.

This article is part of the synthesis coding series.