I have seen AI generate 200 tests for an API endpoint, all passing, while missing the scenario that caused the production outage: the upstream service returned a 200 status code with an empty body instead of the expected response. The suite tested error codes thoroughly. It did not test that mismatch between status and content.
A test suite that passes is not the same as a correct system.
The missing case matters more than the count. For that endpoint, successful communication had to deliver usable data. A check that stopped at the status code could not establish that requirement. Generating more tests from the same assumption would leave the gap intact.
In synthesis coding, the practice of combining human judgment with AI-assisted software development, systematic quality standards are the second of five pillars. AI-generated code should meet the same standards as human-written code. In practice, the bar should be higher because AI produces more code faster, and the cost of letting bad code through scales with volume. The same "I'll fix it later" reflex that produced one shaky function now produces twenty.
Start with the behavior you need
Correctness, maintainability, security, performance, and current documentation remain requirements. The appropriate evidence depends on the change. An authentication change needs different scrutiny from a wording correction; generated authorship alone does not settle the risk.
Write down the expected behavior before asking an assistant to implement it. For an API client, that includes successful responses and unusable ones. For a shared resource, it may include concurrent access. Authentication failures, rate limiting, data validation, error recovery, and performance under load belong in the strategy when the feature depends on them.
AI can turn those requirements into test cases, suggest omitted scenarios, and implement the checks systematically. The engineer who has been on call at 3 AM knows which failures a polished test suite can miss. Write that operational knowledge into the strategy, then ask the assistant to expand and test it. Together, the engineer and AI can build a suite that is more comprehensive and grounded in how the service actually behaves.
The higher-order skill is designing a testing strategy that leaves no important scenario uncovered. AI makes it practical to apply that strategy across a larger volume of code; engineers judge the completeness of the strategy and the quality of the tests.
For an illustrative order-lookup client:
A successful order lookup must return the requested order's identifier. Define expected outcomes for a valid response, an upstream error, a 200 response with an empty body, malformed JSON, a missing identifier, and a different order's identifier. Explain which requirement each expected result comes from. Then implement the tests against the response parser.
A generated test can repeat the implementation's mistaken assumption, assert only that a call completed, or replace the critical behavior with a mock. Check both the correctness of each important assertion and the suite's missing scenarios.
Check that the test detects the defect
Example prepared September 27, 2026. The downloadable example turns the empty-body lesson into a local response-parser exercise.
For this example, a usable response has status 200 and a JSON object whose order_id is a nonblank string matching the requested order. The parser returns that identifier. An unusable upstream response raises OrderResponseError, an exception defined in the example. The rule comes from this example's consumer requirement; an empty body is not automatically wrong for every HTTP operation.
Response to a request for order-7 | Expected result |
|---|---|
200 with {"order_id":"order-7"} | Return order-7 |
| 200 with an empty body | Raise OrderResponseError |
200 with malformed JSON or no order_id | Raise OrderResponseError |
200 with {"order_id":"order-8"} | Raise OrderResponseError |
| 503, even with an apparently usable body | Raise OrderResponseError |
The empty-body assertion requires an error:
with self.assertRaises(OrderResponseError):
PARSER(200, b"", "order-7")
PARSER selects the implementation under test. The package includes a deliberately faulty alternative that returns None for an empty 200 response. A smoke check that asks only whether the call returned without raising an exception accepts that result. The requirement-based assertion rejects it.
After extracting the ZIP, run these commands from its folder:
python3 weak_smoke.py
python3 test_order_response.py
python3 test_order_response.py --implementation empty-fallback
In the September 27, 2026 local Python 3.12.3 run, the smoke check exited successfully and printed Returned value: None. The strict parser passed all 19 tests. The deliberately faulty parser failed the empty-body test and exited with status 1. That failure is the expected result of the final command.
The suite also checks wrong types, a mismatched identifier, duplicate JSON members, invalid encoding, and the example's body-size limit. It runs without a network service or a model call. It does not establish timeout, retry, authorization, load, or deployed-service behavior; those properties need checks at the layers that implement them.
This is one form of testing the tests: introduce a specific incorrect behavior and confirm that the relevant assertion fails. If the empty-body fallback still passed, adding more successful-response cases would not repair the assertion.
Use coverage to find questions
Statement coverage tracks executed code; branch coverage also tracks whether execution took the possible paths between lines. Coverage.py's documentation demonstrates a function whose statements all execute while one branch remains untested.
An uncovered error path is a reason to investigate. A covered path still needs a meaningful assertion. Even executing every branch in an implementation cannot tell you that the implementation contains every behavior the requirement needs. The response example could have extensive status-code coverage and still omit the rule about usable content.
Choose coverage targets for the code and risks you are managing. Pair them with a list of required behaviors and known failure conditions. A percentage alone should not decide whether a change is ready.
Review the lines and the system
An architectural review asks whether the component belongs where it was added, follows the intended dependency direction, and makes compatible assumptions about its callers. A local review asks whether the condition, return value, error handling, and resource use are correct. Both matter in generated code. Pattern-level review requires deep system understanding: you need to know the architecture well enough to recognize when generated code violates it.
For the response parser, the architectural question is where unusable upstream data becomes a defined application error. The local question is whether an empty body can slip through as None. A design diagram would not catch that return value by itself, and a correct parsing function would not decide how the application should present an upstream failure.
Google's code-review guidance covers design, functionality, individual lines, and test validity. It asks whether tests would fail for incorrect code. That is a useful standard for reviewing generated implementations and generated tests together.
Documentation update prepared October 1, 2026: GitHub describes Copilot code review as reviewing changes and producing comments or suggestions. Its documented limitations include missed problems, false positives, and inaccurate or insecure suggestions. Evaluate each finding against the code and test any proposed repair.
Security work needs the same combination of tools and context. Keep vulnerability scanning, dependency checks, and inspection of sensitive code. Define the assets, exposed entry points, trust boundaries, and allowed data flows, as OWASP's threat-modeling guidance recommends. Include the restrictions on where regulated data may go and the adversaries the service must withstand, including nation-state threats if they are relevant to the system. Ask an assistant to help challenge the model and identify tests, then verify the proposed controls against the actual deployment. A comment saying that an API is internal does not establish that unauthenticated callers cannot reach it.
Make the practice available to the team
Shared testing strategies let the whole team benefit from its strongest engineers’ judgment. Store the strategies, important architectural constraints, and repeatable checks with the project. Use continuous integration to run automated checks, and make responsibility for the remaining review explicit. New team members can see how the team achieves quality without having to infer its standards from inconsistent examples across the codebase.
An engineering director I worked with described the shift: "Our test quality improved dramatically when we moved from 'AI writes some tests' to 'engineers design testing strategy, AI implements it comprehensively.' We find bugs in code review now instead of production."
Keep the decisions behind the practice: which scenarios matter, why the assertions test them, and what the team learned from a missed case. Each incident can improve the next engineer’s testing strategy.
For the next feature, write the expected behavior and failure conditions before generation. Have the assistant map each condition to a test, identify conditions it has not exercised, and explain what each result establishes. Review consequential code and assertions, inspect the integration points, and retain the actual test output with the change. When an incident exposes a missing condition, add the condition and the check that detects it to the shared practice.
Connect quality to understanding and shared knowledge
Expanded May 21, 2026.
Active system understanding includes knowing why the empty-body test exists, which dependency can violate the contract, and how the application responds. Reviewing the requirements and checks together makes that understanding visible to the team.
Transferable knowledge makes that practice usable by the next engineer. A standard that lives only in one engineer’s head or one tool’s private memory cannot guide the team when that person or tool is unavailable. Keep the requirement, assertion, and reason for the check together in files the project owns, so a future change cannot quietly turn a known failure back into a passing test.
Quality in synthesis coding comes from practices that produce reliable systems, whoever or whatever wrote the code.