Data Format Contracts for AI Pipelines

Working with Claude Code, I built a pipeline with three services: ingestion parsed logs, transformation normalized the data, and analytics aggregated the results. Each worked correctly in isolation. At their boundaries, they disagreed.

The ingestion service emitted 2025-02-27T22:36:03Z. The analytics service expected Unix timestamps in milliseconds. The transformation service passed the value through unchanged, assuming the formats were consistent. Aggregations fell into nonsensical time buckets; I noticed when the dashboards showed impossible patterns.

I directed Claude to add a data-format contract to the ingestion service’s CLAUDE.md and a matching input contract to the consumer’s. We needed to specify Unix milliseconds as an integer, conversion at ingestion, and what the consumer would do with malformed input. Writing that agreement resolves an ambiguity. The conversion and validation code still have to implement it.

For developers building pipelines with coding assistants, a data contract makes the agreement explicit: field names, representations, units, allowed values and failure behavior. The implementation must then enforce that agreement at the boundary. A document can tell an assistant what to build; a test can show whether the resulting producer and consumer agree.

The complete Python example uses a constructed event record. It includes a converter, a validating consumer, literal expected output and tests you can run without extra packages.

Start with a value you can verify

The correct conversion is:

2025-02-27T22:36:03Z
       1740695763 Unix seconds
    1740695763000 Unix milliseconds

An earlier version of this article paired that ISO timestamp with 1740693363000. That number represents 2025-02-27T21:56:03Z, forty minutes earlier. Independent Python and JavaScript calculations confirm the corrected pair.

A validator can require an integer in the expected range. Both numbers satisfy those conditions. Only a test that knows the intended instant can identify which one is correct.

Keep this pair as a literal fixture:

{"event_id":"evt_001","occurred_at_ms":1740695763000}

If the converter under test also generates the expected number, an arithmetic error can appear on both sides and pass. The example verifies the literal using Python's calendar.timegm, independently of the converter's elapsed-time arithmetic, and checks the reverse conversion too.

Give one component ownership of conversion

The producer accepts a source record with an offset timestamp:

{"event_id":"evt_001","occurred_at":"2025-02-27T22:36:03Z"}

It owns parsing and conversion. The consumer receives occurred_at_ms and validates that representation. It does not guess whether a numeric input means seconds, milliseconds or something else.

For this example, the downstream contract is:

PartRule
FieldsExactly event_id and occurred_at_ms; both required
Event IDNonempty ASCII letters, digits, underscore or hyphen
Timestamp representationJSON integer token, measured in Unix milliseconds
Accepted instantsJanuary 1, 2000 inclusive through January 1, 2100 exclusive, in UTC
Missing or malformed valueReject the record; do not fill a default or coerce its type
Conversion ownerProducer, before emitting the downstream record
Consumer behaviorValidate before using the timestamp

The date range is a chosen constraint for this teaching example. It makes a seconds-versus-milliseconds error detectable: 1740695763, interpreted as milliseconds, falls well before the permitted range. A system processing older history or future schedules needs its own range. A number alone does not carry its unit.

The producer requires seconds and a known offset, permits up to three fractional digits, and rejects missing offsets, invalid dates and unrecognized forms. In this example, -00:00 is rejected as an unknown offset. Those policies belong beside the conversion code so an assistant does not silently choose different ones.

Make the conversion executable

After parsing and validating the source timestamp, the converter uses an aware UTC datetime. Python's documentation distinguishes these from naive datetimes, which several operations interpret as local time. The datetime documentation explains that behavior.

The conversion in the download uses integer arithmetic:

delta = instant - EPOCH
milliseconds = (
    delta.days * 86400000
    + delta.seconds * 1000
    + delta.microseconds // 1000
)

Because the source contract permits at most millisecond precision, dividing the microseconds by 1,000 discards no accepted information. A contract accepting finer precision would need an explicit rounding or rejection policy.

The producer validates its output before emitting it. The consumer applies the same field and type checks at its own entry point. That second check matters when another producer, a manual import or an older process sends data directly to the consumer.

Python's Boolean type is a subtype of integer. The example therefore uses type(value) is int for the JSON integer requirement. It rejects true, null, a quoted number and a floating-point representation such as 1740695763000.0. The interface does not silently convert any of them.

Exercise both ends of the interface

From the extracted example folder, run:

python3 -B producer.py input.json > event.json
python3 -B consumer.py event.json
python3 -B test_contract.py

The producer emits the literal millisecond record above. The consumer validates it and prints a UTC representation so you can inspect its meaning:

{"event_id": "evt_001", "occurred_at": "2025-02-27T22:36:03.000Z"}

All ten tests passed in the local run. The producer-to-consumer command chain was exercised in UTC, New York and Apia. Equivalent explicit offsets produce the same instant, fractional milliseconds survive, and the lower and upper range boundaries behave as declared.

Input to a boundaryRequired outcome
Consumer receives Unix secondsReject as outside the permitted millisecond range
Consumer receives an ISO string in the millisecond fieldReject the type
Producer receives an unzoned or impossible timestampReject before emitting a record
Either side receives missing or extra fieldsReject the shape
Either side receives duplicate JSON keysReject the ambiguous record
Consumer receives the original forty-minute errorAccept its valid format; the semantic fixture must still fail

That final test passes only if the consumer accepts the old number as well formed and decodes it to the wrong instant.

These programs emit no result JSON on rejection, write a reason to stderr and exit with status 1. In a service, choose an equally concrete error path before adding side effects: for example, reject the request or send the invalid record to a reviewed quarantine workflow. Quietly replacing a missing timestamp with the current time would invent event data.

Put the contract in the coding task

Store the readable contract, implementation and fixtures together. Put a pointer in the instruction file your coding client uses, and explicitly name the contract in the task. Instruction discovery varies by client and working directory; a file's existence does not establish that a session read or followed it.

A useful task instruction is:

Read the event contract and its literal fixtures. Implement conversion at the producer and validation at the consumer. Show the conversion for the stated timestamp, exercise the real producer-to-consumer path, and run the invalid-input cases. If a requirement is ambiguous, identify the unresolved choice before changing the expected output.

Ask for the executed commands, results and changed code with the implementation. If the task changes units or accepted input, review the producer, consumer, contract and fixtures together. A contract edit alone leaves running code untouched; a code edit alone can leave the next maintainer working from the wrong specification.

Pagination: keep the API and client in the same agreement

I encountered the same kind of drift in a REST API with paginated results. The API and client evolved separately. Claude added response fields to the API without corresponding handling in the client, and query parameters to the client without support in the API.

Documenting both sides together gave me a place to review those changes. Field names alone were not enough: the contract also needed types, defaults, limits and the meaning of the last page. A page-number interface can start with this agreement:

FieldDirectionType and rule
pageRequestRequired integer, starting at 1
per_pageRequestOptional integer; default 20, maximum 100
totalResponseInteger count of items across all pages

A cursor-based interface also needs a terminal convention, such as next_cursor being a string while more results exist and null on the last page. If an API supports both pages and cursors, define how the modes relate and which parameters can be combined. Listing both fields without that relationship leaves a decision for the client to guess.

My original sketch made another deliberate distinction: clients ignored unrecognized response fields, while the API rejected unknown request parameters with status 400. Those are contract choices, not properties that appear automatically when a table is written. The implementation and its tests must agree with them. A useful acceptance case sends an unsupported query parameter; another follows the final page and checks the declared terminal response.

A limitation can change whether the implementation is acceptable

When I asked Claude to add rate limiting to my API gateway, it worked for a particular definition of “worked.” The sliding-window limiter I was using could briefly exceed limits during high bursts. That surprised me in load testing because I had expected hard limits.

The limitation I documented was a burst tolerance of up to 10%, with gradual decay rather than a hard reset at a window boundary. That figure belongs to the implementation in my account; it is not a property of every sliding-window limiter. The algorithm’s name alone does not tell a caller its worst-case behavior.

I also suggested setting the configured limit 10% lower. That is not a substitute for establishing the bound a hard-limit requirement needs. A tolerance observed in a load test is not a guarantee over every traffic pattern. If the requirement is a hard cap, the implementation must enforce that cap and the tests must challenge its boundary behavior. If a bounded burst is acceptable, put the allowance, time window and rejection behavior in the contract so callers can make that decision deliberately.

Configuration has callers too

Configuration conventions can be hidden in one service’s code just as easily as a timestamp format. An assistant creating the next component may choose a different environment-variable name, Boolean representation or feature-flag key unless those choices are available in its working context.

The configuration example in my original account used these conventions:

## Configuration Contract

Environment variables:
- PREFIX: SERVICE_NAME_ (e.g., ANALYTICS_DB_HOST)
- Format: UPPER_SNAKE_CASE
- Booleans: "true"/"false" strings, not 1/0

Feature flags:
- Format: feature.{domain}.{name}
- Values: boolean only
- Source: LaunchDarkly, cached locally

These are application choices. The Boolean-only rule describes that feature-flag contract, not every kind of value its provider can support. Keep the schema at the repository root or another clearly referenced location, and put the key conventions or a pointer in the relevant instruction file. Then check a value at the boundary where it is read: the environment string "false" must not become true merely because it is nonempty.

The time spent making an interface explicit

Individual format mismatches I debugged consumed 30–60 minutes of investigation. A wrong timestamp can travel through several services before a dashboard makes the error obvious. Against that experience, spending ten minutes spelling out an interface was an easy investment to justify.

For each handoff, I want five things written down: exact field names; formats and units; required and optional values; validation and failure behavior; and limitations. Link the producer’s contract to the consumer’s, keep both beside the code, and update the contract and its checks when behavior changes.

AI-assisted development adds another participant to the interface: the assistant interpreting the requirements and generating code. It cannot infer an unwritten local convention reliably from the fact that another service happens to follow it. Making the agreement explicit gives both human maintainers and the assistant something concrete to work from. Exercising both ends tells us whether the software honors it.

The same discipline applies beyond timestamps. For each interface, choose one representative valid exchange and one plausible disagreement, then make both executable.


This article is part of the synthesis coding series. For related pipeline experience, see RAG Architecture Lessons from Practice.