What ETL Pipelines Taught Me About AI Agent Reliability

An AI agent that calls tools, reads results, and decides what to do next is a pipeline. It ingests an input, transforms it across several stages, and produces an output that something downstream depends on. The fact that one of those stages is a language model rather than a Python function does not change the shape of the system. It changes only how unpredictable the stages are — and unpredictable stages are precisely the problem that data engineering spent two decades learning to contain.

I have built data pipelines for fifteen years, for organisations where a single bad nightly load could misreport revenue to a board the next morning. The failures were rarely caused by a clever bug. They were caused by a pipeline that assumed its inputs were clean, ran without checking its own output, and discovered the damage three stages later when the numbers no longer added up. Agentic systems are now failing in exactly the same ways, and the people building them are reaching for prompt tweaks when they should be reaching for pipeline engineering.

AI agent reliability is a property of the system surrounding the model, not a property of the model itself.

The same model produces dependable or unreliable behaviour depending entirely on what is engineered around it. That is good news, because it means reliability is buildable rather than something you wait for a vendor to ship.

Idempotency: Re-Running Must Not Corrupt State

The first rule of a serious ETL job is that running it twice does not double the data. If a nightly load is interrupted halfway and restarted, it must converge on the same end state, not append a second copy of every row. This property — idempotency — is what makes recovery possible at all. Without it, every failure becomes a manual cleanup before you can even retry.

Agents violate this constantly. An agent that retries a failed step after a timeout will happily send the email twice, create the duplicate ticket, or charge the card again, because the side effect was not designed to be repeatable. The model did nothing wrong. The system around it had no notion that the action might run more than once.

The fix is the same one data engineers already use: make each side-effecting step idempotent by design. Use an idempotency key on external calls. Check whether the ticket already exists before creating it. Write to a staging area first and promote atomically. The point is that retries are inevitable, so the system must be safe to retry. This is the same discipline that governs resilient data ingestion in production ETL, applied to a new kind of stage.

Validation Gates Between Stages

A well-built pipeline validates the output of each stage before the next stage is allowed to consume it. A row that fails a type check, a null where a value is mandatory, a total that falls outside a plausible range — these are caught at the boundary, not three transforms downstream where the cause is unrecognisable.

A validation gate turns a silent failure that propagates into a loud failure that is contained at the point it occurred.

Agentic workflows need the same gates, and need them more, because a language model can produce output that is syntactically perfect and semantically wrong. A hallucinated order ID looks exactly like a real one. A malformed tool call that the model is confident about will sail through any stage that does not check it. Put a validation gate after every model step: does the output parse, does it satisfy the contract the next step expects, is it within bounds that make sense. If it fails the gate, the pipeline stops or quarantines — it does not pass garbage forward.

Schema Contracts at Boundaries

The most durable lesson from ETL is that boundaries need contracts. When two systems exchange data, the structure of that exchange must be declared explicitly and enforced, not assumed. Schema-on-read sounds flexible until an upstream provider quietly renames a field and every downstream consumer breaks at once.

Agents pass data across boundaries on every step: model to tool, tool to model, agent to agent. Each of those boundaries deserves a contract. Define the schema of every tool's input and output. Validate against it. When a model is asked to produce structured data, constrain it to a schema and reject output that does not conform rather than parsing it optimistically. The discipline of treating each boundary as a typed contract is what keeps a multi-step agent from degrading into a chain of hopeful string-parsing. The same thinking underpins why data lakes need structure before they are AI-ready: uncontracted data is a liability whether a model is reading it or writing it.

Observability and Lineage: Trace What the Agent Did and Why

When a pipeline produces a wrong number, the first question is always the same: where did this value come from. Data lineage answers it — every output traces back through the transforms and sources that produced it. Without lineage, debugging a data error is archaeology.

Most agent deployments have no lineage at all. They log the final answer and nothing about the path taken to reach it. When the agent does something wrong, there is no record of which tool returned what, which intermediate decision branched where, or what context the model held at each step. You are left guessing at a non-deterministic process from its output alone.

An agent step that cannot be traced after the fact cannot be debugged, only re-rolled and hoped over.

Instrument the pipeline the way you would instrument an ETL DAG. Log every tool call with its inputs and outputs. Record the model's intermediate reasoning where the task allows it. Assign each run a trace ID that threads through every stage. The goal is that when an agent misbehaves, you can reconstruct exactly what happened rather than reproducing it by luck.

Bounded Retries, Backoff, and Dead Letters

Retries are necessary and dangerous in equal measure. A transient failure — a rate limit, a network blip, a momentarily overloaded service — deserves a retry. A deterministic failure does not; retrying it just burns budget and hammers the failing dependency.

I learned this concretely from the deploy script that publishes this site. An over-eager retry loop against a web application firewall did not recover from the block — it amplified it, sending request after request into a rule that was never going to relent, until the host treated the whole pattern as an attack. The fix was bounded retries with exponential backoff and a hard ceiling: try a few times, wait longer between each attempt, then stop and surface the failure. Agents calling flaky tools need exactly this. Unbounded retry loops are how an agent quietly spends a fortune in tokens achieving nothing.

What happens to the work that fails every retry is the part most agent systems forget. Message-driven data systems solved this with the dead-letter queue: an input that cannot be processed is routed to a separate queue, the pipeline keeps moving on everything else, and a human gets a clear trail to inspect later. Agentic workflows need the same escape valve. The record the agent cannot handle should be isolated, not allowed to block the queue or trigger an infinite loop.

Fail Loud, Not Silent

The most expensive failure mode in data engineering is not the job that crashes. It is the job that finishes successfully and produces wrong numbers. A crash gets noticed within minutes. Plausible-but-wrong output gets shipped to a dashboard, trusted, and acted on for weeks before anyone realises the figures never reconciled.

Language models are a machine for producing plausible-but-wrong output. That is not a flaw to be prompted away; it is the nature of the technology. So the system must be built to fail loud. When a validation gate fails, when a contract is violated, when retries exhaust, the pipeline must halt or quarantine and raise the alarm — not paper over the gap with a confident-sounding answer. An agent that returns a fabricated result rather than admitting it could not complete the task has failed silently, which is the worst way to fail.

This is also the discipline that connects reliability to governance. Enforcing that an agent stops at a failed check rather than improvising past it is, structurally, the same problem as enforcing architectural decisions before code is generated: a constraint only matters if the system cannot quietly ignore it.

Agents Are Pipelines. Engineer Them Like Pipelines.

None of this is new. Idempotency, validation, schema contracts, lineage, bounded retries, dead letters, failing loud — these are the standard equipment of any data engineer who has run pipelines in production long enough to have been burned. The novelty of agents has obscured how familiar their reliability problems are. The model is a new kind of stage, with a wider failure distribution than a SQL transform, but it sits inside the same pipeline shape and yields to the same engineering.

The teams getting reliable behaviour out of agents are not the ones with the best prompts. They are the ones who stopped treating the agent as an oracle and started treating it as a pipeline with one unusually unpredictable stage — and engineered the rest of the system to contain it. The work of reviewing AI-assisted output at scale gets dramatically easier when the pipeline already fails loud at the boundary rather than passing plausible garbage forward.

Reliability was never going to arrive in a model release. It arrives the same way it always has: in the engineering around the unpredictable part.