The Problem With All-Or-Nothing Pipelines
Most ingestion pipelines are built with an unspoken assumption: the upstream API either works or it doesn't. In practice, that's rarely how third-party APIs fail. A batch of 500 records comes back with 480 successes, 15 validation errors, and 5 timeouts. The pipeline wasn't written to handle that shape of failure, so it does one of two bad things. It either aborts the whole batch and reprocesses everything from scratch, duplicating the 480 good records, or it silently drops the 20 bad ones and moves on, and nobody notices until a customer asks why their order from three weeks ago never synced.
Neither outcome is acceptable once a pipeline is feeding anything that matters, whether that's billing data, inventory counts, or customer records. Partial failure isn't an edge case you patch in later. It's the normal operating condition of any pipeline that talks to more than one external system, and the architecture has to assume it from the first design pass.
This matters more as integrations multiply. A pipeline talking to one well-behaved API can sometimes get away with naive all-or-nothing handling for years. A pipeline talking to five vendor APIs, each with its own failure quirks, rate limits, and maintenance windows, will hit partial failure constantly, and the teams that treat it as rare end up firefighting the same category of incident every few weeks under a different name.
Photo by Nic Wood on Pexels
What "Partial Failure" Actually Looks Like In Practice
Partial failure shows up in a handful of recurring shapes, and it helps to name them before writing any handling code. There's the mixed-batch response, where a single API call returns a status per record instead of one status for the whole request. There's the mid-batch timeout, where record 340 of 500 never gets a response and you don't know if it was processed or not. There's the rate-limit throttle, where the API accepts the first 200 records and then starts returning 429s for the rest.
Each of these needs a different response. A mixed-batch response needs per-record bookkeeping. A mid-batch timeout needs an idempotent retry so you don't double-process record 340. A rate-limit throttle needs backoff, not a hard failure. Treating all three as "the pipeline broke, alert and stop" throws away information you already have about exactly what went wrong and where.
There's a fourth shape worth naming on its own: the schema drift failure, where the API technically responds successfully but the payload shape has changed since the integration was written. A field that used to be a string is now an object, or a previously required field is missing. This one is the most dangerous because it often doesn't look like a failure at all until a downstream consumer chokes on the malformed record days later. Validating incoming payloads against a formal JSON Schema at the pipeline's edge turns this silent failure into a loud, catchable one, before the bad shape ever reaches the write layer.
Handling Rate Limits Without Making Things Worse
Rate limits deserve their own section because they're the one failure mode a pipeline can actively cause rather than just react to. An ingestion job that fires requests as fast as the network allows will eventually hit a ceiling on any API with real usage tiers, and how the pipeline responds to that ceiling determines whether the next hour goes smoothly or turns into a cascading backoff storm across every worker in the fleet.
The fix is straightforward in principle: read the rate-limit headers the API actually sends (X-RateLimit-Remaining, Retry-After, or their vendor-specific equivalents) and throttle proactively before hitting the wall, rather than reactively after a 429 comes back. A token-bucket limiter on the client side, sized comfortably under the API's published ceiling, keeps the pipeline from ever triggering the vendor's own penalty box in the first place.
Classify Failures Before You Try To Handle Them
Build a small taxonomy up front: transient (network blip, 5xx, timeout), retryable-with-backoff (429, 503 with a Retry-After header), and permanent (400 validation error, 404 on a resource that was deleted upstream). Every failure your pipeline sees should map to exactly one of these categories, and the mapping should live in one place, not be re-derived ad hoc in every handler.
This classification decides everything downstream. Transient and retryable failures go back into a queue. Permanent failures go into a dead-letter table for a human or a secondary process to review. Mixing these two paths, retrying a permanent 400 forever or dead-lettering a network blip that would have succeeded on retry, is one of the most common reasons pipelines either loop forever or silently lose good data.
Build An Idempotent Write Layer First
Before you write any retry logic, make sure re-processing the same record twice is safe. This usually means an idempotency key, either one the upstream API gives you or one you generate deterministically from the record's natural identity (order ID plus a version or timestamp works for most cases). The write layer checks that key before committing, and a duplicate delivery becomes a no-op instead of a duplicate row.
This single piece of infrastructure is what makes everything else in this article safe to build. Without it, every retry is a gamble on whether you're fixing a partial failure or creating a new one. Wikipedia's overview of idempotence is a good shared reference to point teammates at when this concept needs explaining outside the pipeline itself.
Isolate Bad Records Instead Of Blocking The Batch
Once failures are classified, the pipeline should never let one bad record block the 479 good ones sitting next to it in the same batch. Process records independently where the API allows it, commit successes as they land, and route failures to their appropriate path without waiting for the whole batch to resolve.
For APIs that only support single all-or-nothing batch responses, this means splitting the batch client-side: submit in smaller chunks, or resubmit failed batches as smaller sub-batches with a binary-search-style split until you isolate the specific bad records. It's more API calls, but it turns "the batch failed" into "these three records failed, here's why," which is the difference between a five-minute fix and a two-hour investigation.
Retry Logic That Doesn't Make Things Worse
Retries need jittered exponential backoff, a maximum attempt count, and a circuit breaker that stops hammering an API that's clearly down instead of retrying into a wall. A pipeline that retries aggressively during an upstream outage can turn a 10-minute API blip into a self-inflicted rate-limit ban that takes a day to lift.
Respect a Retry-After header when the API sends one. When it doesn't, a starting backoff of one or two seconds, doubling up to a capped ceiling, with some random jitter added so a fleet of workers doesn't retry in lockstep, covers the overwhelming majority of transient failures without adding meaningful load to a service that's already struggling. The general pattern is well documented as exponential backoff, and it's worth implementing it once as a shared utility rather than re-deriving the constants in every integration.
Give Bad Records A Real Home, Not A Log Line
A dead-letter queue isn't optional infrastructure for a pipeline handling partial failures, it's the mechanism that makes it safe to keep processing the rest of the batch. Every permanently-failed record needs to land somewhere queryable, with the original payload, the failure reason, and a timestamp, so someone can look at "why did these 15 records fail" as a five-minute query instead of a grep through application logs.
The dead-letter table should be cheap to reprocess from. If the upstream data changes (a customer fixes their address, a vendor corrects a SKU), replaying a dead-lettered record through the same idempotent write path should just work, without a special-cased backfill script written under pressure.
Resist the urge to build a bespoke UI for the dead-letter queue on day one. A well-indexed table that a support engineer or on-call developer can query directly, paired with a Slack or email digest summarizing what landed there each day, covers most teams' needs for months. Build the dashboard once the query patterns are clear, not before.
Monitoring That Catches Drift, Not Just Outages
A pipeline that's technically running but silently dropping 2% of every batch is more dangerous than one that's obviously down, because nobody pages on a dashboard that's still green. Track the ratio of successes to failures per run, not just whether the run completed, and alert on that ratio drifting outside its normal band rather than on a hard failure count.
"The pipelines that cause the most damage are the ones that never throw an error, they just quietly process 98% of the truth instead of 100% of it. Alerting on completion instead of accuracy is how that goes unnoticed for months." - Dennis Traina, founder of 137Foundry
Pairing a success-ratio alert with a periodic reconciliation job, one that compares record counts or checksums between source and destination on a schedule, catches the drift that per-run monitoring alone tends to miss.
Testing Partial-Failure Scenarios Before Production
Most teams test the happy path and the total-outage path, and skip everything in between. A pipeline that handles partial failure well needs test fixtures that simulate a mixed-status batch response, a mid-batch timeout, and a rate-limit throttle specifically, not just a generic "API returns 500" mock. Tools like Postman make it straightforward to script these mixed-response scenarios against a mock server before the pipeline ever touches a real upstream API, and an OpenAPI contract for the integration gives you a shared source of truth for what a "success" and "failure" response actually look like, which pays off the first time someone updates the upstream API without warning.
Run these scenarios in staging on a schedule, not just once at launch. Upstream APIs change their failure behavior more often than their success behavior, and a partial-failure handling path that isn't exercised regularly tends to quietly rot.
Chaos-style testing pays off here too, even in a lightweight form. Occasionally injecting an artificial timeout or a malformed payload into a staging run, on purpose, is one of the few reliable ways to confirm the dead-letter path and the alerting actually fire the way the runbook assumes they do, instead of discovering the gap during a real incident.
Putting It Together: A Practical Checklist
None of this requires a rewrite to implement well. Start with the idempotent write layer, since everything else depends on it being safe to retry. Add failure classification next, so transient and permanent failures stop being handled the same way. Then build the dead-letter path, wire up drift monitoring, and only after those are solid, invest in the batch-splitting logic for APIs that don't give you per-record granularity natively.
If your team is weighing whether to build this in-house or bring in outside help to get the architecture right the first time, 137Foundry's data integration work focuses specifically on pipelines like this, and the broader services page covers where that fits alongside the rest of what the team builds. You can also read more about the team's approach on the about page. A pipeline that handles partial failure gracefully isn't a nice-to-have layered on top of a working system. It's what makes the system trustworthy enough to actually rely on.
