Every service you call over a network will eventually fail for no good reason. A load balancer drops a connection mid-handshake, a downstream database hiccups for 400 milliseconds, a DNS lookup times out once in ten thousand tries. None of that means the request was wrong. It means you need a retry strategy that assumes failure is normal, not exceptional.
Most teams start with the wrong instinct: catch the error, wait a fixed second, try again. That works until the day your dependency has a real outage, and every client retries on the same one-second clock, hammering the service the moment it starts to recover. This article collects working retry and backoff snippets in JavaScript, Python, and Go, plus the jitter, budget, and idempotency rules that keep retries from making an outage worse.
Why fixed-delay retries make outages worse
A fixed delay retry looks harmless in a single client. The problem shows up at scale. If a thousand clients all fail at the same moment and all wait exactly one second before retrying, you get a thundering herd hitting the service at the exact instant it's trying to recover.
Photo by Valentin Lacoste on Unsplash
Exponential backoff spreads that load over time instead of concentrating it. Each failed attempt waits longer than the last, so a service under real strain sees a tapering trickle of retries rather than a repeating wave. Jitter, covered a few sections down, is what actually breaks the synchronization between clients.
A JavaScript retry-with-backoff snippet
This wraps fetch with exponential backoff and a maximum attempt count. It only retries on network errors and 5xx responses, since retrying a 4xx just repeats the same mistake.
async function fetchWithBackoff(url, options = {}, maxAttempts = 5) {
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
try {
const res = await fetch(url, options);
if (res.ok || (res.status >= 400 && res.status < 500)) return res;
if (attempt === maxAttempts) return res;
} catch (err) {
if (attempt === maxAttempts) throw err;
}
const delay = Math.min(1000 * 2 ** (attempt - 1), 20000);
await new Promise((r) => setTimeout(r, delay));
}
}Notice the Math.min cap. Without it, attempt six waits 32 seconds, attempt seven waits over a minute, and your caller has long since given up and moved on. A cap of 15 to 30 seconds keeps the tail bounded.
A Python retry-with-backoff snippet
The same shape translates directly to Python, whether you're wrapping requests, httpx, or a database driver's own connection call.
import time
import random
def retry_with_backoff(fn, max_attempts=5, base=1, cap=20):
for attempt in range(1, max_attempts + 1):
try:
return fn()
except (ConnectionError, TimeoutError):
if attempt == max_attempts:
raise
delay = min(base * 2 ** (attempt - 1), cap)
time.sleep(delay + random.uniform(0, delay * 0.3))This version bakes jitter straight into the sleep call rather than treating it as an optional extra. Catching specific exceptions matters too. A bare except Exception will happily retry a bug in your own code, which just delays the failure instead of preventing it.
A Go retry-with-backoff snippet
Go's explicit error handling makes the retry loop easy to reason about, since there's no exception to accidentally swallow.
func retryWithBackoff(fn func() error, maxAttempts int) error {
var err error
for attempt := 1; attempt <= maxAttempts; attempt++ {
if err = fn(); err == nil {
return nil
}
if attempt == maxAttempts {
return err
}
delay := time.Duration(math.Min(float64(time.Second)*math.Pow(2, float64(attempt-1)), float64(20*time.Second)))
jitter := time.Duration(rand.Int63n(int64(delay) / 3))
time.Sleep(delay + jitter)
}
return err
}If you're calling this inside a goroutine pool, pass a context.Context through fn so a caller-side timeout can cancel the retry loop early instead of letting it run its full course after nobody's waiting anymore.
Add jitter or your retries will collide
Backoff alone solves the growing-delay problem. It doesn't solve the synchronization problem, because every client that failed at the same instant is still retrying on the exact same schedule, just a slower one. Jitter breaks that lockstep by adding a random offset to each delay.
Photo by cottonbro studio on Pexels
"Full jitter", picking a random delay anywhere between zero and the computed backoff value, spreads load more evenly than "equal jitter", which only randomizes half the interval. AWS's own architecture guidance popularized full jitter for exactly this reason: it consistently produces less total retry traffic under load than backoff without it. - Dennis Traina, founder of 137Foundry
Either approach beats no jitter at all. The failure mode to avoid is generating your random delay with a seed that's the same across processes, like a container's start time, which quietly recreates the synchronization problem you were trying to fix.
Set a retry budget and honor Retry-After
An unbounded retry loop is a resource leak wearing a resilience costume. Cap the number of attempts, and separately cap the total wall-clock time a single logical operation is allowed to spend retrying, so a slow-failing dependency can't hold a request thread open indefinitely.
Photo by Marek Piwnicki on Pexels
When a response includes a Retry-After header, use it instead of your own backoff schedule. The server is telling you exactly how long it needs, whether that's derived from a rate limit window or a maintenance countdown, and ignoring it in favor of your own guess just adds unnecessary load right when the service asked for a break.
Know what's actually safe to retry
Retrying a GET request that failed is almost always safe, and the HTTP method definitions on MDN spell out which verbs are meant to be safe to repeat. Retrying a POST that creates a resource is not, unless you know the operation is idempotent, meaning running it twice produces the same result as running it once.
The fix is an idempotency key: a unique identifier your client generates once per logical operation and sends with every retry attempt. The server checks whether it's already processed that key and, if so, returns the original result instead of creating a duplicate charge, order, or record. Most payment APIs require this pattern for exactly this reason.
Testing retry logic without waiting real time
Sleeping for real seconds in a test suite is how a five-minute CI run becomes a fifteen-minute one. Inject the sleep function as a dependency, then swap in a fake clock during tests that advances instantly instead of blocking.
Photo by Tima Miroshnichenko on Pexels
def test_retries_three_times_then_raises():
calls = {"count": 0}
def flaky():
calls["count"] += 1
raise ConnectionError("nope")
with pytest.raises(ConnectionError):
retry_with_backoff(flaky, max_attempts=3, sleep_fn=lambda s: None)
assert calls["count"] == 3This also lets you assert on the exact number of attempts and the delay values passed to the mocked sleep function, which is the only reliable way to catch an off-by-one in your backoff math before it ships.
Pairing retries with a circuit breaker
Retry logic and a circuit breaker solve two different problems that people often conflate. Retry logic handles a single request that failed once, on the assumption the next attempt has a reasonable chance of succeeding. A circuit breaker handles the case where a dependency has been failing consistently for a while, and the right move is to stop calling it entirely for a cooldown period rather than keep retrying into a wall.
Without a breaker, a downstream outage turns every upstream caller into a retry storm aimed at a service that's already down, which delays its recovery instead of helping. A basic breaker tracks a rolling count of recent failures, opens once that count crosses a threshold, and rejects calls immediately, with a fast, cheap failure, until a trial request after the cooldown succeeds again.
class CircuitBreaker:
def __init__(self, threshold=5, cooldown=30):
self.failures = 0
self.threshold = threshold
self.cooldown = cooldown
self.opened_at = None
def call(self, fn):
if self.opened_at and time.time() - self.opened_at < self.cooldown:
raise RuntimeError("circuit open")
try:
result = fn()
self.failures = 0
self.opened_at = None
return result
except Exception:
self.failures += 1
if self.failures >= self.threshold:
self.opened_at = time.time()
raiseWrap your retry loop inside the breaker's call, not the other way around, so a single logical operation still gets its configured retry attempts, but once the dependency has clearly gone down, subsequent operations fail fast instead of each spending several seconds working through their own backoff schedule for nothing.
Watching retry rates so problems don't hide behind them
A retry layer that works well has a quiet failure mode of its own: it can mask a real problem long enough that nobody notices until the retries stop being enough to cover for it. If a dependency's error rate creeps from 0.1% to 8% but your retry logic absorbs most of those failures before they reach a user, the underlying degradation can go unnoticed for weeks.
Log every retry attempt with the operation name, attempt number, and outcome, and turn that into a metric your team actually watches, not just a line buried in application logs. A dashboard tracking retry rate per dependency, alongside your normal error rate, surfaces creeping degradation while it's still small enough to investigate calmly instead of during an incident.
Set an alert on retry rate itself, separate from your error rate alert. Error rate measures what users experience after retries; retry rate measures how much work your system is quietly doing to keep that error rate low, and a spike there is often the earliest warning you get.
Wiring this into a real service
None of these snippets are complete production code on their own. They're the core loop you wrap with your own logging, metrics, and circuit breaker so a persistently failing dependency stops being retried at all rather than retried forever at a slower rate.
If you're weighing how much of this to build in-house versus lean on a library your language ecosystem already maintains well, that's usually the right call. The value in understanding the snippets above is knowing what a good library should be doing under the hood, so you can tell when one is cutting corners on jitter or ignoring Retry-After entirely. It's the same reasoning our web development service applies when a client asks whether to build a piece of infrastructure or adopt an existing one, and it's a question worth answering deliberately rather than defaulting to whichever option shows up first in a search.
For a deeper look at how these patterns fit into a larger integration, see 137Foundry's data integration service page, or browse the services hub for the full list of what we build. You can also read more about our approach on the about page.
