Every API eventually gets a rate limiter bolted onto it, usually the week after a single misbehaving client or a scraping bot takes down a shared resource. The fix that goes in under pressure is almost always the same: a flat cap of N requests per minute per API key, enforced with a counter that resets on the clock. It stops the immediate fire. It also quietly starts throttling the customer whose integration legitimately needs to sync 500 records in a burst every morning.
That's the core tension in rate limiting: the traffic pattern that looks like abuse and the traffic pattern that looks like a power user doing exactly what they're paying you for often produce the same shape on a graph. A rate limiter that can't tell them apart isn't protecting your backend, it's just moving the pain from your servers onto your best customers.
Why Fixed Request Caps Punish Your Best Users
A fixed window counter, N requests per fixed clock interval, is the easiest rate limiter to build and the easiest one to get wrong. Its failure mode is the boundary problem: a client can send N requests in the last second of one window and another N in the first second of the next, doubling the effective rate right at the edge with no single request ever technically over the limit.
The other failure mode is blunter. A client with a legitimate need for bursty traffic, a nightly batch sync, a webhook replay after an outage, a dashboard that fans out ten calls on page load, gets treated identically to a script hammering your search endpoint. Both hit 429 responses. Only one of them should.
If your current limiter is a fixed window and you're seeing support tickets about "random" throttling from paying customers, this is very likely why. The fix isn't necessarily a higher limit, it's a smarter algorithm underneath the same limit.
Token Bucket vs Sliding Window vs Fixed Window
Three algorithms cover almost every real-world case. A fixed window resets a counter on the clock and is cheap but has the boundary problem above. A sliding window log or sliding window counter tracks requests across a rolling interval instead of a hard reset, smoothing out the edge case at the cost of slightly more bookkeeping per request.
A token bucket is usually the better default for APIs that need to tolerate bursts. Each client has a bucket that refills at a steady rate up to some maximum, and every request costs one token. A client that's been quiet can spend a burst of saved-up tokens all at once, then has to wait for the bucket to refill, which matches how real integrations actually behave far better than a hard per-minute cap does.
Redis is the common backing store for token bucket state across multiple API servers, since it can atomically decrement a counter with a Lua script or INCR/EXPIRE in a single round trip, which matters once you have more than one process enforcing the same limit.
Where to Enforce the Limit: Gateway, Middleware, or Both
Enforcing limits at the edge, in a reverse proxy or API gateway before a request ever reaches application code, is the cheapest place to reject abusive traffic. Nginx and most managed gateways support this natively, and it means a flood of bad requests never costs you a database connection or an application server thread.
Edge-only enforcement has a blind spot, though: it usually can't see application-level context like which authenticated user or plan tier is making the request, only the IP or a header. Application middleware can key on the actual API key, user ID, or billing plan, which is what you need for the "protect the backend but don't punish paying customers" distinction to actually work.
Most production setups end up doing both: a coarse, high-limit IP-based check at the edge to absorb the worst of a DDoS-shaped spike, and a finer, plan-aware check in application middleware for everything that gets past the edge. Neither layer alone gives you the full picture.
Per-User, Per-IP, or Per-API-Key: Choosing the Right Key
The key you rate limit on determines what kind of abuse you actually catch. Limiting by IP address is the bluntest option and breaks down fast behind NAT or corporate proxies, where hundreds of legitimate users can share one address and all get throttled together because of one heavy user on the same connection.
Limiting by API key or authenticated user ID is more precise and is almost always the right primary key for an authenticated API, since it ties the limit to the entity actually responsible for the traffic. For unauthenticated endpoints, like a public search or signup form, IP-based limiting combined with something like a CAPTCHA challenge after repeated hits is usually the pragmatic fallback.
Some APIs layer a third key on top: per-endpoint limits, so a client can make many cheap reads but very few expensive writes or exports in the same window. This matters most for endpoints that trigger real backend cost, a report generation job or a bulk export, where the cost per request is wildly uneven.
Handling Bursts Without Blocking Legitimate Traffic
A well-tuned token bucket already absorbs most legitimate bursts, but there's a second lever worth using: a queue instead of an immediate rejection for requests that are only slightly over budget. Rather than returning a 429 the instant a client exceeds its limit, some APIs hold the request for a short window, a few hundred milliseconds, and retry it against the bucket before giving up.
This trades a little latency for a lot fewer hard failures, and it's particularly effective for internal service-to-service calls where a brief delay is invisible to the end user but a dropped request means a retry storm. It's not free, a queue that grows unbounded under real overload becomes its own outage, so it needs its own cap and timeout.
"The rate limiters that actually hold up in production are the ones tuned against real traffic logs, not a number someone picked in a planning meeting. We've pulled clients out of self-inflicted throttling incidents just by re-keying the limit from IP to API key." - Dennis Traina, founder of 137Foundry
Communicating Limits Back to the Client (Headers and Retry-After)
A rate limiter that fails silently, just dropping or slowing requests without explanation, forces every client integrator to reverse-engineer your limits by trial and error. Standard rate limit headers, X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset, let a well-behaved client back off proactively instead of finding the ceiling the hard way.
When a request does get rejected, the response should be a 429 status code with a Retry-After header telling the client exactly how long to wait. RFC 6585, which defined the 429 status code, exists specifically because HTTP didn't originally have a clean way to say "you're being throttled, not blocked, try again later."
Good documentation matters just as much as good headers. If your API docs don't state the actual limits per plan tier, developers will guess conservatively, under-using your API, or aggressively, hitting it constantly and generating support tickets. Either way you're paying a cost for something a documentation page would have solved.
Rate Limiting Distributed Systems Without a Single Point of Failure
Once your API runs across more than one server, an in-memory counter per process stops working, since each instance sees only its own slice of traffic and a client can effectively multiply their limit by the number of servers behind the load balancer. Centralizing counter state in something like Redis solves this, but now the rate limiter itself is a dependency your API can't function without.
The usual mitigation is to fail open, not closed, when the counter store is unreachable: if Redis is down, let requests through rather than rejecting all traffic, and alert loudly. A rate limiter that takes your whole API down because its own backing store had a blip is a worse outcome than the abuse it was built to prevent.
For very high-throughput services, some teams accept approximate limiting instead of perfectly synchronized counters, each node enforces a local limit that's a fraction of the global target, trading precision for the ability to enforce limits without a network round trip on every single request.
Testing Your Rate Limiter Before Production Traffic Finds the Gaps
Rate limiting logic is exactly the kind of code that looks correct in review and then behaves differently under concurrent load, since the whole point is coordinating state across simultaneous requests. Load testing with a tool that can fire concurrent requests, checking that the 429 boundary lands where you expect and that legitimate burst patterns aren't getting caught, is worth doing before the first real customer complaint.
Pay particular attention to clock skew if you're running a distributed system across multiple regions, since a sliding window or token bucket implementation that assumes tightly synchronized clocks can behave inconsistently when they're not. Test the failure path too: what happens when the counter store is briefly unreachable, and does your fail-open logic actually engage the way you designed it to.
It's also worth testing from the client's perspective, not just the server's. Write a small script that respects Retry-After and confirm it recovers cleanly after being throttled, since that's the exact behavior you're hoping every real integrator's client will eventually implement.
Monitoring and Tuning Limits Over Time
A rate limit set once at launch is a guess, and traffic patterns change as your API gains real customers with real usage shapes. Dashboards built on something like Prometheus, tracking 429 rate per client, per endpoint, and over time, tell you whether your limits are actually calibrated to how the API gets used or just to what seemed reasonable on day one.
Watch for clients consistently running right up against their limit without ever tripping it. That's often a sign the limit is correctly sized. Watch for clients tripping it repeatedly during normal-looking usage, that's usually a sign the limit, the key you're limiting on, or the algorithm itself needs a second look before you assume the client is doing something wrong.
Treat the rate limiter as a piece of product infrastructure, not just a defensive measure. The limits you choose shape what your API can be used for, and the difference between a rate limiter that protects the backend and one that quietly caps your platform's usefulness usually comes down to how much attention it gets after the initial build.
If you're rebuilding rate limiting on an API that's already live, 137Foundry's web development team has done this migration enough times to know where the traffic surprises usually hide. Our services page covers the broader backend work we take on, and you can read more about how we approach projects on the about page. For infrastructure-level questions around edge enforcement, Cloudflare's and nginx's own documentation are solid starting points if you're evaluating where to put the first layer of defense, and Redis remains the most common backing store for the counter state itself. For the HTTP-level details, MDN has the full reference on the headers involved.
