A customer integrates with your API, batches ten requests together because their workflow naturally produces ten records at once, and gets rate-limited on request six. Meanwhile, a scraper making one request every two seconds, sustained for hours, sails under the same limit without ever tripping it. Both outcomes come from the same root cause: a rate limiter built around counting requests in a fixed window, which is easy to implement and wrong about what it's actually trying to prevent.
Rate limiting exists to protect infrastructure from being overwhelmed and to keep any single client from starving others of capacity. It is not supposed to be a tax on normal usage patterns. Getting that distinction right is mostly a matter of picking the right algorithm and applying it at the right granularity, not writing more code.
Photo by Kevin Ache on Unsplash
Why Fixed-Window Counting Punishes Legitimate Bursts
The simplest rate limiter counts requests in a fixed time window, say sixty seconds, and rejects anything past a threshold until the window resets. It's easy to reason about and easy to implement with a single counter per client. The problem is the boundary between windows. A client that sends 50 requests in the last second of one window and 50 more in the first second of the next window has technically stayed within a "50 per minute" limit each window, but has actually sent 100 requests in about two seconds, twice the intended rate, right at the boundary.
The opposite failure is more common in practice: a client with genuinely bursty but low-average traffic gets capped mid-burst even though their total usage over any reasonable period is well within intended limits. A batch import job that fires 20 requests in the first five seconds of a minute, then goes quiet, looks identical to sustained abuse to a fixed-window counter, because the counter has no concept of burst versus sustained load. It only knows a threshold was crossed.
The Token Bucket Algorithm and Why It Handles Bursts Better
A token bucket works differently. Each client has a bucket that holds a maximum number of tokens and refills at a steady rate, say one token per second up to a cap of twenty. Every request consumes one token. If the bucket has tokens available, the request goes through immediately, regardless of how many requests arrived in the last few seconds. If the bucket is empty, the request waits or gets rejected.
This naturally accommodates bursts up to the bucket's capacity while still enforcing a long-run average rate through the refill speed. A client that's been idle can burst up to twenty requests instantly, then has to wait for tokens to refill before bursting again. A client sending a steady, moderate stream never empties the bucket at all. The algorithm distinguishes burst from sustained load structurally, which a fixed-window counter can't do without extra bookkeeping bolted on.
Sliding Window Counters as a Middle Ground
If token bucket feels like more infrastructure than you want to stand up, a sliding window counter is a reasonable middle ground. Instead of a hard reset at fixed intervals, it weights the current window's count against the previous window's count proportionally, based on how far into the current window you are. This smooths out the boundary problem from fixed windows without requiring the full token-bucket refill mechanism, at the cost of being an approximation rather than an exact count.
For most APIs, the choice between token bucket and sliding window comes down to how precisely you need to enforce the limit versus how much operational complexity you're willing to carry. Token bucket is more precise and more common in production systems; sliding window is simpler to reason about and often good enough.
Choosing Limits Per Client vs Per Endpoint
A single global limit per API key treats a cheap, read-only endpoint the same as an expensive, write-heavy one, which rarely matches actual infrastructure cost. A more accurate approach sets separate limits per endpoint category, generous limits on cheap reads, tighter limits on expensive writes or search operations, layered under a global ceiling per client to prevent one client from monopolizing overall capacity across every endpoint at once.
This layered approach costs more to implement and reason about than a single flat limit, so it's worth reserving for APIs where endpoint cost genuinely varies enough to matter. A small internal API with uniformly cheap endpoints doesn't need this complexity. A public API fronting a mix of simple lookups and expensive aggregation queries usually does.
Where to Enforce the Limit: Gateway vs Application Layer
Enforcing rate limits at a gateway or reverse proxy, rather than inside application code, keeps rejected requests from ever reaching your application servers, which matters during an actual traffic spike when application capacity is exactly what you're trying to protect. Nginx and similar reverse proxies support token-bucket-style rate limiting natively, and enforcing there means a flood of rejected requests never competes with legitimate traffic for database connections or application threads.
The tradeoff is that gateway-level limiting typically has less context about the request than application code does, business logic like "this client's subscription tier allows a higher limit" usually lives in the application, not the proxy. Many production systems end up with both: a coarse, high-capacity limit at the gateway to absorb floods, and a finer-grained, business-aware limit in the application for per-tier or per-feature enforcement.
Communicating Limits to API Consumers
A rate limiter that silently rejects requests without explanation forces every integrating developer to reverse-engineer your limits through trial and error. Returning a 429 status code (defined in the HTTP status code extension for rate limiting) along with headers indicating the current limit, remaining capacity, and reset time lets well-behaved clients self-throttle before they hit the wall, rather than after.
"The number one support ticket we see around rate limiting isn't 'your limits are too strict,' it's 'nobody told me I had a limit until I hit it in production.' Headers that expose remaining quota turn a mystery outage into a five-minute fix on the client's side." - Dennis Traina, founder of 137Foundry
Handling Backoff and Retry Correctly
When a client does get rate limited, the response should include a Retry-After header telling them how long to wait, and well-behaved clients should implement exponential backoff with jitter rather than retrying immediately or on a fixed interval. A fixed retry interval across many clients synchronizes their retries into another burst right when the limit window resets, which recreates the exact problem the limiter was trying to prevent. Jitter, a small random delay added to each client's backoff, spreads retries out and avoids that synchronized thundering-herd effect.
Photo by panumas nikhomkhai on Pexels
Testing a Rate Limiter Before It Reaches Production
Load-testing a rate limiter deserves the same rigor as load-testing the API it protects. Simulate realistic burst patterns, not just steady request streams, since steady-rate testing won't reveal boundary problems or reveal whether your token bucket's capacity is tuned correctly for real client behavior. It's also worth deliberately testing what happens when a client with a legitimate reason for high burst traffic (a bulk import, a webhook replay after downtime) hits your limits, to confirm the rejection behavior and retry guidance actually help that client recover gracefully rather than leaving them stuck.
Common Mistakes to Avoid
A few implementation details separate a rate limiter that works from one that quietly causes support tickets:
Sharing one limit across a distributed system without coordination. If your API runs across multiple servers and each one tracks its own local request count, a client can effectively multiply their real limit by the number of servers handling their traffic. A shared store like Redis that all instances check against is the standard fix, since it gives every server the same view of a client's current usage.
Applying the same limit to authenticated and unauthenticated traffic. Anonymous or unauthenticated requests should generally have tighter limits than authenticated ones, since you have less ability to hold a specific bad actor accountable when you can't identify who they are.
Forgetting to rate limit expensive read endpoints. Teams often focus rate limiting energy on write endpoints because that's where data corruption risk feels obvious, and leave expensive read or search endpoints unprotected, even though a search endpoint doing a full-table scan can be more expensive per request than a simple write.
What to Monitor After Launch
Shipping a rate limiter isn't the end of the work. Track how often clients actually hit their limits, broken down by client and by endpoint, and watch for patterns that suggest the limits themselves are miscalibrated rather than the traffic being genuinely abusive. A limit that a large share of legitimate integrators trip regularly is usually a sign the threshold was set too conservatively for real usage patterns, not a sign those integrators are misbehaving.
It's also worth tracking rejected-request volume as its own signal separate from overall traffic volume. A sudden spike in 429 responses without a corresponding spike in legitimate traffic is often the earliest signal of an actual abuse attempt or a misconfigured client retrying aggressively, and catching that pattern early is usually more useful operationally than the rate limiter's rejection behavior itself.
Building This Into Your Own API
Rate limiting is one of those systems that's straightforward in concept and genuinely easy to get subtly wrong in the details, particularly around window boundaries and distributed coordination. Getting it right protects both your infrastructure and your legitimate users' experience, which is the actual goal, not simply rejecting a fixed count of requests per minute.
If you're designing or auditing rate limiting for a public API, 137Foundry works with teams on exactly this kind of backend infrastructure problem, from choosing the right limiting algorithm to deciding where in the stack to enforce it. You can see the broader scope of our data integration work or browse the full services overview for related backend engineering work, and the about page has more on how we approach this kind of infrastructure problem.
