Every team building background jobs eventually ships a version that works fine until the moment it doesn't: a deploy goes out while a batch of jobs is mid-run, the process restarts, and whatever was in flight just disappears. No error, no log entry pointing at the cause, just a report that never generated or an email that never sent. The bug report that follows is always some version of "it worked yesterday," because it did, right up until the process that was holding the job in memory stopped existing.
This isn't really a bug in the job logic. It's a gap in the architecture: the queue itself was never designed to survive the one event that's guaranteed to happen regularly in production, which is the process restarting.
Photo by Anna Shvets on Pexels
Why in-memory queues quietly lose work
The most common first version of a job queue is an array or a simple in-process scheduler: push a job into memory, a worker loop pulls it off and runs it. It's fast to build and works perfectly in local testing, where the process never restarts mid-job.
Production is a different environment. Deploys restart the process. Autoscalers terminate instances. A crashed dependency can take the whole process down with it. None of these events ask permission first, and none of them give the in-memory queue a chance to hand its pending work to anyone else. Whatever hadn't finished simply stops existing the moment the process does.
The fix isn't a smarter in-memory data structure. It's moving the queue's state out of the process entirely, so that "the process that enqueued the job" and "the process that runs the job" don't have to be the same process, or even the same instance of the same process.
Persisting the queue outside the worker process
A durable job queue needs its state in something that outlives any single process: a database table, or a purpose-built store like Redis. Each job becomes a row or record with a status field (pending, running, completed, failed), not a value sitting in a variable somewhere.
Two common approaches, both reasonable depending on scale:
Database-backed queue. A jobs table with columns for status, payload, attempt count, and timestamps. Workers poll for pending rows, using SELECT ... FOR UPDATE SKIP LOCKED (on Postgres) or an equivalent locking read so two workers never grab the same job. Simple to reason about, and it means your job queue shares the same backup and observability tooling as the rest of your data, since it lives in the same database documented at PostgreSQL's site.
Redis-backed queue. Libraries built on Redis lists or sorted sets, often paired with a broker like RabbitMQ for more complex routing needs, give you faster enqueue and dequeue at the cost of an extra piece of infrastructure to run and monitor. Worth it once job volume is high enough that database polling becomes a bottleneck; overkill for a team processing a few hundred jobs a day.
Either way, the core requirement is the same: the job's existence and status live somewhere a process restart can't erase.
Making workers safe to die mid-job
Persisting the queue solves half the problem. The other half is making sure a worker that dies while holding a job doesn't leave that job stuck forever in a running state that nothing ever picks back up.
The standard pattern is a visibility timeout, sometimes called a lease. When a worker claims a job, it doesn't just mark it running, it also sets an expiration on that claim, say 5 minutes from now. If the worker finishes first, it marks the job complete and the lease is irrelevant. If the worker dies first, the lease expires, and a separate reaper process (or the next worker's poll query) notices the job has been running past its lease and returns it to pending for another worker to pick up.
This single mechanism, more than any other design decision, is what separates a job queue that survives restarts from one that just looks like it does in a demo. Without it, a crash during a job leaves that job's status stuck at running with no path back to pending, invisible to any monitoring that isn't specifically checking for exactly this condition.
Idempotency: the part that makes retries safe
A worker retrying a job it already partially completed is only safe if re-running the job doesn't cause harm the second time. An email job that already sent the email shouldn't send it again. A payment job that already charged the card shouldn't charge it twice.
The practical fix is an idempotency key: a stable identifier attached to the job (an order ID, a request ID) that the downstream system, or your own database, can check before acting. "Have I already sent the confirmation email for order 4821?" is a query you can answer cheaply if you log a row every time you actually send one, and check that row before sending again on retry.
Not every job needs this treatment. A job that recomputes a cached value from scratch is naturally idempotent, no key required. But any job with a side effect that shouldn't happen twice, a charge, an email, a webhook, needs an explicit guard, because the visibility timeout pattern above guarantees that eventually, some job will run twice.
Dead-letter handling for jobs that never succeed
Some jobs fail every time, not because of a transient blip but because the payload itself is malformed or the downstream system is permanently rejecting it. Without a limit, these jobs retry forever, each attempt burning a worker slot and cluttering logs with the same failure.
Cap the retry count (three to five attempts is typical, with exponential backoff between them) and move anything that exhausts its retries into a dead-letter queue: a separate table or list holding jobs that need a human to look at them, rather than a worker retrying blindly. This keeps the main queue's throughput protected from a handful of permanently broken jobs, and gives you a single place to check when someone asks why a particular action never completed.
A worked example: a job queue for order confirmation emails
Consider a queue responsible for sending order confirmation emails after checkout. Here's how the pieces above fit together in a concrete implementation.
A jobs table stores one row per confirmation email, with columns for order_id, status, attempts, locked_until, and created_at. Checkout inserts a row with status = pending the moment an order completes, inside the same database transaction as the order itself, so an order can never exist without a corresponding job row.
A worker process polls for rows where status = pending or (status = running and locked_until has passed), claims one with a locking update that sets status = running and locked_until five minutes out, then attempts to send the email. Before sending, it checks a sent_confirmations table keyed on order_id, guaranteeing the send never happens twice even if the same job runs on two different workers due to a race. On success, it marks the job completed and inserts a row into sent_confirmations. On failure, it increments attempts and, if under the retry cap, sets status back to pending with a backoff delay; if the cap is exhausted, it moves the job to a dead_letter_jobs table and alerts the team.
Photo by Thư Tiêu on Pexels
"The failure mode teams don't plan for isn't the job crashing. It's the job crashing at exactly the moment it's holding a lock nobody else knows to release. Design for that moment specifically, and the rest of the queue takes care of itself." - Dennis Traina, founder of 137Foundry
If the deploy happens mid-send, the worker process dies, the lease expires five minutes later, and a different worker (running the new deployed code) picks the job back up, checks sent_confirmations first, and either sends the email or discovers it already went out and skips straight to marking the job complete. No lost confirmations, no duplicates, no manual intervention required.
Operating the queue once it's live
A durable queue needs a small amount of ongoing observability to stay trustworthy: a dashboard showing pending job count over time (a steadily growing backlog means workers can't keep up), average time-to-completion per job type, and a count of jobs currently in the dead-letter table. None of this needs to be elaborate. A daily check of these three numbers catches most queue problems before they become incidents, because a queue that's silently falling behind looks identical to a healthy one until someone actually looks at the backlog size.
Bringing it together
A background job queue that survives a restart rests on four decisions: persist job state outside the worker process, use a visibility timeout so crashed workers release their claims, make side effects idempotent so retries are safe rather than dangerous, and cap retries with a dead-letter path for jobs that are never going to succeed on their own. None of these require exotic infrastructure. Most teams can implement all four on top of a database they already run.
The teams that skip this work usually get away with it for months, right up until a deploy lands during a batch run and support tickets start asking why half the confirmation emails from Tuesday never went out. Building the durability in from the start costs a few extra hours of design. Retrofitting it after the fact, with production data already in an inconsistent state, costs considerably more.
If your team is building or auditing background job infrastructure and wants a second set of eyes on the durability design, 137Foundry's web development service works through exactly this kind of reliability review for production systems. You can read more about how we work.
For further reading, the Wikipedia entry on message queues is a useful primer on the underlying concepts if you're building the case for this work to a non-engineering stakeholder, and the Node.js documentation is worth bookmarking if your workers are built on it. Start with the homepage at 137Foundry if you want to see the rest of our writing on production reliability.
