A nightly sync job takes eleven minutes for most of a year. Then one quarter the source table grows, the same job starts taking seventy minutes, and nobody notices because it still finishes. The trouble is that it is scheduled every hour. For a stretch of each night, two copies of the job are running at the same moment, reading the same rows and writing to the same tables.
Nothing crashes. No alert fires. The totals are just a little off, a few records appear twice, and a handful of updates quietly vanish. By the time someone finds the discrepancy, weeks of data have been touched and the cause is nearly impossible to reconstruct from logs that all look healthy.

Photo by Tima Miroshnichenko on Pexels
Why overlapping runs are so easy to miss
A scheduler like cron knows one thing: what time it is. When the clock says the job is due, it starts a new process. It does not ask whether the previous process is finished, because that was never part of its contract. Most hosted schedulers behave the same way unless you explicitly configure them otherwise.
That design is fine when jobs are fast and quiet. It becomes a hazard when a job's runtime drifts toward its interval. Data volumes grow, an upstream API slows down, a database picks up a heavy neighbor, and the margin between "finishes in time" and "overlaps with the next run" shrinks without any code change at all. Overlap is a property of runtime and schedule together, so reviewing the code will never reveal it.
There is also a second source of overlap that people forget. Someone triggers a manual run to fix a problem while the scheduled run is already going. Or a deployment restarts a worker and the orphaned process keeps running while its replacement starts fresh. Both situations produce two writers where the design assumed one.
What actually goes wrong when two runs collide
The damage depends on what the job does, but it tends to fall into a few patterns. Naming them helps you decide which guard you actually need.
Double processing. Both runs select the same unprocessed rows, because neither has marked them done yet. Each run then sends the invoice, emails the customer, or inserts the derived record. The result is duplicates that look entirely legitimate.
Lost updates. Run A reads a value, run B reads the same value, both compute a new one, and the slower writer overwrites the faster one. This is a textbook race condition, and it is silent because each write succeeds.
Interleaved partial state. A job that rebuilds a summary table by deleting and reinserting can leave the table half empty from the point of view of a second run that starts mid-rebuild. Anything reading the table during that window sees nonsense.
Resource pile-up. Even when the data survives, two heavy runs compete for the same database connections and API quota. Both slow down, which makes the next overlap more likely. The problem feeds itself.
Decide what "overlap" should mean for each job
Before adding any locking, decide the policy. Different jobs deserve different answers, and picking one deliberately beats inheriting whatever the scheduler does by default.
The three common policies are skip, queue, and replace. Skip means that if the previous run is still going, the new one exits immediately and waits for the next tick. This suits periodic syncs where the next run will catch up anyway. Queue means the new run waits its turn and starts when the old one ends, which suits jobs where every trigger carries unique work, such as processing an uploaded file. Replace means the new run cancels the old one, which is rare and only safe for jobs that are fully restartable.
Write the chosen policy next to the job definition, in one sentence. A future teammate who sees a job silently skipping should be able to find out that skipping is intentional. This small habit prevents a lot of confused debugging.

Photo by 小雨 on Pexels
Use a real lock, and make it expire
The usual guard is a lock that a run must acquire before doing any work and release when it finishes. The important design choice is where the lock lives. A lock file on one server does nothing once the job runs on two servers or inside containers that restart. The lock needs to live somewhere every worker can see.
Two practical options cover most teams. A relational database such as PostgreSQL offers advisory locks, which let a job claim a named lock for the duration of a session and release it automatically if the connection drops. A key-value store such as Redis can hold a lock key with an expiry, set atomically only when the key does not already exist. Both are well understood and both work, so choose whichever your system already depends on rather than adding a new piece of infrastructure.
Whichever you choose, give the lock a time limit. A job that crashes without releasing its lock would otherwise block every future run forever. The expiry should be comfortably longer than a normal run, and the running job should renew it periodically while it is making progress. If the renewal stops, the lock expires and the next run can safely take over.
Guard against the stale lock holder
Expiring locks create a subtle new problem. If a run stalls, loses its lock to a successor, and then wakes up, it will happily continue writing as though it still owned the work. Now you have two writers again, despite the lock.
The fix is a fencing token. Each time a run acquires the lock, it receives an increasing number. Every write the run performs carries that number, and the storage layer rejects any write whose number is lower than one it has already seen. A stalled run that wakes up late finds its writes refused instead of quietly corrupting newer data.
You do not always need the full machinery. For many jobs, a simpler version works: store the current run identifier in a status row, and have each batch of writes check that the row still names this run before committing. If it does not, the run stops. This is cheap and catches the worst cases.

Photo by Robert So on Pexels
Make the work itself safe to repeat
Locks reduce overlap, but they should not be the only defense. A lock can fail, expire early, or be bypassed by someone running a script by hand. The stronger design makes a second concurrent run harmless even when the lock does not catch it.
That means writing jobs so that processing the same row twice produces the same result as processing it once. Use upserts keyed on a stable identifier instead of blind inserts. Claim rows with a single atomic statement that marks them in progress and returns them, so two runs can never receive the same row. Record a processed marker in the same transaction as the output, so a crash cannot leave one without the other.
"A lock tells you overlap is unlikely. Idempotent writes tell you overlap is boring. We build both, because the day the lock fails is always the day a customer is waiting on that data." - Dennis Traina, founder of 137Foundry
Teams that adopt this pattern tend to find it also helps with retries and backfills, since all three problems come down to the same question: what happens if this work runs twice? The AI automation service we offer treats that question as a design requirement from the first sketch, not a patch applied after the first incident.
Detect overlap before it hurts
Prevention is better, but you also want to know when overlap happens or nearly happens. Instrument the job with three cheap signals and you will catch drift long before it causes damage.
First, record the start time, end time, and duration of every run, and alert when the duration crosses a fraction of the interval. A job that routinely uses seventy percent of its window is a job about to overlap. Second, count how often the skip policy fires. A skip that happens once a month is a blip, while a skip that happens every night means the schedule no longer fits the workload. Third, log the lock holder and any failed acquisitions with a run identifier, so that when something odd shows up in the data you can see which runs were alive at the same time.
Tools built for workflow orchestration, such as Apache Airflow, expose settings that limit concurrent runs of the same pipeline, and they surface run durations out of the box. If you already use one, turn those settings on deliberately. If you rely on plain cron, add the duration logging yourself, because nothing else will do it for you.
A short checklist for every scheduled job
When you review an existing job or design a new one, walk through these questions. They take minutes and cover most of the ground discussed above.
- What is the overlap policy, and is it written down beside the job?
- Where does the lock live, and can every worker reach it?
- Does the lock expire, and does the running job renew it?
- Would a stalled run that wakes up late be able to overwrite newer data?
- If two runs did collide, would the writes still produce a correct result?
- Do you alert on duration approaching the interval, not just on failure?
If any answer is "I am not sure," that job is a candidate for the next quiet afternoon of cleanup. Our data integration work often starts with exactly this kind of audit, because overlap bugs hide in the jobs that everyone has stopped thinking about.

Photo by Paul Lichtblau on Pexels
Fix the schedule, not just the symptom
One last point. Adding a lock to a job that overlaps every night treats the symptom. If the job really needs seventy minutes, an hourly schedule is simply wrong, and every guard in this article will be working overtime to cover for it. Either lengthen the interval, split the job into smaller independent pieces that each finish quickly, or make it incremental so it only processes what changed since the last run.
Overlap guards are insurance, and good insurance is cheap when the underlying risk is small. Choose a sensible schedule, add a lock with an expiry, make the writes safe to repeat, and watch the durations. Do those four things and the quiet corruption that starts with one slow night never gets the chance to begin. If you want a second pair of eyes on the scheduled jobs behind your reporting, you can see how we work and get in touch.