How to Write Runbooks for Failing Data Automation Jobs That On-Call Engineers Can Follow at 3 AM

An open notebook with handwritten checklist pages on a desk

The alert fires at 3:07 AM. A nightly job that syncs orders into the reporting database has failed, and the person holding the pager joined the team six weeks ago. They did not write the job, they have never seen it fail, and the only documentation is a wiki page titled "Order Sync Overview" that explains the architecture in three paragraphs and says nothing about what to do right now.

This is the gap a runbook fills. A good one turns a confusing alert into an ordered list of things to check, with the commands already written out and the decision points marked. A bad one, or no one at all, turns it into forty minutes of guessing while downstream reports quietly go stale.

Open notebook with handwritten checklist pages on a desk
Photo by cottonbro studio on Pexels

What a runbook is and what it is not

A runbook is a short, task-oriented document that answers one question: this specific alert fired, so what do I do? It is not an architecture overview, a design rationale, or a history of past incidents. Those documents are useful, but nobody reads them while their phone is buzzing in the dark.

The Wikipedia article on runbooks describes the idea in general terms, and the practice is older than most of the tools we use today. What matters for data automation is the specific shape of the problem. Data jobs fail quietly, fail partially, and fail in ways that look fine from the outside. A web server is either up or down. A data job can report success while loading half the rows.

That is why runbooks for data work need more care than runbooks for stateless services. The responder has to answer two questions, not one: is the job broken, and is the data it produced still trustworthy?

Start from the alerts, not from the system

The most common mistake is writing a runbook per system. You end up with a document called "The Billing Pipeline" that tries to cover everything, so it covers nothing well. Write one runbook entry per alert instead, and name it after the exact text of the alert.

If the alert says order_sync: last successful run older than 90 minutes, the runbook heading should say exactly that. The responder can search for the string they were paged with and land on the right page. Link the runbook URL inside the alert itself so they never have to search at all.

This also gives you a useful test for your alerting. If you cannot write a clear runbook for an alert because nobody knows what the right response is, the alert is probably not worth firing. Either define the action or demote it to a dashboard. Our related thinking on building an alerting strategy that does not cry wolf starts from the same principle.

The anatomy of a useful entry

Every runbook entry should follow the same skeleton, so a tired reader always knows where to look. Consistency matters more than cleverness here. Use these sections in this order:

  • What this alert means. One or two sentences in plain language. Not "the DAG failed" but "the nightly order sync did not finish, so the reporting tables are missing today's orders."
  • How bad is it. Who or what is affected, and how long before it matters. A finance report that is read at 9 AM has a deadline. An internal dashboard might not.
  • First checks. Three to five things to look at, in order, each with the exact command or link.
  • Likely causes and fixes. A short list, ordered by how often each cause actually occurs.
  • When to escalate. A clear threshold and a named person or channel.
  • How to confirm it is fixed. A concrete check, not "make sure it looks okay."

If an entry does not fit on one screen, it is probably doing too much. Split it, or move the background material to a separate document and link to it.

Write the first checks as commands, not sentences

"Check whether the database is reachable" is a sentence. psql "$REPORTING_DB_URL" -c "select 1" is a command. At 3 AM the command wins every time, because it removes a decision and a chance to mistype.

For each first check, write down three things: the exact command or link, what a healthy result looks like, and what an unhealthy result looks like. The healthy and unhealthy examples are the part people skip, and they are the part that makes the check usable by someone who has never seen the output before.

Typical first checks for a failed data job include:

  1. Did the job actually start? Look at the scheduler's run history for the expected time.
  2. Where did it stop? Pull the last fifty lines of the job log and look for the first error, not the last one.
  3. Is the source reachable? Test the API or database the job reads from.
  4. Is the destination reachable and writable? Test the target the same way.
  5. Did a credential or quota expire? Check the secret's expiry date and the vendor's rate limit headers.

A workflow tool such as Apache Airflow or Dagster already records run history and logs, so the first check is often just a link to the right run page. Put the link pattern in the runbook with the job name filled in.

Magnifying glass resting on a printed log page
Photo by cottonbro studio on Pexels

Separate "restore the job" from "restore the data"

This is the section that data runbooks need and service runbooks do not. Getting the job to run again is only half of the recovery. The other half is making sure the data it was supposed to produce is correct and complete.

Give each entry two explicit parts. The first is the fix for the job: restart it, rotate the credential, raise the memory limit. The second is the data check afterward: compare the row count against the source, confirm the latest timestamp in the target table, and check that no partial batch was left behind.

If your jobs are safe to rerun, say so loudly and say how. "Rerunning is safe. The job is idempotent and will overwrite today's partition" is one of the most valuable sentences you can put in a runbook. If a rerun is not safe, say that even louder, and list what has to be cleaned up first. A responder who reruns a non-idempotent job and doubles every invoice has made the incident worse, and the runbook is the only thing that could have stopped them.

Document the decisions that need a human

Some failures cannot be resolved by following steps. A schema change upstream, a vendor outage, or a spike in volume might need a judgment call. The runbook should not pretend otherwise.

For these cases, write down the decision, the options, and who has the authority to choose. For example: "If the vendor API has been down for more than two hours, post a notice in the finance channel and skip today's sync. Only the data lead can approve a backfill over the weekend." That tells the responder when to stop trying to fix it and start communicating.

"A runbook should make the next action obvious, and it should say plainly when the next action is to wake someone up. Nobody should have to decide at 3 AM whether escalating is allowed." - Dennis Traina, founder of 137Foundry

Escalation paths deserve the same precision as technical steps. Name the channel, the person, and the expected response time. If the only escalation is "ask the team," the runbook is not finished.

Keep the runbook next to the code and test it

The reason most runbooks rot is that they live in a wiki that nobody updates when the job changes. Store them in the same repository as the job, in a runbooks/ folder, and review them in the same pull request. When someone changes the job's retry behavior, the diff shows the runbook that describes it.

You can add a small check to your continuous integration that fails if a defined alert has no matching runbook file. It is a crude rule, and it works. A short script that reads your alert definitions and looks for a file with the same name will catch the gap before it reaches production.

Then actually use the runbooks. Once a quarter, pick an alert and have someone who did not write the job follow its runbook against a staging failure you deliberately cause. Watch where they hesitate, and fix those lines. The Google SRE book treats this kind of rehearsal as normal practice for good reason: a runbook you have never run is a guess about what will work.

Learn from every incident, then edit the runbook

After each real incident, spend ten minutes updating the runbook entry before closing the ticket. Add the cause you just discovered, correct any step that was wrong, and delete anything nobody used. A runbook that grows without pruning becomes the same wall of text you were trying to replace.

Track two simple numbers over time: how long it takes from alert to first useful action, and how often the responder had to ask someone else for help. If both go down as the runbooks improve, they are working. If a particular alert keeps needing human help, that is a sign the job itself needs fixing, not just better documentation. Tools like Prometheus and Grafana can chart the alert-to-resolution time so the trend is visible without manual bookkeeping.

Wall of sticky notes arranged during a planning session
Photo by Vitaly Gariev on Pexels

A starter template you can copy today

If you have no runbooks at all, do not try to write them all this week. Pick the three alerts that fired most often last quarter and write those. Use this skeleton for each:

  1. Alert text exactly as it appears when paged.
  2. Meaning and impact in two sentences, including the deadline that matters.
  3. First checks as a numbered list of commands with healthy and unhealthy examples.
  4. Fixes by cause, most common first, with a note on whether a rerun is safe.
  5. Data verification steps to run after the job is green again.
  6. Escalation with a named person or channel and a time threshold.
  7. Last reviewed date and the name of whoever reviewed it.

That last field matters more than it looks. A runbook marked as reviewed eight months ago tells the reader to trust it less, which is honest and useful. Pair it with a rule that any runbook older than six months gets re-read during the next on-call handoff.

If your team would like an experienced second pair of eyes on the reliability of the jobs themselves, not just the documentation, that is the kind of work we do. You can read about our AI automation services from 137Foundry, or look at how we approach data integration when the failing job is a sync between two business systems. The broader list of what we build is on the services page, and you can start from the 137Foundry homepage if you are not sure which fits.

Whatever you do, write the first runbook before the next incident, not during it. The best time to describe what to do when a job fails is while you still remember how it works, and well before someone else has to figure it out in the middle of the night.

Need help with your next project?

137Foundry builds custom software, AI integrations, and automation systems for businesses that need real solutions.

Book a Free Consultation View Services