Data integration pipelines have a particular failure mode: they stop working without producing any visible error. A scheduled sync misses a run. A webhook drops a batch of events. A transformation silently rejects records that do not match a changed schema. Users notice the problem hours later when they see stale data in a report or a missing record in a dashboard.
Monitoring a data integration pipeline is different from monitoring a web API. An API either responds or it does not. A pipeline can appear healthy because all its processes are running while still producing incorrect or incomplete output. The distinction between "the pipeline is up" and "the pipeline is producing correct data" requires different instrumentation.
This guide covers what to measure, where to instrument, how to structure dead letter queues, and how to build alerts that fire on the right signals without overwhelming your team.
Photo by Brett Sayles on Pexels
What Makes Pipeline Monitoring Different
A data integration pipeline typically consists of several stages: extract from source, transform, load to destination, and often a reconciliation or validation step. Each stage can fail independently, and a failure in one stage does not automatically produce a failure in the stages that follow. A transformation that quietly drops malformed records still allows the load stage to complete successfully.
This means monitoring must happen at stage boundaries, not just at pipeline entry and exit. A pipeline that reports 10,000 records in and 9,800 records out has dropped 200 records somewhere. Without per-stage counts, you do not know whether the drop happened in extraction, transformation, or loading.
The starting point is instrumenting a record count at each stage transition. Not just a success or failure status, but actual counts of records seen, records processed, records skipped, and records errored. This gives you a material balance to check: if records in never equals records out plus records skipped plus records errored, something is wrong.
Core Metrics for Integration Pipelines
Throughput and Record Counts
Track records processed per run, per hour, and as a running total since the last full reconciliation. Sudden drops in throughput often indicate rate limiting on a source API, schema changes in the source data, or network issues affecting extraction.
Compare current throughput against a rolling baseline. A 20% drop in record throughput that persists for more than two consecutive runs is worth investigating, even if no explicit error has been raised. Tools like Prometheus can track these counters and alert on significant deviations from the rolling average.
Latency per Stage
Each stage should have a recorded execution time. Dramatic latency increases in the transformation stage often indicate schema validation failures (every record now requires error-path processing). Latency increases in the extraction stage often indicate rate limiting or source system slowdowns.
Error Rates
Track errors as both an absolute count and a rate (errors per thousand records). A transformation stage that errors on 5% of records is a different problem from one that errors on 0.1%. The rate tells you whether this is a systematic issue with a data pattern or an occasional edge case.
Dead Letter Queues and the Importance of Not Silently Dropping Records
A dead letter queue (DLQ) is a separate storage location for records that fail processing and cannot be retried. The alternative to a DLQ is either retrying indefinitely (which blocks the pipeline) or silently dropping the record. Silent dropping is almost always wrong for data integration: the pipeline produces output that is subtly incomplete, which is often worse than producing no output at all.
Every failed record should be written to the DLQ with:
- The original record payload
- The error message that caused rejection
- The stage where the failure occurred
- A timestamp
This allows you to investigate failures after the fact, fix the underlying issue (schema mismatch, validation rule, upstream data quality), and replay the DLQ records without reprocessing the entire dataset.
Photo by Atlantic Ambience on Pexels
Monitor the DLQ as a metric. A DLQ that is growing indicates a systematic problem: new record types are consistently failing. A DLQ that receives occasional entries and then stops growing is an edge case issue. The difference in pattern tells you whether to prioritize the fix.
Structured Logging for Pipelines
Pipeline logs should be structured (JSON), not plain text. Structured logs can be queried programmatically, filtered by stage or run ID, and aggregated across runs. Plain text logs require string parsing to extract any useful signal, which degrades quickly as log volume grows.
At minimum, each log entry for a pipeline run should include:
- A run ID that uniquely identifies the execution
- The pipeline name and version
- The stage name
- A timestamp
- A severity level
- A record count (in, out, error) where applicable
- Any relevant context (source ID, batch offset, target table)
Tools like OpenTelemetry provide a vendor-agnostic standard for structured logging, metrics, and distributed traces across pipeline stages. Using a standard instrumentation library makes it easier to switch observability backends without rewriting your logging calls.
For log aggregation and search, Grafana integrates with multiple log backends (Loki, Elasticsearch, others) and provides a unified view across metrics and logs in one dashboard.
Alert Strategy: Firing on the Right Signals
Two failure modes make pipeline alerting difficult: false positives (alerts that fire when nothing is actually wrong) and missed detections (no alert when something is wrong). Alert fatigue from false positives causes teams to start ignoring alerts. Missed detections cause the monitoring to provide false confidence.
Avoid alerting on transient conditions that resolve themselves within one or two retry cycles. Most integration pipelines have retry logic for network errors and rate limits. Alerting on a first failure instead of a sustained failure produces noise without useful signal.
Effective alert conditions for integration pipelines:
- Record count below expected threshold for two or more consecutive runs
- Error rate above X% for any stage for one or more runs
- DLQ growth exceeding Y records in Z minutes
- Pipeline run duration more than 2x the rolling average
- No run recorded within N minutes of the scheduled run time (missed run detection)
The last condition is often forgotten: a pipeline that crashes at startup may not produce any error logs, because the logging infrastructure itself never initializes. Monitoring for the absence of a run is as important as monitoring for failures within a run.
"The first sign of a data integration problem is usually a data consumer asking 'why is this wrong' rather than any automated alert. Getting ahead of that requires monitoring at the record level, not just the process level." - Dennis Traina, founder of 137Foundry
End-to-End Reconciliation
Per-run monitoring catches process-level issues. End-to-end reconciliation catches accumulation errors: records that were individually processed correctly but are missing or duplicated at the destination due to idempotency failures, partial runs, or offset miscalculations.
A basic reconciliation job queries the source for a canonical count of records (or a checksum) and compares it against the destination. This does not need to run on every pipeline execution, but should run at least daily for high-importance pipelines and after any major pipeline restart.
For pipelines that process event streams rather than fixed datasets, reconciliation looks different: compare event offsets at the consumer against the source broker to confirm no events were skipped.
Photo by Pixabay on Pexels
Using Existing Observability Infrastructure
Pipeline monitoring does not require a separate observability stack. If your application already uses Datadog or a similar platform for infrastructure monitoring, pipeline metrics can be emitted to the same system using custom metrics. The advantage is centralized alerting with a consistent escalation policy.
If your pipelines run as separate services with no shared infrastructure, Sentry provides lightweight error tracking that works with any language runtime. It does not replace a full metrics system but handles the error capture and notification use case with minimal setup.
Schema Change Detection
A common pipeline failure mode that sits between process monitoring and data monitoring is schema change at the source. An upstream system adds a required field, renames an existing field, or changes a field's data type. The pipeline continues running, but transformation logic that assumed the old schema starts silently rejecting or mishandling records.
Schema change detection requires comparing the inferred schema of incoming records against a registered expected schema on each run. When a discrepancy is detected, the pipeline should raise an alert before processing the batch, not after. Processing a full day's worth of records against a broken transformation and then discovering the issue is far more expensive to recover from than catching the schema drift at ingestion time.
Libraries like Apache Avro, JSON Schema, and Protobuf provide formal schema definitions that can be validated at ingestion. For systems that do not use a formal schema registry, a lightweight validation step that checks for required fields and expected types on a sample of records at run start provides most of the protection at a fraction of the complexity.
Where to Start
For an existing pipeline with no monitoring: add structured logging to each stage boundary first. Record counts in and out, errors, and run IDs. This alone makes most failure modes visible. Add alerting on error rate and run absence next. Dead letter queues and end-to-end reconciliation can follow once the basic logging layer is operational.
For a new pipeline: build the logging layer before deploying to production. It is significantly harder to add instrumentation to a pipeline that is already processing live data than to include it from the start.
The 137Foundry data integration service works with teams to design, build, and instrument pipelines for production reliability. More technical guides are available at 137Foundry and through the services hub at 137foundry.com/services.
Photo by Towfiqu barbhuiya on Pexels
