Delta Lake Change Data Feed in Production: Six Things That Will Break Your Pipeline
- Jun 18
- 7 min read
A practitioner's guide to the failure patterns that the Databricks documentation doesn't warn you about — and how to recover from each one
I've been building incremental ETL pipelines on Azure Databricks for a while now, and Change Data Feed has become one of the most powerful tools in that stack. The concept is elegant: rather than re-scanning an entire Delta table on every pipeline run, you subscribe to a feed of row-level changes — inserts, updates, deletes — and process only what actually changed.
This is a guide to Delta Lake Change Data Feed production failure patterns — what causes them, how they surface, and the recovery code that actually works.
This article documents the six failure patterns I've encountered building CDF pipelines in production on Databricks — what causes them, how they surface, and the recovery patterns that work. If you're implementing CDF for the first time or hardening an existing pipeline, this is what I wish I'd had before I started.
How Delta Lake Change Data Feed Works in Production
Before getting into the failure patterns, it's worth being precise about what CDF does under the hood — because the failure modes make more sense with that context.
When you enable CDF on a Delta table (delta.enableChangeDataFeed = true), Databricks writes additional _changedata/ files alongside the standard data files for every UPDATE and DELETE operation. INSERT operations don't generate separate change files — the data files themselves serve as the change record.
Each change record includes three metadata columns: _changetype (insert, update_preimage, update_postimage, or delete), _commitversion (the Delta table version), and _committimestamp. UPDATE operations generate two records per row — a preimage (the before state) and a postimage (the after state). This is important for auditing and for SCD Type 2 implementations where you need both the old and new values.
You read changes using either the DataFrame API (readChangeFeed option) or the SQL table_changes() function. Both return identical results — choose based on your preferred interface.
One critical constraint that trips up nearly every first implementation: CDF is not retroactive. You cannot read changes from before the moment CDF was enabled. If you enable CDF on an existing table, your consumers need a full snapshot baseline before they can start reading incremental changes. Plan for this before you touch production tables.
The Six Failure Patterns
1. VACUUM removes your change history
This is the most common and most disruptive failure pattern. VACUUM removes orphaned files from Delta tables — including the _changedata/ files that back your CDF reads. If your pipeline falls behind by more than the VACUUM retention window (default 7 days), those versions become permanently unavailable.
The error is unambiguous: VersionNotFoundException: Cannot find version X of table. What's less obvious is that it can happen silently. Your pipeline might process version 50 successfully on Monday, then fail on Wednesday when it tries to read version 51 — because VACUUM ran Tuesday night and swept the files backing versions 40-60.
Recovery: Read the current table as a full snapshot, use that as your new baseline, and reset your watermark to the current version plus one. Future CDF reads start from there.
Prevention: Keep your pipeline's processing lag shorter than your VACUUM retention. If your VACUUM runs with 7-day retention, your pipeline should never be more than 5-6 days behind. Set retention explicitly rather than relying on the default and monitor the gap between your last-processed version and the current table version.
2. Schema evolution silently breaks downstream consumers
An upstream ALTER TABLE ADD COLUMNS or column type change will cause your CDF consumer to encounter columns it wasn't built to handle. Depending on how your consumer is implemented, this either throws a schema mismatch error or — more dangerously — silently produces incorrect results when it projects only the columns it knows about.
The tricky version of this failure is when the new column contains data that your downstream system actually needs, and you don't realise it's being dropped until someone notices the data is wrong.
Recovery: Build defensive schema projection into your CDF reader. Select only the columns your consumer knows about explicitly, fill any missing expected columns with NULL, and log any new upstream columns you're ignoring. This makes schema changes visible without breaking the pipeline.
Prevention: Implement schema change notifications as part of your data governance process. If you're on Unity Catalog (which you should be), use the system tables to monitor schema changes on source tables before they hit your pipelines.
3. Streaming checkpoint corruption leaves you blind
Structured Streaming checkpoints track exactly where your stream has processed up to. If that checkpoint directory gets corrupted, accidentally deleted, or becomes inaccessible, your stream cannot resume — it starts fresh, which typically means either re-processing everything (if your sink is idempotent) or skipping everything (if it restarts from the latest version).
The less obvious version of this problem: checkpoints stored in /tmp/ or in cluster-local paths are ephemeral on serverless compute. Your stream "works" in development, then loses its checkpoint every time the cluster restarts in production.
Recovery: Query your sink table for the highest _commitversion you successfully processed. Use that as your startingVersion for the recovery stream, with a new checkpoint path. This avoids both data loss and duplicate processing.
Prevention: Store all streaming checkpoints in a Unity Catalog Volume (/Volumes/<catalog>/<schema>/<volume>/). Volumes are persistent across cluster restarts and serverless sessions, are governed by Unity Catalog permissions, and are browsable in Catalog Explorer. Use versioned naming — stream_v1, stream_v2_recovery — so you can see the history of your checkpoint evolution.
4. CDF disabled mid-pipeline creates an invisible gap
This one is particularly insidious: someone runs ALTER TABLE UNSET TBLPROPERTIES ('delta.enableChangeDataFeed') on your source table — accidentally, during a migration, or as part of an optimisation effort. Your pipeline doesn't fail immediately. It fails on the next run, when it tries to read CDF from a version created while CDF was disabled.
The gap is invisible because DESCRIBE HISTORY still shows all the DML operations — they just have no associated change data files. You can see that rows were updated, but you can't see what changed.
Recovery: Re-enable CDF. Use DESCRIBE HISTORY to identify exactly which versions were written during the disabled window. If the operations were simple (INSERT-only), you may be able to reconstruct the changes from the history. If they included UPDATEs or DELETEs, your only option is a full snapshot diff against your last-known good state, accepting that the intermediate states are lost.
Prevention: Add CDF status to your pipeline health checks (see pattern 6). Run it before each batch, not after.
5. Out-of-order data undermines your sink state
With multiple concurrent writers and retry logic in the mix, commits don't always arrive in business-logic order. Your pipeline might process a correction (higher version) before it processes the original insert (lower version) — because the lower version arrived late due to a retry. Without version-aware merge logic, the late-arriving stale record can overwrite the correct current state.
The symptom is subtle: rows in your sink table that are correct most of the time but occasionally show stale values. Very hard to catch in testing.
Recovery and prevention (same answer here): Use an idempotent MERGE pattern with a version guard. The guard — source._commit_version > target.last_commit_version — ensures that a lower-version (stale) record can never overwrite a higher-version (current) record, regardless of processing order. Always store _commitversion in your sink table.
This pattern also makes your pipeline safe to replay. Because the version guard prevents regressions, you can re-deliver any batch — including the entire change history — and the sink state will converge correctly.
6. Failures happen silently unless you build for detection
CDF pipelines fail in ways that don't always surface immediately. A VersionNotFoundException is obvious — your job fails. A processing lag that's slowly creeping toward your VACUUM window is not. Neither is a CDF-disabled gap that only becomes visible when you're debugging stale data three weeks later.
The pattern I've settled on is a health check function that runs before each batch — not after. It validates three things: CDF is still enabled on the source table, the version you want to start from is still available, and your processing lag is within acceptable bounds.
Run this before your pipeline reads any data. If it returns unhealthy, you alert rather than process — and you get the diagnosis immediately rather than from a downstream data quality complaint.
What This Looks Like as a Production Pattern
Pulling the six Delta Lake Change Data Feed production recovery patterns
together, a resilient CDF consumer has a consistent shape.
Health check before processing — validate CDF status, version availability, and lag before touching data
Checkpoint in Unity Catalog Volumes — persistent, governed, discoverable
Version tracking in the sink — _commitversion stored on every row, used for recovery and version-guard MERGE
Idempotent MERGE logic — safe to replay any batch in any order
Defensive schema projection — log new columns, fill missing ones, don't fail silently
Lag monitoring against VACUUM retention — alert well before the window closes
None of these are complex individually. The discipline is implementing all of them together rather than only the ones that have already caused you problems.
What To Do Next
If you're building CDF pipelines and want to pressure-test the design before it hits production, a structured review of the pipeline architecture and failure handling is worth doing early. The patterns in this article are learnable from experience — but they're also learnable from a well-structured design review that doesn't require a production incident as the teaching mechanism.
All six recovery patterns are available in full on GitHub - https://github.com/keithjenneke/databricks-engineering-patterns
This article was written by Keith Jenneke, Principal Consultant & Practice Lead at Cypher Agency. Keith leads our data platform and AI engineering practice, building on Azure Databricks across professional services, resources, and government.
Comments