Streaming Postgres changes into Apache Iceberg via Debezium, Kafka, and Flink solves a freshness problem most teams do not actually have, while introducing three failure modes (equality-delete read amplification, small-file explosion, and replication-slot bloat) that turn a lakehouse into a second production system requiring its own on-call rotation.
The pitch is straightforward: connect Debezium to Postgres's write-ahead log, publish change events through Kafka, sink them through Flink into Iceberg, and you get a queryable, near-real-time replica of your OLTP database sitting on cheap object storage.
Plenty of vendor blogs walk through the wiring. Fewer walk through what happens six months in, once the table has taken a few million updates. That's the part we want to walk through here.
That is because the architecture's core mechanism, the equality delete, was chosen for a good reason and still costs you later, as we'll see.
Read performance degrades before anything else visibly breaks. A CDC writer only has a primary key and new column values from the WAL, never a physical file path or row offset, so it has no way to issue an Iceberg position delete.
Equality deletes are the only mechanism available, marking a row deleted by a value match like id = 5 rather than a location. The Iceberg spec is explicit that both delete types are tracked by partition and merged against older data files at read time, this is merge-on-read, and it works. It just is not free.
Every query against a CDC-fed table now has to scan the data files, scan the accumulated equality-delete files, and filter matches row by row. As delete files pile up under high-frequency change volume, read amplification becomes severe. In our experience, nobody notices for the first few weeks. Then a routine dashboard query starts taking noticeably longer to return, and the root cause is buried three layers below the SQL.
Siemens hit this hard enough that, before adopting a CDC platform with built-in compaction, they ran periodic Spark jobs purely to strip delete files from their tables, because Snowflake's external table reader can't consume Iceberg equality deletes at all. That's a whole separate batch pipeline whose only job is cleaning up after the streaming one.
It fixes it, but it's not optional and it's not free. Flink, by design, favors small frequent commits to keep end-to-end latency low. Iceberg tables fed this way accumulate thousands of tiny Parquet files, and each one is another file that must be opened and merged at query time, slowing scans independent of the delete-file issue. BladePipe's analysis puts it plainly: Flink's commit behavior directly causes the small-file explosion that then requires standing up a dedicated compaction service just to keep the table queryable.
AWS has published numbers on what compaction actually buys you. In one EMR case study, 58,176 small Parquet objects totaling 2 GB were consolidated into roughly 437 MB files, and query runtime dropped from 1 minute 39 seconds to about 59 seconds, a 40% improvement. That's a real, measurable win, and one we'd take. It's also a second scheduled job, with its own optimize-data.commit-threshold tuning (default 10 commits, configurable down to 1 to compact after every commit) and its own failure modes to monitor.
Everything above assumes the pipeline is running. In production, keeping it running is the harder half of the job, and the failures here, we've found, are sharper than a slow dashboard.
If a Debezium connector goes down, its replication slot stays open on the primary and WAL keeps accumulating at the database's write rate. It does not degrade gracefully — it fills a disk.
A busy OLTP system can produce 20 to 50 GB of WAL per hour, so a connector that's down for even a few hours can push a primary toward disk-full — an outage on your production database caused entirely by your analytics pipeline.
Setting max_slot_wal_keep_size as a hard cap is the standard mitigation: for a database generating 10 GB/hour of WAL with 100 GB of free disk, a 1 GB threshold gives roughly 10 hours of warning before things get dangerous, and it's the kind of setting we only ever see added after the first incident, not before.
In practice, avoiding a repeat incident comes down to a short, repeatable procedure:
pg_replication_slots continuously for slot lag, not just connector uptime.max_slot_wal_keep_size as a hard cap so a stalled connector cannot take the primary down via disk-full.One2N's production writeup describes a subtler version: a Postgres publication defaults to publish_via_partition_root = false, so when Debezium is configured against root table names but the underlying table is partitioned, every change routed to a child partition gets silently dropped. No error, no alert, just missing rows.
It ran undetected for three weeks because the dev environment happened to have a different publication configuration and never reproduced the bug. The fix was one line, CREATE PUBLICATION ... WITH (publish_via_partition_root = true), available since Postgres 13. Three weeks of silent data loss for a one-line DDL statement is a brutal trade.
RisingWave's own catalog of production failures reads like a pattern, not a coincidence. One e-commerce team had an outage wipe a replication slot, forcing a 10-hour snapshot re-run across every table.
An S3 outage in another deployment broke the Iceberg sink, and because Debezium's LSN stopped advancing while the sink was down, WAL backed up toward the primary's disk again. A SaaS company found that the initial snapshot of a large table crushed OLTP performance badly enough that they had to restrict snapshotting to nighttime windows, stretching a planned two-day migration into a week.
None of these are exotic edge cases, in our reading. They're the ordinary operating conditions of a three-hop distributed system: source database, Debezium, Kafka, Flink, sink. BladePipe frames the honest cost of that topology as managing "at least three complex distributed systems," each an additional hop where latency can spike, and where a checkpoint timeout forces the entire Flink job to restart from the beginning rather than resume mid-stream.
The three failure modes above all originate at different points in the same pipeline, but they converge on the same symptom: a query that used to be fast, or a primary database that used to have headroom, no longer does.
Before reaching for this stack, it's worth asking what freshness requirement is actually driving the decision, because the answer, we've found, is often "nobody asked for sub-minute" once you push back on it.
.yeedu-cmp-wrap{width:100%;margin:32px 0;font-family:Inter,sans-serif}.yeedu-cmp-box{border:1px solid #334155;border-radius:14px;overflow:hidden;background:#020617}.yeedu-cmp-table{width:100%;border-collapse:collapse}.yeedu-cmp-table thead{background:linear-gradient(90deg,#020617,#0f172a)}.yeedu-cmp-table th{padding:14px 16px;text-align:left;font-size:11px;text-transform:uppercase;letter-spacing:.08em;color:#94a3b8;border-bottom:1px solid #1e293b}.yeedu-cmp-table td{padding:16px;border-top:1px solid #1e293b;font-size:14px;color:#e2e8f0;vertical-align:top;line-height:1.6}.yeedu-cmp-table tbody tr:hover{background:rgba(59,130,246,.04)}.yeedu-cmp-table td:first-child{font-weight:700;color:#f8fafc}
DimensionStreaming CDC (Debezium + Kafka + Flink)Batched micro-merge (minutes-to-hours)Typical freshnessSub-second to a few seconds1 minute to a few hours, tunableDelete handlingEquality deletes only; read-time merge cost grows with churnSame equality-delete mechanism, but fewer, larger delete files per merge cycleCompaction cadenceContinuous, often event-driven; effectively a standing serviceScheduled alongside each merge cycleOperational surfaceThree distributed systems: Debezium, Kafka, FlinkOne scheduler plus the sink jobDominant failure modeReplication slot bloat, silent partition drops, checkpoint restartsMissed merge window; stale reads until the next runBest fitSub-second dashboards, event-driven downstream consumersAnalytics, reporting, most BI workloads
Most teams don't, and it shows up repeatedly in how practitioners talk about this after they've run it. On the Hacker News thread discussing Iceberg's equality-delete design, one commenter argued flatly that batch updates every few minutes are sufficient for the overwhelming majority of use cases, and that genuine real-time-plus-transactional-consistency-plus-Iceberg is a rare combination of requirements, not a default one.
Another went further, arguing that Postgres, CDC, and Iceberg is often an architectural mismatch outright: Iceberg is columnar storage built for historical analytical scans, not for relational, highly mutable, low-latency data, no matter how the pipeline is wired.
A third angle from that same thread is worth sitting with: full column scans against billion-row tables every few minutes to keep merge-on-read current "could be quite pricey" on AWS or Databricks billing, an economic argument that rarely makes it into the architecture diagrams vendors publish.
Periodic batched merges, tuned to the freshness your consumers actually need rather than the freshness the tooling makes possible. RisingWave exposes this directly as a single tunable, the Commit Checkpoint Interval, defaulting to one minute; extending it trades some freshness for materially lower compaction and API-call cost. That's the knob we'd turn before adding a fourth distributed system to your stack.
Where genuine high-churn CDC is unavoidable, Iceberg v3's deletion vectors are the more targeted fix than architectural sprawl. Instead of separate delete files joined against data files at read time, a deletion vector is a single bitmap in a Puffin file mapped one-to-one against its data file, an O(1) lookup instead of a join. Dremio's benchmarking shows 50-80% read performance improvement over v2 positional deletes, and it's the kind of format-level improvement that reduces the tax on merge-on-read without requiring you to run Kafka. Compaction, via OPTIMIZE TABLE, is still required afterward to clear the vectors and rewrite clean files; deletion vectors lower the overhead, they don't remove the maintenance obligation.
The pattern across every practitioner account we looked at here is consistent: the pain doesn't come from CDC itself, it comes from committing to sub-minute freshness before confirming anyone downstream needs it, and then discovering that every component added to hit that latency target, Kafka, Flink, aggressive checkpoint intervals, is also a component that can silently drop a partition's worth of events for three weeks or fill your primary's disk overnight.
