Compaction gets pitched as routine table hygiene, the Iceberg equivalent of vacuuming a Postgres table. It isn't. Run it on a fixed schedule with default settings and it becomes a second, poorly understood cost center: one that can run 20 to 30 times more expensive than a tuned job and still lag hours behind the ingestion it's supposed to be cleaning up after.
We've watched teams treat rewrite_data_files as a checkbox: schedule it nightly, move on. Then the compaction cluster shows up as a line item bigger than the query compute it was meant to protect, and nobody can explain why.
Every streaming or micro-batch writer commits a new snapshot on each flush. Iceberg tracks every one of those commits faithfully, which is the whole point of the format, and also the source of the problem.
A streaming job checkpointing every few seconds doesn't just produce small data files. It produces a long chain of snapshots, each with its own manifest update, and that metadata churn compounds independently of file count.
Fixing small files in Iceberg tables means treating commit frequency as a first-order cost driver on streaming tables, not an afterthought tuned after the fact.
Delete operations make it worse. Iceberg implements updates as a delete plus an insert rather than an in-place rewrite, so a table under moderate churn generates data files and delta files simultaneously.
Starburst's TPC-DS benchmark found that once 3% of rows in a table had gone through delete-and-insert cycles, query performance dropped by a factor of 1.5x, purely from the resulting file explosion. Every file, however small, carries a fixed open cost. A single scan can explode into thousands of file-open operations before a single byte of relevant data gets read.
Planning time grows with file and manifest count, not with the size of the result set. Each additional small file adds a fixed per-file cost to opening, listing, and reading manifest entries, so a table fragmented into thousands of small files pays that tax before a single predicate gets evaluated.
Three patterns dominate: frequent trickle-load commits from streaming writers, derived datasets persisted without regard for downstream file layout, and delete/merge-heavy workloads that pile up delta files faster than compaction clears them.
Amazon's EMR team demonstrated the fix's payoff directly: compacting 58,176 small objects totaling 2GB into roughly 437MB files cut a representative query's runtime from 1 minute 39 seconds to 59 seconds, a 40% improvement, using EMR's event-based compaction triggered after 10 successful commits.
That's the upside case. It's also the case everyone quotes. The downside case gets talked about far less.
Here's the part that surprises people who've only read the marketing docs: max-concurrent-file-group-rewrites often sits at just 1 on tables nobody has retuned, leaving compaction to run sequentially in practice.
Iceberg's rewrite planner groups files, then parallelizes across groups, not across individual files. If 10,000 small files get bucketed into five file groups, you get exactly five parallel tasks no matter how large the cluster is. One executor churns while the rest sit idle.
How the file-group bottleneck produces one busy executor regardless of cluster size:
IOMETE's breakdown of this bottleneck calls it a structural constraint, not a tuning failure. It's how Iceberg avoids extreme shuffle pressure and preserves atomic commits by design, and it means naive scaling (bigger cluster, same config) doesn't touch the actual bottleneck.
Z-order sorting compounds the risk. Applying it to a 200GB+ partition is, according to the same analysis, the single most common cause of out-of-memory failures in Spark-based Iceberg maintenance jobs.
Managed compaction doesn't sidestep this, it can amplify it. Onehouse benchmarked AWS S3 Tables' managed maintenance against a DIY EMR job on a 100GB, 10,000-file dataset.
S3 Tables running roughly 29x more expensive than EMR for the identical compaction outcome:
| Compaction approach | Dataset | Cost | Cost vs. cheapest option |
|---|---|---|---|
| AWS EMR (self-managed Spark) | 100GB, ~10K files | $0.17 | 1x (baseline) |
| Onehouse | 100GB, ~10K files | $2.29 | ~13.5x |
| AWS S3 Tables (managed) | 100GB, ~10K files | $5.04 | ~29.6x |
On a larger 953.7GB run, the comparison against Onehouse's own engine showed the same pattern: $47.69 for S3 Tables against $2.29, roughly 20x.
Worse, it wasn't fast. Observability was close to nonexistent: no dashboard, just a CLI command showing the timestamp of the last run. Teams paying a premium for "managed" got neither speed nor visibility in return.
Spark's JVM startup and task-coordination overhead, what RisingWave's engineering team calls the "distributed tax," dominates short compaction jobs where actual data movement is small relative to orchestration cost.
| Engine | Wall-clock time | Compute cost |
|---|---|---|
| Apache Spark | 1,533 seconds (~25.5 min) | $0.33 |
| Rust/DataFusion (RisingWave) | 277 seconds (~4.6 min) | $0.06 |
That's a 5.5x gap in time and an 82% reduction in compute cost for the identical bin-pack workload.
On a harder workload mixing 20,000 data files with 20,000 position deletes and 20,000 equality deletes, Spark failed with out-of-memory errors on the same hardware where the Rust engine succeeded.
We're not suggesting every team rip out Spark for compaction. Most shops already run Spark for everything else, and adding a second execution engine has its own operational cost. But it's worth knowing that the JVM overhead isn't imaginary: it's measured, it's repeatable, and it's the reason a five-minute compaction job can burn fifteen minutes of wall clock on coordination alone.
One rule of thumb from production lakehouse operators: if the compaction cluster's bill exceeds roughly 30% of query compute spend, the strategy itself, not the cluster size, needs to change.
LakeOps' FinOps analysis of Iceberg fleets across several production deployments found that 25 to 40% of object-storage spend on typical lakes covers bytes no query will ever read again, orphaned or superseded files left behind by compaction runs that never got followed up with snapshot expiration.
Table layout itself carries a startling multiplier. The same analysis cites a 1TB table with a selective filter costing about $0.25 to scan when well laid out, versus $5.00 for the same query against a fragmented, unsorted copy. That's a 20x difference driven entirely by file layout, before anyone touches a WHERE clause.
Sequencing matters, and it's an easy mistake to make: running expire_snapshots ahead of rewrite_data_files clears snapshot metadata but skips over data files that compaction hasn't superseded yet, so much of the intended cleanup never actually happens. In practice, a maintenance pass on a table under active write pressure should run in this order:
rewrite_data_files on the partitions showing small-file or delete-file buildup, so small files get bin-packed and outdated data files get superseded.expire_snapshots now that the files it's cleaning up have actually been superseded, rather than before compaction has run.Why maintenance order matters: running cleanup before compaction leaves superseded files stranded:
Compaction treats a symptom. The disease is commit frequency and file-size configuration at write time, and it's cheaper to fix there. Set write.target-file-size-bytes (or the Databricks equivalents, delta.targetFileSize and iceberg.targetFileSize) explicitly rather than relying on defaults.
Databricks auto-tunes based on table size: 256MB for tables under 2.56TB, scaling linearly to 1GB between 2.56TB and 10TB, and 1GB beyond that, via its file size controls.
Auto-compaction has its own knobs, spark.databricks.iceberg.autoCompact.enabled and its Delta counterpart, distinct from the manual OPTIMIZE/rewrite_data_files path, and mixing the two without understanding which one owns a given table is a common source of duplicated, wasted work.
Streaming writers deserve particular scrutiny. Lengthening checkpoint intervals reduces snapshot count directly. A table committing every 10 minutes can accumulate over 4,000 snapshots a month; stretching that interval doesn't just reduce data files, it thins the manifest chain that query planning has to traverse on every read.
None of this argues against compaction. Iceberg needs it, and skipping it entirely just relocates the cost to every downstream query, forever, instead of paying it once on a schedule you control.
It argues against treating the default configuration as good enough for a production table under real write pressure. The teams getting burned aren't running compaction wrong so much as running it blind: default concurrency, default schedule, no upstream tuning, and a bill that shows up looking like someone else's problem.
