Enterprise data pipelines increasingly treat dimension tables as first‑class writable assets. CRM records, device profiles, ad‑tech user states and inventory catalogs experience frequent changes; supporting efficient upserts matters for latency, cost, and query performance. In 2026 the three dominant table formats — Apache Iceberg, Delta Lake, and Apache Hudi — all advertise robust upsert/merge capabilities. But the operational reality for high‑churn, high‑cardinality tables depends on how each format manages tombstones, file rewrites, metadata overhead, compaction policies and query pruning.

Scope and why this matters now

This analysis focuses on "high‑churn dimension" workloads—tables with tens of millions to hundreds of millions of keys and steady update rates ranging from 5k–100k updates per second during peak windows (typical for large e‑commerce, ad platforms, and IoT profiles). We measure the key operational metrics data teams care about:

  • Write amplification: total bytes written to storage per logical update
  • Query latency and read amplification caused by tombstones/small files
  • Compaction cost and its scheduling implications
  • Operational complexity: tooling, monitoring, and failure modes

Understanding the tradeoffs helps teams choose the right format and compaction strategy, and avoid surprises in cloud storage bills and cluster CPU consumption.

What causes write amplification in upserts

Write amplification arises whenever a logical change requires writing more data than the delta itself. For lakehouses, common sources include:

  • Appending tombstones rather than in‑place updates (payload + tombstone metadata)
  • Rewriting entire files during compaction or when applying merges
  • Small files generated by micro‑batches that later need consolidation
  • Metadata churn: transaction logs, manifests and snapshots that grow with updates

Formalized: write amplification = (bytes written to object storage over a period) / (bytes of logical changes). An amplification of 1.0 is ideal; practical systems often operate between 1.5× and 12× depending on workload and policy.

Behavioral comparison: Iceberg, Delta, Hudi

Apache Iceberg

Iceberg separates data files from manifests and snapshot metadata. Upserts are typically implemented via MERGE INTO or by using CDC streams and rewriting affected partitions/files. Iceberg's strategy favors file‑level replaces: a compaction or MERGE rewrites files containing updated rows and produces new manifests.

  • Pros: Strong partition/file pruning due to explicit manifests; snapshot isolation reduces query disruption.
  • Cons: If updates are spread across many files, MERGE can rewrite many large files, driving high write amplification. Tombstones are represented by replaced files rather than row‑level markers, which can mean heavy rewrite cost until compaction is coordinated.

Delta Lake

Delta treats transactions in a transaction log and supports row‑level MERGE via rewrite semantics similar to Iceberg. Delta's optimistic concurrency and file‑based operations are mature in many managed services.

  • Pros: Good tooling and community patterns for using MERGE with micro‑batch or streaming update streams; built‑in compaction utilities in many distributions.
  • Cons: The transaction log can grow quickly with high update rates, increasing metadata overhead. Delta's default behavior often results in tombstones and small files unless compaction is tuned aggressively.

Apache Hudi

Hudi provides two table services: Copy on Write (CoW) and Merge on Read (MoR). These semantics directly influence write amplification.

  • CoW: Updates rewrite full files (similar to Iceberg/Delta) — higher write amplification but simpler read paths.
  • MoR: Writes deltas (delta log or columnar incremental files) and performs read‑time merges, reducing immediate write amplification at the expense of read complexity and higher read amplification.
  • Pros: MoR can deliver much lower write amplification for high‑churn workloads when read latency tolerates merge costs.
  • Cons: MoR adds complexity to reader implementations and increases CPU at query time; compaction still needed to bound read amplification.

Empirical scenarios and back‑of‑envelope calculations

Consider a 100M‑row table with average row size 1 KB (100 GB logical). Suppose daily churn is 10% (10M updates, 10 GB of logical change). We'll compare three coarse strategies:

  1. Append tombstones + periodic compaction
  2. Immediate MERGE that rewrites affected files
  3. MoR/Delta‑style small delta files with deferred compaction

Scenario A — append + weekly compaction: If each update appends a small tombstone record (100 B metadata per update) and the weekly compaction rewrites 50% of data to coalesce files, bytes written over a week ≈ logical changes (70 GB for daily churn over 7 days = 70 GB) + tombstones (7 GB) + compaction rewrite (50 GB). Total ≈ 127 GB. Amplification ≈ 127 / 70 ≈ 1.8×.

Scenario B — MERGE that rewrites 30% of files immediately: each day's 10 GB of changes triggers rewrites of 30 GB of existing data. Total daily bytes ≈ 40 GB → amplification 4.0×.

Scenario C — MoR/delta files with aggressive micro‑batches: delta writes 10 GB of delta files and defers compaction; weekly compaction rewrites 20% of dataset = 20 GB. Weekly total ≈ 10×7 + 20 = 90 GB over 70 GB logical → amplification ≈ 1.29×. Read cost during week increases substantially because queries must merge deltas.

These numbers are illustrative but match patterns seen in production: MoR/delta approaches reduce short‑term write amplification but shift cost to reads and later compaction. Immediate MERGE strategies can spike storage writes and network egress when many files are affected.

Operational tradeoffs and recommendations

There is no one‑size‑fits‑all answer. Choose based on the dominant constraint:

  • If read latency and simple query semantics are critical (analytics that rely on fast scans), tolerate higher write amplification and use CoW / immediate MERGE with aggressive compaction windows to keep files optimized.
  • If reducing cloud storage write bills and cluster CPU for compaction is primary (and read latency can accept some merge overhead), use MoR/delta patterns (Hudi MoR or Delta + incremental read merges) with targeted background compaction.
  • If update patterns are hotspotty (small subset of keys updated frequently), consider subpartitioning or key‑based bucketing to localize rewrites; this reduces files rewritten per update.
  • For truly massive update rates, shift frequently changing columns to a separate change‑stream table and maintain a materialized, periodically rebuilt projection for analytics—this hybrid lowers steady‑state rewrite costs.

Practical checklist for deployment

  • Measure logical change volume: track bytes and row counts of updates per window, not just event counts.
  • Estimate file‑rewrite impact: monitor how many data files a typical MERGE touches using explain plans and file manifests.
  • Tune compaction frequency and targets: balance immediate rewrite cost against long‑term read performance.
  • Use bloom filters / min/max stats and Z‑ordering / clustering where supported to reduce read amplification after compaction.
  • Monitor metadata growth: transaction logs and manifests can become a hidden cost; prune snapshots where SLA allows.

Decision matrix (quick guide)

  • Low update volume, fast reads: Iceberg or Delta with regular compaction (CoW).
  • High update volume, read latency tolerant: Hudi MoR or delta‑plus‑merge approaches with background compaction.
  • Mixed needs, hotspot updates: bucket/key partitioning + targeted compaction (works across formats).

Conclusion

By 2026 the core formats give data engineering teams similar primitives: ACID transactions, merges, and compaction. The meaningful differences are in the operational patterns they encourage. Hudi's MoR explicitly trades immediate write cost for read complexity, while Iceberg/Delta often push rewrite costs earlier. The right choice depends on your read SLA, update distribution and cost priorities—measure your logical churn, model write amplification under realistic compaction schedules, and pick the format and compaction policy that keep your storage and compute bills predictable without sacrificing necessary query performance.

For teams facing surprise bills or CPU spikes, the fastest remediation is to quantify logical update volume, identify hotspot keys, and implement partitioning or bucketing to localize writes. Over the medium term, invest in compaction policies and automated monitoring that tie compaction windows to business SLAs rather than rigid schedules.