Small files and accumulated deletes (tombstones/positional deletes) remain the single most persistent performance and cost problem for lakehouse deployments in 2026. Despite improvements in object-store throughput and lakehouse metadata systems, engineering teams still face runaway metadata read costs, unpredictable query latency, and heavy operational burden from ad‑hoc repair jobs.

Why compaction still matters — and what “cost” really is

At the platform level, three interconnected effects make small‑file and delete bloat expensive:

  • Metadata amplification: query engines must enumerate and read file manifests, partitions and delete files. Millions of file entries increase planning time and memory usage in engines like Spark, Trino, and Flink.
  • I/O inefficiency: small objects mean higher per‑byte overhead for reads; many seeks and partial reads inflate egress and read amplification on object stores.
  • Operational churn: ad‑hoc compaction jobs, manual repair and repeated re‑writes consume cluster capacity and create platform complexity.

Those translate into slower ETL/BI queries, higher cloud bills (request and egress costs, not just storage), and increased maintenance time. Fixing the symptom (running one big rewrite) is easy; building a sustainable strategy is hard.

How bad is "bad"? Concrete metrics to measure before you act

Before choosing a compaction strategy, measure these metrics over a representative week of production traffic:

  1. Files per partition (median and 95th percentile). Aim for partition median file size above 128 MB for analytic workloads; high cardinality partitions may need special handling.
  2. Average file size across the table and by ingestion path. Very small files (1 MB) indicate micro‑batches or streaming writes without coalescing.
  3. Tombstone ratio — bytes in delete files vs live data bytes. High ratios (>10–20%) mean read amplification from applying deletes.
  4. Manifest/metadata size and number of manifest files. Track planning latency tied to manifest read counts.
  5. Query planner time and memory for representative queries, and percentage of failures or retries caused by planner OOMs.
  6. Object-store request rate (GET/LIST/PARTIAL GET). High request rates on reads correlate directly to cost and latency.

Collect these into a weekly report and prioritize tables where compaction yields the highest ROI (e.g., high query volume + severe file fragmentation).

Compaction patterns: pick the one that matches your ingestion and access profile

There is no one-size-fits-all compaction. Below are patterns that have proven effective in 2026 deployments.

1. Periodic batch rewrite (table-level rewrite)

Full-table or partition-level rewrites performed during off-peak hours. Suitable for tables with regular, predictable insert patterns and tolerable rewrite windows.

  • Pros: Simple, deterministic, reduces both small files and tombstones.
  • Cons: Heavy temporary I/O and compute; can interfere with concurrent writers unless the engine handles transactional isolation well.
  • Best for: Daily‑ingested tables with moderate size (10s–100s GB/day) where single big rewrite is feasible.

2. Incremental/streaming compaction (background rewriters)

Continuously rewrite small fragments into larger files with low‑priority workers or serverless jobs. This is the dominant pattern for high‑velocity streaming tables.

  • Pros: Smooths out write amplification, avoids large maintenance windows, better concurrency with writers.
  • Cons: Requires careful throttling to avoid contention and cascade re-writes; harder to guarantee a full delete cleanup.
  • Best for: Streaming feature stores, event tables with continuous ingestion.

3. Targeted compaction / clustering

Rather than rewriting entire partitions, cluster compaction focuses on hot partitions or files identified by the metrics above. Engines like Hudi, Iceberg and Delta all support targeted rewrite APIs.

  • Pros: Lower resource footprint, faster impact on hot queries.
  • Cons: Requires good hot‑spot detection and lifecycle rules.
  • Best for: Tables with skewed access patterns (e.g., partitioned by date where recent dates dominate).

4. Tombstone consolidation and rewrite

Delete accumulation requires different handling from small files. Techniques include coalescing positional deletes into fewer delete files, and rewriting to eliminate logical deletes by producing new clean files.

  • Pros: Eliminates per‑read apply overhead, reduces manifest churn.
  • Cons: Can be expensive because it touches many files; must be coordinated to preserve ACID semantics.
  • Best for: Slowly changing dimension tables, GDPR-related deletes, and high‑churn CDC targets.

Engine-specific considerations (Iceberg, Hudi, Delta)

Each format provides APIs and trade-offs you must consider:

  • Iceberg — Supports manifest and metadata rewrite utilities; rewrite-parquet can compact files without moving snapshot history. Iceberg’s rewrite-manifest and rewrite-data tools are good for both small files and delete consolidation when combined with snapshot-expire policies.
  • Hudi — Offers built-in compaction for MOR (Merge On Read) and clustering features that are designed around write amplification control. Hudi can provide near‑real-time compaction triggers tied to ingestion throughput.
  • Delta Lake — OPTIMIZE and ZORDER can compact and reorder files; VACUUM and retention settings handle historical snapshot cleanup. Delta’s transaction log size and checkpointing must be tuned to avoid long recovery times during heavy compaction.

Key operational point: test compaction utilities on clones or snapshots. Rewriting production tables without a controlled canary leads to data skew or transient availability issues.

Automation and safety: how to deploy compaction sustainably

Moving from ad‑hoc jobs to an automated system is the most common multiplier for success:

  1. Policy-driven triggers — Define rules like "if median file size < 32 MB and query volume > 200 qps, trigger targeted compaction for hot partitions."
  2. Backpressure and cost limits — Throttle compaction workers by CPU/memory budget and set a hard monthly cap on rewrite bytes to control cost.
  3. Canary and staging — Run compaction on test clones and measure query planning latency before promoting to production tables.
  4. Idempotence and observability — Compaction jobs must be idempotent and emit detailed metrics: files rewritten, bytes rewritten, planner time reduction, tombstone bytes removed.
  5. Retention and snapshot hygiene — Coordinate compaction with snapshot expiration. Aggressive compaction without snapshot expiry keeps historical manifests and negates benefits.

Cost‑benefit model: when compaction pays for itself

Estimate payoff by combining three numbers:

  1. Monthly cost of query latency and operator time (bad-case SLA penalties, analyst productivity loss).
  2. Monthly object-store request and egress cost attributable to fragmented tables.
  3. One‑time and recurring compaction compute costs.

If the expected monthly savings in (1)+(2) exceed compaction costs within 1–3 months for a table, prioritize it. In practice, high‑volume fact tables and real‑time feature stores often cross that threshold quickly; slowly queried historical archives rarely do.

Real-world lessons from 2026 deployments

Across multiple mid‑to‑large deployments we’ve tracked this year:

  • Incremental compaction with low‑priority workers reduced average planner time by 60% and lowered manifest read bytes by 70% for a 150 TB events table after 8 weeks.
  • A targeted tombstone cleanup on a CDC sink removed 30% of metadata entries and halved query CPU usage, with an operational cost equal to 10% of the weekly ETL budget.
  • Teams that combined compaction with partition design changes (coarser partitions for older data, micro‑partitioning for hot ranges) saw the largest sustained benefits.

Checklist to get started this quarter

  1. Run the metrics collection for your top 20 tables (files/partition, avg file size, tombstone ratio, manifest count).
  2. Classify tables into: archive (no action), periodic rewrite candidate, streaming compaction candidate.
  3. Build an automated, policy-driven compaction pipeline with throttling and canary stages.
  4. Instrument compaction impact in dashboards (planner time, manifest reads, query P95 latency, object-store request rate).
  5. Align snapshot retention policy with compaction cadence to avoid preserving unnecessary metadata.

Conclusion

Compaction remains a high-leverage engineering effort for lakehouse operators in 2026. The trick is not merely running one big rewrite but designing feedback‑driven automation that balances cost, concurrency and correctness. Teams that measure aggressively, prioritize tables using ROI, and automate with safe throttles consistently restore query performance and reduce cloud costs — turning compaction from an annual fire drill into a durable reliability practice.