Skip to content
Logo

Several machines, one bucket

Point each machine's config.toml (or POND_* env) at the same URL and credentials, and run pond sync on each.

Concurrent writers are safe: Lance serializes commits through the object store's conditional writes (optimistic concurrency), so two syncs at once cannot corrupt the dataset.

Within one machine, sync is single-flight: a second pond sync against the same store waits for the first (naming the holder), and the scheduled run passes --no-wait so ticks skip cleanly instead of queueing. pond status --hosts shows which machines feed the store - sessions and latest activity per ingest host.

The machines need not share an OS: macOS, Linux, and Windows hosts can all feed one store, and each schedules its own syncs through its native scheduler.

Mixed versions

A release that changes the store schema (0.13.0 added materialized tool columns) migrates the store in place the first time the new binary opens it - a one-time backfill with a stderr notice - after which older binaries fail at open with a schema-mismatch error. When several machines share one bucket, upgrade pond on all of them together; a machine left on an older version stops syncing until it upgrades.

Not every skew is loud. The release that added the messages_timestamp_zonemap index (0.16.0) introduces a silent one: a pond 0.15.1 or older binary reading a store that any newer binary has synced or optimized returns empty results for date-filtered searches - --from-date / --to-date on the CLI, from_date / to_date over MCP - with no error and no warning. Unfiltered search, pond get-session, pond get-message, and pond sql are unaffected. The index appears on the first pond sync or pond optimize by an upgraded binary, so one upgraded machine is enough to change what every other machine sees. Upgrade every machine that reads the store, not just the ones that write it.

Writers are worse than readers. Compaction orphans the zonemap whenever it rewrites fragments the index covers - Lance never remaps address-domain index payloads on stable-row-id datasets - and every pond version's compaction does this, 0.16.0's own included, not just older binaries. What differs by version is the repair. From 0.16.1, every pond sync and pond optimize probes the index right after the compaction phase and recreates it in the same run, so damage from a machine's own maintenance never outlives that run, and damage from another machine's is healed by the next sync; the recreate is a full rebuild of the timestamp index, a modest recurring cost that only fires when compaction actually rewrote covered fragments. On 0.16.0 nothing repairs automatically: date-filtered searches fail with a storage_unavailable error naming a missing fragment (both search modes) until someone runs pond optimize --rebuild. On 0.15.x and older the same syncs both cause the damage and mask it (empty results, no error). The store's data is never at risk - only the index goes stale. Until the whole fleet is on 0.16.1+, prefer stopping the schedule (pond schedule stop) on lagging machines over leaving them writing.