feat(meta): add experimental SlateDB metadata engine

[solracsf]

DO NOT MERGE AS-IS, EXPERIMENTAL, TESTING ONLY

SlateDB is an object-storage-backed, single-writer LSM key-value store.

Note

One bucket holds the whole filesystem. Today the standalone engines (SQLite/Badger) die with the node, and durable deployments require running Redis/MySQL/TiKV. With SlateDB, metadata gets the same 11-nines durability as the data blocks, in the same bucket. Node dies → remount anywhere, zero metadata migration, no backup cron.

This adds it as a TKV-based metadata engine behind the opt-in slatedb build tag (same pattern as FoundationDB), using the official Go binding (slatedb.io/slatedb-go, cgo, links libslatedb_uniffi).

• txn maps to SlateDB serializable snapshot isolation; commit conflicts surface through shouldRetry into the existing kvMeta retry loop.
• simpleTxn uses read-only snapshots; writes in it are rejected.
• Meta URL forms: slatedb://memory, slatedb:///local/dir, slatedb://s3://bucket/prefix; query params dbpath, durability=remote|memory, and settings= (applied over defaults via Settings.Set).
• Build with make juicefs.slatedb, test with make test.slatedb.

The .spike/ scripts are the evaluation harness: FUSE mount smoke tests on file:// and MinIO backends, plus the mdtest benchmark used to compare against Badger and SQLite.

Important

Building the shared library — the version must match. The Go binding checks the UniFFI contract version and per-function checksums at startup and aborts on a mismatch, so libslatedb_uniffi.so has to come from the tag that matches slatedb.io/slatedb-go in go.mod (currently v0.15.0):

git clone --branch bindings/go/v0.15.0 https://github.com/slatedb/slatedb
cd slatedb && cargo build --release -p slatedb-uniffi
export LD_LIBRARY_PATH=$PWD/target/release CGO_LDFLAGS=-L$PWD/target/release

Notes for reviewers

• No CI job builds the slatedb tag, so a green CI run here says this change is non-disruptive, not that the engine works. The engine is covered by make test.slatedb (the standard testMeta/testTKV suites plus SlateDB-specific tests), run locally against v0.15.0, plus FUSE mount smoke tests on a local directory and on MinIO (format → mount → I/O → unmount → remount → verify). Adding a CI job that builds the Rust library and runs make test.slatedb, analogous to the existing build fdb step, is the obvious follow-up if there is interest in this engine.
• Two behaviours the driver depends on are pinned by tests rather than assumed: reads inside a transaction observing that transaction's own writes and deletes (scan, exist, get), and shouldRetry classifying a genuine write-write conflict as retryable so kvMeta.txn restarts instead of surfacing an error.
• durability=memory logs a warning, since commits within the last flush interval are lost if the process is killed. Short-lived commands such as juicefs format should use the default durable URL, as they exit without closing the meta client.
• Scans use DbIterator.NextBatch, so a scan crosses the cgo boundary once per batch instead of once per row.
• The Go binding is Apache-2.0 and has no Go dependencies of its own; all of SlateDB is in the Rust shared library, which is supplied by the operator and is not covered by go.sum.

---

mdtest numbers (31 dirs serial, then 2,480 file creates across 8 threads)

Measured with slatedb-go v0.15.0. Best of two runs, Docker on a Windows dev host. The Badger column was measured with the read-only simpleTxn from #7173 applied, so it is not flattered by this branch predating that fix.

Meta engine	dirs/s (serial)	files/s (8 threads)
Badger	20,187	60,429
SlateDB file://, durability=memory	5,904	29,642
SlateDB MinIO, durability=memory	4,367	26,180
SQLite	2,667	5,370
SlateDB file://, durable, flush_interval=5ms	149	881
SlateDB file://, durable, 100ms default	9	56
SlateDB MinIO, durable, 100ms default	9	54

Note

Absolute throughput is very host-dependent and Badger in particular is noisy: across sessions on identical code it has measured anywhere between 28,220 and 82,122 files/s, and 42,288 vs 60,429 within a single pair of runs. Treat the create-throughput column as an order of magnitude, not a precise figure. The durable rows are the stable ones, because they are bound by a fixed flush interval rather than by CPU.

What the numbers say:

• Memory-durability SlateDB is genuinely fast: roughly half of Badger and ~6× SQLite, and moving the store from local disk to MinIO costs little, since flushing is asynchronous. This is the mode to use for the single-node niche, with a ≤ flush-interval loss window comparable to Redis AOF everysec.
• Durable mode is purely flush-interval-bound: file:// and MinIO are statistically identical (49 vs 50 files/s), which confirms the cost is the WAL flush cadence and not object-store PUT latency. Serial throughput is 1/interval (~9–10/s); 8 threads only reach ~5 commits/interval because JuiceFS's per-inode transaction lock serializes same-directory creates, capping the group-commit benefit.
• flush_interval is the lever: 5 ms → 821 files/s, a 16× improvement, entirely reasonable against MinIO/S3 Express (on S3 Standard it would just mean every commit waits on the actual PUT).
• A FUSE-level probe of 50 serial mkdirs against MinIO with durable commits averaged 101 ms/op, matching the 100 ms default flush interval exactly.

[jiefenghuang]

SlateDB doesn’t seem very stable at the moment, so let’s keep this pending for now.

[solracsf]

I'll keep this PR up-to-data date in the meanwhile.