Skip to content

fuse: experimental write-path FUSE passthrough (WIP) - #7202

Open
EngHabu wants to merge 14 commits into
juicedata:mainfrom
unionai:haytham/fuse-passthrough
Open

fuse: experimental write-path FUSE passthrough (WIP)#7202
EngHabu wants to merge 14 commits into
juicedata:mainfrom
unionai:haytham/fuse-passthrough

Conversation

@EngHabu

@EngHabu EngHabu commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown

Status: experimental / WIP, opt-in and off by default. Nothing changes for normal mounts.

What this is

Write-path FUSE passthrough (kernel ≥ 6.9) for JuiceFS. When a file is opened for write, JuiceFS hands the kernel a local staging file as a passthrough backing file (FOPEN_PASSTHROUGH + OpenOut.BackingID). The kernel then serves that file's read()/write() directly against the staging file, with no per-op upcall to the JuiceFS daemon. On release, the staging file is reconciled into normal JuiceFS slices via the existing writer path, and the kernel's attr/data cache is invalidated so readers see the committed file.

This removes the dominant cost of small random writes over FUSE — the kernel↔daemon round trip per operation.

How it works

  • Open/Create (write): allocate ‹staging-dir›/‹ino›-‹fh›.tmp, register it with Server.RegisterBackingFd, return FOPEN_PASSTHROUGH + BackingID. Reads/writes now bypass the daemon.
  • Release: read the staging file back, replay it through vfs.Write → vfs.Flush (normal slice upload + metadata commit), UnregisterBackingFd, delete the staging file, then InodeNotify to refresh the kernel cache.
  • Durability model: writes are durable to object storage at release/commit, not per-op — the same commit-style semantics passthrough implies (writes land in the local staging file first). For overwrite-heavy workloads this also reduces object-store write amplification, since reconcile uploads each block's final state once.

Opt-in (prototype): JUICEFS_PASSTHROUGH=1 and JUICEFS_PASSTHROUGH_DIR=‹dir›. The staging dir must be on a non-stacked filesystem (tmpfs/ext4/xfs) — an overlayfs/fuse backing returns ELOOP from the kernel's max_stack_depth check. On a real node the JuiceFS cache dir (node disk) satisfies this.

Dependency

Requires FUSE passthrough support in the go-fuse binding, which release-2.5 doesn't have yet — see juicedata/go-fuse#53. This PR currently wires it via a replace to that branch; it would point at the merged go-fuse commit once #53 lands.

Benchmarks (4 KiB random write, local file:// backend, kernel 6.12)

vanilla passthrough factor
buffered ~780 IOPS ~1.88M IOPS ~2400×
durable (fdatasync) ~270 IOPS ~152k IOPS ~560×

The per-op FUSE daemon tax is gone — writes hit the local staging file at native speed. (Benchmark staged on tmpfs, so the durable figure is RAM-fsync-bound; on node NVMe expect lower but still ~10–50k, i.e. well above the ~270 vanilla floor.) Reads and metadata ops are unchanged — this is purely a write-path optimization.

Correctness

  • Data + metadata are always committed correctly (verified: post-flush metadata size matches bytes written, and remount reads match the original).
  • Read-your-writes verified for 1 / 17 / 64 / 200 / 333 MB (multi-chunk) after the InodeNotify cache-invalidation fix.
  • The cache-invalidation fix adds no throughput cost (two notify calls at close; measured IOPS unchanged).

What's changed

  • pkg/fuse/passthrough.go (new) — staging-file lifecycle + reconcile.
  • pkg/fuse/fuse.go — Open/Create inject passthrough; Release reconciles; Serve negotiates the capability.
  • pkg/vfs/vfs.go — mirror the two new MountOptions fields into FuseOptions.
  • go.mod — replace go-fuse with the passthrough-enabled fork.

Remaining WIP (before this is merge-ready)

  • getattr returns a stale size while a passthrough file is open and being written (daemon never sees the writes); report the backing file's size.
  • Sub-millisecond race if a file is stat'd microseconds after close() (async FUSE notify); closeable by disabling attr-cache for passthrough files.
  • Concurrent handles to the same inode; promote the env-var opt-in to a real --passthrough mount flag; add a regression test per AGENTS.md.

@CLAassistant

CLAassistant commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@EngHabu @claude
fuse: experimental write-path FUSE passthrough (WIP)
5e0a1bd 
Newly written files get a local staging file (tmpfs/ext4) the kernel reads/
writes directly via FUSE_PASSTHROUGH, bypassing the daemon per-op; reconciled
into JuiceFS slices on release. Opt-in via JUICEFS_PASSTHROUGH=1 +
JUICEFS_PASSTHROUGH_DIR (must be a non-stacked fs).

Results (4K randwrite, local file backend): buffered ~780 -> ~1.88M IOPS,
durable ~270 -> ~152k IOPS. Reconcile reads the staging file back into slices
(per-chunk buffer copy; reopen-by-path), then invalidates the kernel attr/data
cache (InodeNotify) so same-session reads see the committed file. Verified
correct read-your-writes for 1/17/64/200/333 MB. The cache fix adds no
throughput cost (only 2 notify calls at close).

Requires the passthrough-enabled go-fuse fork (local replace).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Haytham Abuelfutuh <haytham@union.ai>
@EngHabu
EngHabu force-pushed the haytham/fuse-passthrough branch from 155bf7e to 5e0a1bd Compare June 24, 2026 17:46
@EngHabu
EngHabu marked this pull request as ready for review June 24, 2026 19:56
@EngHabu

EngHabu commented Jun 24, 2026

Copy link
Copy Markdown
Author

Depends on juicedata/go-fuse#53, which adds the FUSE passthrough primitives (RegisterBackingFd / FOPEN_PASSTHROUGH / INIT negotiation) to release-2.5. This PR's go.mod currently replaces go-fuse with that branch; once #53 merges it will point at the merged commit. #53 should land first.

@EngHabu EngHabu mentioned this pull request Jun 24, 2026
Open
@EngHabu

EngHabu commented Jun 25, 2026

Copy link
Copy Markdown
Author

@jiefenghuang please take a look.

@jiefenghuang

jiefenghuang commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

@jiefenghuang please take a look.

Thank you very much for your contribution. We’ll be handling the 1.4 release this week, I’ll review this a little later.

EngHabu and others added 7 commits July 7, 2026 06:22
@EngHabu @claude
fuse: never passthrough internal/control files (fixes checkpoint hang)
d037dd5 
Open() consulted pt.tryOpen() before the IsSpecialNode() branch, so a
write-open of a JuiceFS internal inode (.control, .stats, .config, ...)
could be granted FOPEN_PASSTHROUGH and served by a kernel backing file.
The .control protocol then broke silently: the checkpoint verb's write/read
on .control went to a plain temp file instead of the daemon's in-process
handler, so the op never dispatched and 'juicefs checkpoint' hung
indefinitely (with passthrough enabled).

Guard tryOpen to refuse special nodes. Verified before/after with
passthrough genuinely engaged: no-fix checkpoint hangs (.control opened
special=true and passthrough'd); with the guard .control is skipped, data
files still use passthrough, and checkpoint completes with a valid
snapshot. Regression test in passthrough_special_test.go.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxvU7KdSpd1oRRmAgWTA6Z
@EngHabu @claude
fuse: pool passthrough backing registrations across opens
38c04c6 
Registering a backing fd costs an ioctl — or, on unprivileged broker
mounts, an RPC round trip to the node broker — plus a staging-file
create, once per write-open. Small-file workloads open thousands of
times and pay it every time (measured 2x slower small-file creates over
the broker on EKS, ENG26-869).

Keep reconciled staging files registered: truncate to zero and park
them (LIFO, capped at 64 per mount) for the next write-open. After
warm-up a small-file loop performs no registrations, no unregistrations
and no file creates. A backing is never attached to two live opens:
checkout is exclusive and a file is only parked after its reconcile
copied the data out and truncated it. Failed reconciles retire the
backing instead of recycling it; stale staging files from a crashed
process are swept at mount.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWLe95UTvfBpTXptpBXLrB
@EngHabu @claude
fuse: disable passthrough for the mount after EPERM registration
1938b2e 
The backing-registration ioctl needs CAP_SYS_ADMIN in the init user
namespace. A non-root mount whose container merely *adds* the
capability (bounding set, no ambient) fails EPERM on every attempt —
measured 2000+ doomed ioctl+create+warn cycles in one small-file run
on EKS (ENG26-869). EPERM is permanent for the process, so latch
passthrough off for the mount on first sight; other errnos stay
retryable. The warning now says what to do about it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWLe95UTvfBpTXptpBXLrB
@EngHabu @claude
vfs+fuse: fence passthrough reconcile for consistency points
9a50674 
close(2) returns before the RELEASE-time reconcile copies the staging
file into JuiceFS slices, so a checkpoint/commit issued right after a
close could FlushAll+snapshot mid-copy and publish a short file
(reproduced: 256MB write + immediate checkpoint -> 44MB in snapshot;
also seen on EKS as a finalize missing exactly the last written file).
Add an external-flush counter to VFS that reconcile brackets; the
checkpoint verb (stacked branch) waits for it to reach zero before
flushing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWLe95UTvfBpTXptpBXLrB
@EngHabu @claude
vfs+fuse: cover the unmount path for in-flight passthrough reconciles
c8ba37e 
Two more legs of the reconcile fence (the checkpoint verb was fenced in
9a50674, but finalize goes through unmount, not the verb):

- fuse.Serve waits out in-flight reconciles after the kernel session
  ends, so a daemon exit (umount or SIGTERM->lazy umount) can't strand
  a closed file's staging data. Reproduced on EKS: finalize published
  volumes short by exactly the last written 256MB file, both privileged
  and brokered.
- Export the in-flight count as gauge 'passthrough_staging_blocks'.
  The name contains 'staging_blocks' on purpose: durability watchers
  that sum staging gauges from .stats before sealing a snapshot (the
  flyteplugins drain-before-commit path) then wait for reconciles with
  no client change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWLe95UTvfBpTXptpBXLrB
@EngHabu @claude
fuse: fix passthrough data-corruption bugs found in review
c4b2f86 
Four independent-review CRITICALs on the write-passthrough path:

1. Existing-file corruption: tryOpen gated only on write-intent, so an
   O_RDWR/O_WRONLY/O_APPEND open of a NON-empty file got a backing that
   starts empty — reads served as zeros (read-modify-write corruption)
   and reconcile (linear copy from offset 0) overwrote the real prefix
   with holes. Gate on emptyAtOpen: Create, O_TRUNC, or zero length.

2. Cross-mount corruption: staging lived in a host-shared dir with a
   startup glob-sweep and per-process pool-N names — a second mount's
   sweep unlinked the first's live staging, and pool-N collided across
   processes. Isolate each mount in a private os.MkdirTemp subdir; drop
   the shared-root sweep; O_EXCL so a stale name can't be truncated.

3. Read-error-as-EOF: a mid-copy read error broke the loop like io.EOF
   and committed a truncated file as complete. Distinguish io.EOF; bail
   loudly otherwise.

4. Reconcile failure destroyed the only copy: the retire path os.Remove'd
   the staging file holding data whose close(2) already returned. Rename
   it to an .orphan sibling and log at error instead.

Also scope the consistency-point fence past the pf==nil check so plain
(non-passthrough) releases don't add spurious checkpoint contention.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWLe95UTvfBpTXptpBXLrB
@EngHabu @claude
fuse: one passthrough writer per inode (close reconcile-window reuse …
d378c47 
…race)

An immediate reopen-for-write of a file just written via passthrough sees
metadata size 0 (its reconcile is still in flight), so emptyAtOpen looks
true again and a second empty backing would clobber the first's data on
reconcile. Reserve the inode across open+reconcile; overlapping write
opens fall to the daemon path. Also covers two concurrent write-opens of
one empty inode.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWLe95UTvfBpTXptpBXLrB
@EngHabu
EngHabu force-pushed the haytham/fuse-passthrough branch from c6b5208 to d378c47 Compare July 8, 2026 02:16
@EngHabu @claude
fuse: wait out in-flight reconcile on reopen (close-to-open consistency)
b5f97b9 
A reopen of a file just written via passthrough could race the async
reconcile: until it lands, the size reads 0, so a new write (daemon or
passthrough) interleaves with the reconcile's linear copy and is lost.
Block Open (bounded 30s) until the inode's reconcile finishes, so the
reopen sees authoritative size/content.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWLe95UTvfBpTXptpBXLrB
@EngHabu
EngHabu force-pushed the haytham/fuse-passthrough branch from 6800364 to b5f97b9 Compare July 8, 2026 02:19
EngHabu and others added 5 commits July 8, 2026 03:02
@EngHabu @claude
fuse: close remaining passthrough durability holes (ENG26-874)
2bdc902 
Four fixes for the deferred HIGH/MEDIUM passthrough hardening items:

- SIGTERM force-exit: the 30s force-exit path now extends its deadline
  (10 min hard cap) while passthrough reconciles are in flight, so
  FlushAll can no longer commit a mid-copy prefix as a durable short
  file when umount is stuck.

- fsync/fdatasync honesty: FSYNC on a passthrough fh used to return
  success while every byte sat only in the local staging file. It now
  copies the staging content into JuiceFS slices (open stays live,
  backing stays registered; release reconcile remains the authority)
  and flushes the writer, matching non-passthrough fsync durability.
  Copies are serialized per-open (ptFile.mu) and fenced with
  Begin/EndExternalFlush so checkpoints wait them out.

- SIGHUP smooth upgrade: a handover while passthrough opens are live
  would strand their staging data (the successor has no record of
  them). The SIGHUP handler now drains — blocking new passthrough
  opens, bounded 10s — and refuses the restart if opens remain,
  re-enabling passthrough.

- truncate divergence: SETATTR size changes never reach the backing
  file (the kernel only diverts read/write/mmap); mirror successful
  size changes onto the live backing so reads and the release-time
  reconcile don't undo the truncate.

Also drops the obsolete google.golang.org/grpc/stats/opentelemetry
carve-out requirement (package merged into grpc; ambiguous import).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011M2cGhQ5cyh5t9u7DrMv2y
@EngHabu @claude
fuse: note the fsync exception in the passthrough durability contract
26bf421 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011M2cGhQ5cyh5t9u7DrMv2y
@EngHabu
build: point go-fuse replace at the unionai fork (repo transferred fr…
d1c5b3d 
…om EngHabu)
@EngHabu @claude
fuse: never reuse passthrough backings across inodes
aa93a4f 
Cross-inode reuse (register-once, checkin/checkout pool) was measurably
cheaper for small-file workloads, but is unsafe: once mmap(2) is called
on a passthrough fd, fuse_passthrough_mmap() repoints the VMA directly
at the backing file (vma_set_file), decoupling the mapping from the fd.
close(2) can still trigger reconcile+recycle while a dirty mapping keeps
writing to what is now, after recycling, a different inode's staging
file — silent cross-tenant data corruption, with no FUSE opcode able to
observe mmap activity on a passthrough-diverted file to guard against it.

Remove the pool entirely: checkout() always registers a fresh backing,
and reconcile() always retires it (unregister + close + remove) instead
of truncating and parking it for reuse. This also removes the
truncate/checkin race window between a completing reconcile and a new
open landing on the same backing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0127fkXsGz1PMmfkFKERDmtM
@EngHabu @claude
fuse: account passthrough reconcile's copy buffer against the memory …
3796448 
…budget

copyStagingLocked's 4MB read buffer was a raw make(), invisible to
utils.AllocMemory() — the counter vfs/writer.go and vfs/compact.go use
as backpressure (utils.AllocMemory()-store.UsedMemory() vs BufferSize).
Concurrent reconciles could inflate RSS well past the configured budget
without the writer/compactor ever seeing it and throttling. Use the
pooled utils.Alloc/Free instead, same as chunk/page.go already does for
its buffers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0127fkXsGz1PMmfkFKERDmtM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@EngHabu @CLAassistant @jiefenghuang