Skip to content

[BUG]: curvine-fuse buffered writes return EIO after FuseConf mount CLI refactor #1646

Description

@uhmh123

Is your feature request related to a problem?

After building Curvine at upstream/main 1fadcd0 and mounting curvine-fuse on /curvine-fuse, buffered user I/O fails with Input/output error (errno 5). fio sequential and random read/write workloads with CRC32C verification fail on the first substantive write; basic FUSE regression checks such as reading back echoed file content and creating a 10MB file with dd also fail with EIO.

Mount succeeds and directory operations work, but writes and verified reads through /curvine-fuse return Input/output error. All five fio sub-tests fail; the FUSE suite reports 8 failed cases including read-after-write, large file creation, sed edits, and hard-link content checks.

Reproduction:

  1. Build Curvine at upstream/main 1fadcd0 and start master, worker, and curvine-fuse with the default cluster configuration.
  2. Confirm /curvine-fuse is mounted and create /curvine-fuse/fio-test/fio_seq.
  3. Run: fio --name=seq_write --directory=/curvine-fuse/fio-test/fio_seq --ioengine=libaio --direct=0 --bs=256k --size=500m --numjobs=1 --rw=write --verify=crc32c --group_reporting
  4. Observe fio exits with Input/output error (err=5) instead of completing the write and verification.
  5. Optionally run: echo test > /curvine-fuse/fuse-test/sample.txt && cat /curvine-fuse/fuse-test/sample.txt to observe read/write EIO.

Impact

Blocks FUSE performance and durability validation; any sustained buffered I/O through curvine-fuse may fail in production-like workloads.

Describe the solution you'd like

Buffered and verified read/write workloads on a healthy curvine-fuse mount should complete without EIO and preserve written data for subsequent reads and verification.

Review e01cf91 changes in curvine-fuse/src/cli/mount_args.rs and crates/common/curvine-config/src/fuse_conf.rs ClientConfCliOverrides generation; compare effective mount flags against eb76f94 and restore correct buffered I/O behavior.

Additional context

FUSE profile passed on the immediately prior daily run at eb76f94. Regression correlates with e01cf91 refactor(fuse): generate mount CLI overrides from FuseConf. Closed GitHub issue #1225 documented a similar EIO symptom class on an older commit; fuse had been passing until this commit.

Root-cause hypothesis: After commit e01cf91, curvine-fuse mount option generation from FuseConf appears to mis-apply or omit settings required for buffered read/write, causing user I/O on /curvine-fuse to fail with Input/output error. Mount and metadata operations succeed, but sustained or verified writes fail deterministically.

Evidence summary: fuse-fio-serialized.log: fio io_u error Input/output error write offset=262144; dd error writing large_file.dat Input/output error; grep read failure after echo write. fio-test-results.json: 5/5 FIO tests FAIL. Prior run 20260829T014809Z fuse profile PASS on eb76f94.

Related issues: None identified.

Acceptance / test gaps:

  • Buffered fio sequential and random read/write tests with CRC32C verification pass on /curvine-fuse without EIO.
  • FUSE regression basic read-after-write and large-file dd operations succeed.
  • No Input/output error on verified writes at offsets beyond the first 256KB block.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions