Skip to content

[BUG]: BlockWriter construction does not cancel replicas opened before a later open failure #1648

Description

@gangump82

Describe the bug

BlockWriter::new() opens replicas sequentially. If an earlier WriterAdapter::new() succeeds but a later replica open fails, the ? operator returns immediately without explicitly cancelling writers that were already opened:

let mut inners = Vec::with_capacity(locate.locs.len());
for addr in &locate.locs {
    let adapter =
        WriterAdapter::new(fs_context.clone(), &locate, addr, pos, block_size).await?;
    inners.push(adapter);
}

Opening a local or remote writer can already create Worker-side writing metadata, staging storage, and reserved-capacity accounting. Dropping WriterAdapter is not a sufficient cleanup contract because cancellation requires asynchronous local/remote operations. A partial-open failure can therefore leave resources associated with the replicas that opened successfully.

This is independent of worker.free_ratio, although strict space admission can make the sequence easier to trigger when an earlier replica opens and a later replica returns DiskOutOfSpace.

To Reproduce

  1. Allocate a block with at least two replica locations.
  2. Allow the first replica's WriterAdapter::new() to complete successfully.
  3. Inject an open failure for a later replica, for example a storage-capacity rejection or RPC failure.
  4. Observe that BlockWriter::new() returns the later open error without explicitly calling cancel() on the earlier writer.
  5. Inspect the earlier Worker for writing metadata, staging data, the active write context, and reserved bytes.

Expected behavior

When a later replica fails to open, BlockWriter::new() should:

  1. Best-effort cancel every writer that was opened earlier in the same construction attempt.
  2. Attempt all cancellations even if one cancellation fails.
  3. Preserve the original replica-open error as the primary returned error.
  4. Log or attach cleanup failures without replacing the original error.
  5. Never cancel writers after all replicas open successfully.

Screenshots

Not applicable.

OS Version (please complete the following information):

  • OS: platform-independent client lifecycle issue; validate on Linux
  • Curvine version/branch: current main
  • Relevant configuration: block replication greater than one; no specific storage backend required

Additional context

  • Relevant code: crates/client/curvine-client-core/src/block/block_writer.rs, in BlockWriter::new().
  • Related issue [FEATURE] Add caller-level fault-injection coverage for replication writer cleanup #1645 covers caller-level cleanup after a writer has been successfully constructed. It explicitly does not cover partial failures inside the constructor.
  • Tests should cover local, remote, and mixed writer combinations; cancellation failure; original-error preservation; and cleanup of staging metadata/files and reserved bytes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions