[FEATURE] Relocate existing-block rewrites away from low-space workers

[gangump82]

Is your feature request related to a problem? Please describe.

worker.free_ratio is a strict physical free-space admission floor. This protects space shared with other applications, but it also means that an existing FileLayout block can become temporarily non-writable when its current Worker directory cannot reserve the full rewrite staging copy without crossing the floor.

FileLayout rewrite is copy-on-write: the committed block remains present while a complete staging copy is created. A same-size rewrite therefore needs approximately one additional block of physical space, and an expanding rewrite from 10 GB to 20 GB may require approximately 30 GB at peak. Bypassing free_ratio, or accounting for only the growth delta, would violate the configured protection floor.

Describe the solution you'd like

Before rewriting an existing block, automatically relocate replicas that cannot safely create the staging copy:

1. Detect a typed DiskOutOfSpace admission failure for an existing-block rewrite.
2. Select a healthy Worker and storage directory with enough headroom above its configured protection floor.
3. Copy or rebuild the existing committed replica on the replacement location.
4. Atomically update the Master's block-location metadata while preserving the configured replication factor and minimum-success semantics.
5. Retry the rewrite against the new location.
6. Roll back or cancel partially opened writers and partial relocation state on failure.
7. Make retries idempotent so client, Worker, or Master retries cannot create duplicate replicas or leak reservations.
8. Apply per-directory relocation concurrency and staging-byte budgets to avoid a relocation storm when disks become constrained.
9. Expose relocation attempts, successes, failures, bytes moved, and throttling through metrics and logs.
10. Preserve DiskOutOfSpace as the user-visible result when no safe destination exists.

Describe alternatives you've considered

• Allow rewrites to bypass free_ratio: rejected because a rewrite can consume the physical space intentionally protected for other applications.
• Reserve only the rewrite growth delta: rejected because FileLayout creates a complete staging copy rather than modifying the committed file in place. For example, a 10 GB to 20 GB rewrite can require about 30 GB at peak, not merely the 10 GB delta.
• Allow only same-size rewrites: rejected for the same reason; a 10 GB to 10 GB rewrite can still require another 10 GB staging copy.
• Keep the current write freeze permanently: safe, but it leaves existing files unavailable for modification until capacity is manually freed or data is manually moved.

Additional context

This is a follow-up to the strict worker.free_ratio behavior discussed in PR #1644. The existing behavior must remain the safe fallback until relocation completes: preserve the committed block, do not create staging data below the floor, and return DiskOutOfSpace/ENOSPC when no safe destination is available.

Goal

Sub-requirement	Problem solved
Detect rewrite-specific DiskOutOfSpace failures	Distinguish relocation candidates from unrelated I/O failures
Select a destination with sufficient protected headroom	Avoid moving the failure to another constrained directory
Relocate or rebuild the committed replica before rewrite	Restore write availability without weakening free_ratio
Atomically update Master block locations	Avoid exposing incomplete or duplicate replica state
Preserve replication and minimum-success semantics	Prevent availability or durability regression during migration
Make retries and cleanup idempotent	Prevent leaked staging files, metadata, or reserved bytes
Add per-directory concurrency and byte budgets	Prevent relocation storms and excessive temporary disk usage
Add metrics and structured logs	Make relocation decisions and failures observable

Not Goal

• Bypass or dynamically lower the configured free_ratio floor.
• Write staging data into protected physical space.
• Change FileLayout's committed-plus-staging atomic publish model.
• Implement general-purpose cluster rebalancing unrelated to a blocked rewrite.
• Delete the committed source replica before the replacement is safely available.
• Hide a final capacity failure when no safe destination exists.

Test Plan

Test case	Test objective	Modules involved
Same-size rewrite on a source below its floor with a healthy destination	Relocate first, then complete the rewrite without crossing either floor	client writer, master block metadata, worker storage
Expanding rewrite, such as 10 GB to 20 GB	Reserve full target staging size and keep peak usage within admitted headroom	FileLayout, worker admission, relocation scheduler
No eligible destination	Preserve the original committed data and return DiskOutOfSpace/ENOSPC	client, worker, FUSE
Destination fails during copy	Cancel partial work and retain the original source location	worker replication, block store
Master update fails after copy	Retry or roll back idempotently without exposing duplicate active locations	master metadata, worker cleanup
Concurrent blocked rewrites	Enforce per-directory concurrency and staging-byte budgets	scheduler, metrics
Client retries after timeout	Do not create duplicate relocations or leak reservations	client, master, worker
Successful relocation and rewrite	Preserve data integrity and configured replication factor	end-to-end cluster

[szbr00]

Thank you for submitting this issue. Are you interested in working on it? If so, we would be happy to assign it to you as a priority. Thank you very much!

[gangump82]

Yes，I would like to handle this issue.