[BUG]: Replication scheduler deadlocks permanently when a source worker restarts — inflight semaphore permits are never released

[gangump82]

Environment

• Curvine v0.5.0-alpha (also verified present on current main as of ff777c31), 3 masters (Raft HA) + 8 workers
• block_replication_enabled = true, block_replication_concurrency_limit = 1000 (default)

Symptom

After a worker node reboot took longer than worker_lost_interval (10 min) — the worker was declared lost, re-replication kicked in — the master's replication scheduler deadlocked permanently:

• replication_inflight_number froze at exactly 1000 (= block_replication_concurrency_limit)
• replication_staging_number froze at 6377
• replication_failure_count stayed 0
• Zero replication log activity on the master from that point on — no Successfully replicated, no errors, nothing
• The worker later came back and re-registered with its original worker_id (cluster fully healthy, live=8/lost=0) — the queue stayed frozen for 80+ minutes, and remained frozen even after we deleted every file it was supposed to replicate
• Only a master restart (failover to a fresh Active) cleared it: metrics went back to 0/0 on the new leader

Root cause (source analysis, curvine-master/src/master/replication/master_replication_manager.rs @ v0.5.0-alpha)

The scheduler is a single serial loop:

recv.recv().await
  → replication_semaphore.acquire_owned().await     // L94, no timeout
  → replicate_block(block_id, permit).await         // L101

On successful submission, the OwnedSemaphorePermit is moved into an InflightReplicationJob and stored in the inflight_blocks DashMap (L198–207). The only code path that releases the permit is finish_replicated_block() (L249–268), which is triggered only by the source worker sending ReportBlockReplicationResult back.

If a source worker restarts (or its process dies) while jobs are inflight, those reports never arrive. There is:

1. No timeout / reaper on inflight jobs. inflight_blocks has exactly 4 references in the codebase (L46/77/198/249) — insert and remove only, no periodic sweep. Once 1000 permits leak, acquire_owned().await at L94 blocks forever, recv is never polled again, and the scheduler is silently dead.
2. block_replication_retry_interval is dead config. It's defined (master_conf.rs, default "5s") and has an accessor block_replication_retry_interval_ms(), but that accessor has zero callers. Retry is not implemented — the code even says so: // todo: retry on failure of block replication (L243) and // todo: error handling (master_replication_handler.rs:47).
3. Worker re-registration does not clean up its pending inflight jobs. The registration/heartbeat paths never touch MasterReplicationManager, so a worker coming back with the same worker_id does not unblock anything.
4. Worker-side submission is fire-and-forget. worker_replication_handler.rs:37-50 accept_job returns success: true as soon as the job is queued in the worker's in-memory mpsc. A worker restart evaporates that queue, and the master-side inflight entry leaks. Additionally, on the worker, report_job() early-returns with err_box! when job.storage_type is None (source block unreadable), so in that case the result report is never even sent — same leak, fully silent.

Also worth noting: report_under_replicated_blocks() (L212–240) does no dedup — the _worker_id param is ignored, blocks already inflight or already queued get re-enqueued, and staging_number only decrements on the success path (L206), never on the 6 early-return error paths in replicate_block() — so the staging gauge drifts upward and stops reflecting the real queue depth.

Reproduction

1. Cluster with block_replication_enabled = true, replicas=2 data spread across workers.
2. Reboot a worker node so that the outage exceeds worker_lost_interval (any GPU node with slow driver init easily exceeds 10 min).
3. Watch replication_inflight_number climb to block_replication_concurrency_limit and freeze there, with no further replication activity, even after the worker rejoins.

Impact

Any environment where a worker holding many blocks goes down non-gracefully (node reboot, OOM, power loss) will permanently disable re-replication cluster-wide until the active master is restarted. Since node reboots routinely exceed the 10-minute lost threshold on GPU machines, this makes the self-healing guarantee of block_replication_enabled unreliable in practice. Data itself is not damaged.

Suggested fixes

1. Add a timeout/reaper for inflight_blocks entries (wire up the existing-but-unused block_replication_retry_interval), releasing the permit and either re-queueing or counting the block as failed.
2. On worker lost/re-registration, cancel/clean that worker's inflight jobs.
3. Worker: always send a result report, including when storage_type is None / the source block can't be opened (report failure instead of silently dropping).
4. Decrement staging_number on all replicate_block() early-return paths, and dedup enqueues against inflight_blocks + queued set so the gauges stay meaningful.

Recovery workaround (for anyone hitting this)

Restart the active master (or force a failover). The scheduler state is purely in-memory; the new Active starts with a clean queue and the post-registration full block report re-detects genuinely under-replicated blocks.

[gangump82]

Second reproduction from a clean state — leak path confirmed end-to-end with worker-side logs

We hit the deadlock a second time, this time starting from a completely clean queue (inflight=0 staging=0, all junk data deleted, fresh Active master), which pins down the exact leak path.

Sequence

1. Rebooted a worker node (again ~11 min > worker_lost_interval) while 6 reader + 4 writer pods ran against FUSE. Business impact: zero read failures, zero write failures (writes routed to surviving workers) — data plane failover is fine.
2. At the 10-min lost mark the master staged ~7048 blocks and inflight shot to exactly 1000 (= block_replication_concurrency_limit) — then froze permanently again. Zero replication lines in the master log. replication_failure_count stayed 0.
3. The lost worker rejoined with its original worker_id (cluster live=8/lost=0). Queue stayed frozen.

Worker-side smoking gun

Every surviving worker (the assigned replication sources) logged ~280–300 error pairs in those minutes, all of this exact shape:

ERROR worker_replication_manager.rs:104 Errors on replicating block: 16928210946. err: ... ERROR: block 16928210946 not exists(curvine-worker/src/worker/block/block_store.rs:263)
ERROR worker_replication_manager.rs:109 Errors on reporting block: 16928210946. err: ... ERROR: missing storage type when reporting replication result for block 16928210946(curvine-worker/src/worker/replication/worker_replication_manager.rs:121)

That is the leak path, confirmed live:

1. Master stages block IDs on worker-lost and assigns a source worker. Many of these block IDs referenced files that had been deleted the day before (block not exists on the assigned source) — the staging path apparently enqueues from stale state without validating the block still exists in current metadata, and with no dedup.
2. The source worker accepts the job (accept_job returns success immediately → master inserts the inflight entry and takes a semaphore permit).
3. The worker fails to open the block (block not exists), so job.storage_type is never set; report_job() then aborts on missing storage type when reporting replication result — the failure report is never sent to the master.
4. The master's inflight entry + OwnedSemaphorePermit leak forever. After 1000 leaks the scheduler blocks silently on acquire_owned().await.

So the worker knows the job failed and even logs it — it just can't tell the master. A one-line fix on the worker side (send a failure report even when storage_type is None) would already prevent the permit leak; the master-side reaper/timeout is still needed for the worker-crash case.

Reproduction recipe (reliable, 2/2 so far)

1. block_replication_enabled = true, cluster with some replicas=2 data.
2. Delete a batch of files (leaves stale block state somewhere the staging path reads).
3. Make any worker unavailable for > worker_lost_interval (e.g. node reboot).
4. Watch replication_inflight_number freeze at block_replication_concurrency_limit with replication_failure_count = 0 and no master log activity; surviving workers spam the two errors above.

Recovery is unchanged: restart the Active master / force failover (in-memory state resets).