[BUG]: Master restart aborts on duplicate inode allocation during follower replay

[bigbigxu]

Describe the bug

Master cannot restart. Raft apply stops fatally while replaying a committed Mkdir journal entry:

refusing duplicate inode allocation during follower replay:
inode_id=208477849, last_inode_id=208677538, journal=Mkdir(...)

The guard in JournalLoader rejects any follower-replay allocation whose inode_id <= last_inode_id. In this incident the allocated id is 199,689 smaller than the current high-water mark, so this is not a near-miss of the next id. After the error, raft_node.rs stops the node and master stays down.

This looks like a restart/replay availability failure. Either:

1. An older/duplicate Mkdir is still present in the Raft log after last_inode_id has already advanced (snapshot restore, overlapping log, or a previously allocated id being replayed), and the fail-fast check treats that as fatal; or
2. A real duplicate inode allocation was committed earlier (see related [BUG]: Master can reuse op_id/inode_id after leader change when FSM apply lags committed log #957), and the fix(raft): fence leader promotion after metadata apply #1557 guard now correctly detects it — but then master has no way to restart or recover.

Either way, a master that already has the metadata on disk cannot come back.

Relevant code:

• curvine-master/src/master/journal/journal_loader.rs (follower replay duplicate-inode check, currently around line 273)
• curvine-master/src/master/journal/entry.rs (JournalEntry::allocated_inode_id() for Mkdir / CreateFile / Symlink)
• crates/metadata/curvine-raft/src/raft/raft_node.rs (fatal apply → raft node stop)
• Introduced by fix(raft): fence leader promotion after metadata apply #1557 (fix(raft): fence leader promotion after metadata apply)

To Reproduce

1. Run a master cluster under a Spark / Hive warehouse workload that creates many staging directories (.spark-staging-* / _temporary/attempt_*).
2. Restart master (or let a follower replay committed logs after snapshot restore / role change).
3. Apply hits a committed Mkdir whose dir.id is already <= last_inode_id.
4. Raft apply returns the duplicate-allocation error and the raft node stops.

We do not yet have a minimal standalone repro. The production log below is enough to identify the invariant and the restart failure.

Expected behavior

• Master restart must not stay down solely because a replayed create carries an inode id that is already covered by last_inode_id / the restored tree, if that entry is historical or already applied.
• If the log really contains a duplicate allocation assigned to a different namespace object, fail with a clear corruption message and a documented recovery path. Do not leave the process in a loop of crash-on-replay.
• inode_id / last_inode_id remain monotonic at cluster scope. A committed log should never assign the same inode id to two different objects (related to [BUG]: Master can reuse op_id/inode_id after leader change when FSM apply lags committed log #957).
• Distinguish “already applied / snapshot overlap” from “true duplicate reuse”.

Screenshots

N/A. Production error log:

2026/09/17 10:36:03.375 ERROR raft_node.rs:271 raft node stop:
[raft-rpc] ERROR: refusing duplicate inode allocation during follower replay:
inode_id=208477849, last_inode_id=208677538,
journal=Mkdir(MkdirEntry {
  op_id: 789738094,
  rpc_id: 0,
  path: "/com1-hdfs/warehouse/cn_ib_dw/ib_nebula.db/ads_asscreen_intent_user_order_bitmap_inc_d/.spark-staging-f4796596-aeeb-48aa-a66a-ca6582a48079/_temporary/0/_temporary/attempt_202609171026008995989504865249896_0004_m_000224_712",
  dir: InodeDir {
    id: 208477849,
    parent_id: -1,
    mtime: 1789612012354,
    atime: 1789612012354,
    nlink: 2,
    storage_policy: StoragePolicy { storage_type: Disk, ttl_ms: 0, ttl_action: None, ufs_mtime: 0, state: Cv },
    features: DirFeature { x_attr: {}, acl: AclFeature { owner: "", group: "", mode: 493 } },
    children: Map({})
  }
})
(curvine-master/src/master/journal/journal_loader.rs:273)

OS Version (please complete the following information):

• Component: Curvine master (Raft journal replay)
• Observed: 2026-09-17
• Workload: Spark staging dirs under a Hive warehouse path (*.db/.../.spark-staging-*/_temporary/...)
• Source: journal_loader.rs follower-replay guard from fix(raft): fence leader promotion after metadata apply #1557

Additional context

Observed anomalies in the failing entry:

• inode_id=208477849 vs last_inode_id=208677538 (inode_id << last_inode_id)
• parent_id: -1 on the Mkdir inode (unexpected for a nested Spark staging path)
• rpc_id: 0

ignore_reply_error can log-and-skip this error, but it defaults to false, so production restart still dies.

Related:

• [BUG]: Master can reuse op_id/inode_id after leader change when FSM apply lags committed log #957 — duplicate op_id / inode_id after leader change when FSM apply lags committed log
• fix(raft): fence leader promotion after metadata apply #1557 — added this fail-fast check and the follower replay test follower_replay_rejects_duplicate_allocated_inode_id

Suggested investigation:

1. Confirm whether the Mkdir inode already exists in the restored tree / snapshot under the same path (safe skip) vs a different path (true duplicate).
2. Check snapshot compact vs remaining Raft log overlap: restore sets last_inode_id to the tree max, then replay of older create entries will always trip inode_id <= last_inode_id.
3. Explain parent_id: -1 on this MkdirEntry.
4. If the check stays fail-fast, add a recovery path so master can start after snapshot+log overlap without disabling all replay errors.
5. Add a regression that restores a snapshot (high last_inode_id) and then replays an older committed Mkdir that is already in the tree; restart must succeed.