[BUG]: Wiped master panics on join with to_commit out of range instead of restoring from snapshot

[bigbigxu]

Describe the bug

After a master node's local metadata (journal / meta) is wiped, rejoining an existing cluster panics in raft-rs instead of downloading a snapshot and recovering. The empty node still has last_index = 0, but it first receives the leader's high commit index.

Observed panic:

2026-09-21 11:41:36.678 panic occurred at /root/.cargo/registry/src/mirrors.aliyun.com-0671735e7cc7f5e7/raft-0.7.0/src/raft_log.rs:292: to_commit 105311900 is out of range [last_index 0], raft_id: 3
backtrace:

This is distinct from #1551. #1551 is a restart crash-loop after a snapshot has already been installed (hs.commit … out of range [0, 0] in RawNode::new). This issue happens during join, on the RaftLog::commit_to path, before snapshot catch-up completes. It is also distinct from #1368, which covers an existing voter whose persisted HardState.commit is beyond the local log/snapshot range.

To Reproduce

1. Run a 3-master HA cluster and write enough metadata that the leader's commit / snapshot index is far above 0 (in this case to_commit = 105311900).
2. Stop one master and wipe its local metadata directories (journal / meta) so it starts as an empty node.
3. Keep the remaining masters healthy so the cluster still has quorum.
4. Start the wiped master and let it rejoin with its original raft id (raft_id: 3 in this report).
5. Observe the node panic during recovery instead of downloading and applying a snapshot.

Expected behavior

A wiped master should catch up via snapshot:

1. Detect that the local log is far behind the leader (last_index = 0 while the leader commit is large).
2. Request / download a snapshot from the leader.
3. Install the snapshot so local last_index / applied advance to the snapshot index.
4. Continue replicating later journal entries and become a healthy follower.

to_commit … out of range [last_index 0] must not be the recovery outcome. A heartbeat or append whose commit is ahead of the local log should trigger snapshot restore, or at least fail with a controlled error, instead of aborting the process.

Screenshots

N/A. Panic log is included above.

OS Version (please complete the following information):

• Occurred at: 2026-09-21 11:41:36
• Component: master / raft journal
• raft-rs: 0.7.0
• raft_id: 3
• to_commit: 105311900
• Local last_index: 0
• Trigger: wipe master metadata, then rejoin the cluster
• Host OS / exact Curvine version / config: not fully specified (log path is Linux /root/.cargo/registry/...)

Additional context

Related code and existing coverage:

• Panic site: raft-0.7.0/src/raft_log.rs:292 (RaftLog::commit_to fatals when to_commit > last_index())
• Characterization test: empty_static_voter_panics_on_leader_commit_above_local_last_index in crates/metadata/curvine-raft/tests/raft_node_test.rs already records that an empty voter panics when a heartbeat advances commit past last_index
• Snapshot install path: gen_apply_snapshot_job in crates/metadata/curvine-raft/src/raft/storage/peer_storage.rs (download snapshot → log_store.apply_snapshot → app_store.apply_snapshot)
• Related but different issues:
  • [BUG]: master crash-loops with "hs.commit … out of range [0, 0]" after a wiped follower rejoins via snapshot #1551: wiped follower crash-loops on restart after snapshot install (hs.commit … out of range [0, 0])
  • [BUG]: Master should fail gracefully when persisted Raft HardState commit is beyond local log range #1368: existing voter panics at startup when persisted HardState.commit is outside the local log/snapshot range

Possible causes (to confirm):

1. The leader sends a MsgHeartbeat / MsgAppend with a high commit before sending a snapshot, and the empty follower runs commit_to while last_index = 0.
2. Snapshot request / download / apply does not complete or is skipped (for example an empty snapshot placeholder is refused), then a later heartbeat still tries to advance commit.

Suggested fix direction:

• Prefer snapshot catch-up for a follower that is too far behind and has no matching local log/snapshot, instead of handing an ahead-of-log commit to raft-rs.
• Before RawNode::step, intercept messages whose commit > last_index and request a snapshot or return a controlled error.
• Add a regression test: wipe one master's journal/meta and rejoin; the node should download a snapshot and recover instead of panicking.