[BUG]: Master restart create_tree drops Spark/Hive partition dirs as duplicate parent edges

[bigbigxu]

Describe the bug

On Master restart, InodeStore::create_tree / build_tree_from_data reports inconsistent RocksDB namespace metadata and then auto-repairs it by dropping edges.

Two failure modes are seen together:

1. Duplicate parent edges for a directory inode (directories are not allowed to have multiple parents / hard links). Both edges share the same parent id but different names. create_tree keeps the first edge it sees and durably deletes the other.
2. Orphaned edges: a parent edge points at a child_id that has no inode record. create_tree drops the edge.

This is not a clean restart. The repair is first-seen-wins and can hide committed Hive/Spark partition directories. In the logs below, _temporary is kept and the partition name (dayno=…, brand=…, hour=…) is dropped.

Typical names in the log match Spark FileOutputCommitter (_temporary, part-*.orc) plus Hive-style partitions (dayno=, brand=, hour=). That strongly suggests the inconsistency is produced on the mkdir/rename/delete path used by job commit, then only becomes visible when Master rebuilds the in-memory tree from CF_EDGES + CF_INODES.

Relevant restore logic:

curvine-master/src/master/meta/store/inode_store.rs
  create_tree -> load_snapshot_data -> build_tree_from_data

• Duplicate directory edges: keep first (parent_id, name), delete_child the rest.
• Orphaned edges: delete_child when data.inodes.remove(child_id) is None.
• Edge key is (parent_id, name) -> child_id (RocksInodeStore::add_child). Two names under the same parent can both point at the same directory inode in RocksDB, even though the in-memory tree cannot represent that.

To Reproduce

Exact local repro is still unknown. Observed sequence:

1. Run Spark/Hive jobs that write partitioned ORC output to Curvine (dynamic partitions + _temporary commit dirs).
2. Restart Master (or restore from checkpoint / rebuild the inode tree).
3. Watch Master logs during create_tree.

Expected behavior

• After a successful Master restart, the namespace in RocksDB and the in-memory tree must be identical and valid:
  • each directory inode has exactly one parent edge
  • every edge points at an existing inode
• Spark/Hive committed partition paths (dayno=…, brand=…, hour=…) must remain visible.
• If metadata is already corrupt, restart should fail loudly or apply a deterministic, correct repair — not silently keep _temporary and drop the committed partition name.

Screenshots

N/A. Master logs (2026-09-21):

2026/09/21 11:02:23.433 WARN inode_store.rs:681 create_tree: directory inode 208610812 has multiple parent edges: keeping parent 208610811 name '_temporary', dropping parent 208610811 name 'dayno=20260917'
2026/09/21 11:02:23.434 WARN inode_store.rs:681 create_tree: directory inode 208610331 has multiple parent edges: keeping parent 208610330 name '_temporary', dropping parent 208610330 name 'brand=OPPO'
2026/09/21 11:02:24.558 WARN inode_store.rs:714 create_tree: orphaned edge detected, parent_id=208610565, edge_name='part-00249-374cb717-1137-4ec5-b942-cb4579f7fa3e.c000.zlib.orc', child_id=208610582 has no inode, dropping edge
2026/09/21 11:02:26.500 WARN inode_store.rs:681 create_tree: directory inode 208610942 has multiple parent edges: keeping parent 208610941 name '0', dropping parent 208610941 name 'dayno=20260917'
2026/09/21 11:02:26.558 WARN inode_store.rs:681 create_tree: directory inode 208610817 has multiple parent edges: keeping parent 208610816 name '_temporary', dropping parent 208610816 name 'hour=09'

OS Version (please complete the following information):

• Component: Master (create_tree on restart / snapshot restore)
• Date: 2026-09-21
• Workload: Spark partitioned ORC write (_temporary + Hive partition dirs)
• Curvine version / commit: unknown (please fill in)

Additional context

Impact:

• Restart itself may continue after WARN + durable delete_child.
• The dropped edge is often the user-visible partition path. After repair, data may only be reachable under _temporary / 0, so queries against dayno= / brand= / hour= miss files.
• Orphaned part-*.orc edges mean committed output files can disappear from the namespace even if blocks still exist.

Likely write-path bugs to investigate (not confirmed):

1. Directory rename under the same parent (_temporary → dayno=…) adds the new CF_EDGES key but fails to delete the old name. delete_child of a missing key is a no-op, so a mismatched src edge name would leave two edges to the same inode.
2. apply_rename deletes the store edge by src_inp.name(), while the in-memory tree deletes by src_inode.name(). If those diverge, live list_status can look fine while RocksDB already has two directory edges; restart then exposes it.
3. Journal mkdir/rename/delete replay vs snapshot restore leaving a directory inode with two names, or a file edge whose inode was already deleted.
4. First-seen-wins repair is non-deterministic (HashMap / scan order) and prefers whichever name is scanned first, which in this incident was _temporary rather than the committed partition.

Suggested fix direction:

• Find and stop the writer that inserts a second directory parent edge (rename / mkdir / journal replay).
• Make create_tree repair choose the committed/canonical name (match inode.name() / parent_id()), not the first scanned edge.
• Add regression tests: same-parent directory rename, Spark-like _temporary → partition rename, crash between delete-old-edge and add-new-edge, journal replay of mkdir+rename, orphaned file edges after delete.

I can follow up with cluster version, full restart log (create_tree: edges=… orphaned=… repairs=…), and whether ls of the partition path failed after restart.