Repository navigation
Conversation
…ow it in cluster - IoTConsensus snapshot receive folder now always uses MinFolderOccupiedSpaceFirstStrategy, balancing received files by least occupied space independent of the global dn_multi_dir_strategy. - Allow MinFolderOccupiedSpaceFirstStrategy in cluster mode (CLUSTER_ALLOWED_MULTI_DIR_STRATEGIES) so it can be configured for DataNodes. - Fix NPE in the dn_multi_dir_strategy validation rollback path when the previous value is null, so the informative RuntimeException surfaces. - IT: add DataNodeConfig.setDnMultiDirStrategy and exercise the strategy in IoTDBRegionMigrateWithDeletionMultiDataDirIT.
When IoTConsensus spreads snapshot fragments across recv folders, disable keepSameDiskWhenLoading so fileTarget groups companion files on one data dir.
…s receiver receiveSnapshotFragment now selects the receive folder once per TsFile group (fileKey = file name before the first '.', matching SnapshotLoader) via a per-snapshotId ConcurrentHashMap.computeIfAbsent, so a TsFile and its .resource/.mods companions land in the same receive folder and the load phase relinks them within one data dir instead of falling back to a cross-disk copy. The mapping is dropped once the snapshot is loaded or cleaned up. Convert IoTDBRegionMigrateWithDeletionMultiDataDirIT to the tree model and add the table-model twin IoTDBRegionMigrateWithDeletionMultiDataDirTableIT (@TableClusterIT) so both cluster CI profiles cover the snapshot mods transfer.
jt2594838
approved these changes
Jun 25, 2026
MileaRobertStefan
pushed a commit
to MileaRobertStefan/iotdb
that referenced
this pull request
Jun 26, 2026
apache#18020) * Use MinFolderOccupiedSpaceFirstStrategy for IoTConsensus recv and allow it in cluster - IoTConsensus snapshot receive folder now always uses MinFolderOccupiedSpaceFirstStrategy, balancing received files by least occupied space independent of the global dn_multi_dir_strategy. - Allow MinFolderOccupiedSpaceFirstStrategy in cluster mode (CLUSTER_ALLOWED_MULTI_DIR_STRATEGIES) so it can be configured for DataNodes. - Fix NPE in the dn_multi_dir_strategy validation rollback path when the previous value is null, so the informative RuntimeException surfaces. - IT: add DataNodeConfig.setDnMultiDirStrategy and exercise the strategy in IoTDBRegionMigrateWithDeletionMultiDataDirIT. * Fix multi-recv snapshot loading to keep tsfile and mods together When IoTConsensus spreads snapshot fragments across recv folders, disable keepSameDiskWhenLoading so fileTarget groups companion files on one data dir. * Group snapshot companion files into one receive folder on IoTConsensus receiver receiveSnapshotFragment now selects the receive folder once per TsFile group (fileKey = file name before the first '.', matching SnapshotLoader) via a per-snapshotId ConcurrentHashMap.computeIfAbsent, so a TsFile and its .resource/.mods companions land in the same receive folder and the load phase relinks them within one data dir instead of falling back to a cross-disk copy. The mapping is dropped once the snapshot is loaded or cleaned up. Convert IoTDBRegionMigrateWithDeletionMultiDataDirIT to the tree model and add the table-model twin IoTDBRegionMigrateWithDeletionMultiDataDirTableIT (@TableClusterIT) so both cluster CI profiles cover the snapshot mods transfer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
IoTConsensus receive folder strategy
The IoTConsensus snapshot receive folder always uses
MinFolderOccupiedSpaceFirstStrategy(balances by least-occupied space), independent ofdn_multi_dir_strategy.Allow the strategy in cluster mode + NPE fix
Adds
MinFolderOccupiedSpaceFirstStrategytoCLUSTER_ALLOWED_MULTI_DIR_STRATEGIES. Also fixes an NPE in thedn_multi_dir_strategyvalidation rollback path (IoTDBDescriptor): the previous value can benullat startup, so.trim()masked the real "cluster mode only allows ..." error.Keep a TsFile and its companions together across multiple data dirs
With multiple data dirs, the fragments of one TsFile (
.tsfile/.tsfile.resource/.tsfile.mods2) can be received into different disks, causing cross-disk copies (or lost mods) at load. Two changes keep a group together:IoTConsensusServerImpl#receiveSnapshotFragment): selects the receive folder once per TsFile group (key = file name up to the first., matchingSnapshotLoader) via a per-snapshotIdConcurrentHashMap#computeIfAbsent; the map is cleared on load/cleanup.SnapshotLoader#loadSnapshotFromMultipleDirs): a sharedfileTargetrelinks a TsFile and its companions into the same data dir even when received across different disks.Integration tests
DataNodeConfig#setDnMultiDirStrategy.IoTDBRegionMigrateWithDeletionMultiDataDirIT(tree) andIoTDBRegionMigrateWithDeletionMultiDataDirTableIT(table,@TableClusterIT): 1C3D + IoTConsensus + multiple data dirs; a deletion must survive region migration.This PR has:
Key changed/added classes in this PR
IoTConsensusServerImplSnapshotLoaderDataRegionConsensusImplIoTDBConfig/IoTDBDescriptorDataNodeConfig/MppDataNodeConfig/RemoteDataNodeConfigIoTDBRegionMigrateWithDeletionMultiDataDirIT/IoTDBRegionMigrateWithDeletionMultiDataDirTableIT