feat(sync): 支持 RDB 临时文件压缩#1055
Closed
DCjanus wants to merge 1 commit into
Closed
Conversation
Assisted-by: Codex:gpt-5.5
Member
|
不希望有压缩,RDB 一般比较小,该功能会带来额外复杂度。 |
Contributor
Author
了解,我们之前在内部 fork 版本里实现了类似功能,因为我们有些 GB 级别的 RDB。 不过这确实是我们内部服务治理的问题,看起来不是通用需求 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
sync_reader当前会先把源端全量 RDB 写入本地临时dump.rdb文件,再从该文件解析并写入目标端。对于较大的 RDB,这会占用较多本地磁盘空间,也会增加全量同步阶段的磁盘 I/O 压力。这个 PR 先实现 #1053 中 RDB 临时文件压缩的部分。AOF 临时文件涉及持续写入、读取、轮转和 offset 管理,后续可以在统一临时存储抽象后继续推进。
改动内容
advanced.temp_file_compression_rdb,用于控制sync_reader内部 RDB 临时文件压缩。none、snappy、zstd三种取值。snappy,在降低临时文件磁盘占用的同时尽量控制 CPU 开销。rdb_reader.filepath不受影响,仍按原始 RDB 文件读取。默认压缩算法
这里有一个希望 reviewer 重点关注的取舍:默认把 RDB 临时文件从原始格式改为
snappy压缩是否合适。从目前的本地测试结果看,
snappy压缩速度很快,CPU 开销较低,因此默认启用看起来是可以接受的行为变化。对最终使用者来说,RedisShake 4.x 本身不支持 checkpoint/resume;sync_reader启动时会清理并重建当前 reader 的临时目录,RDB 解析完成后也会删除临时文件。因此这个默认值变化不会影响最终同步数据语义,也不会涉及旧进程 RDB/AOF buffer 的复用。如果用户需要保留原始临时文件格式,可以显式配置:
Related to #1053
验证
go test ./...