docs: outputs: file: add log rotation support and configuration examples#2572
docs: outputs: file: add log rotation support and configuration examples#2572SagiROosto wants to merge 1 commit into
Conversation
📝 WalkthroughWalkthroughDocumentation for the File output plugin adds log rotation details: new parameters ( ChangesFile Output Plugin Log Rotation Documentation
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
SagiROosto
force-pushed
the
aws-iot-creds-docs
branch
from
713685c to
2fc7093
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pipeline/outputs/file.md`:
- Line 123: Update the documentation string that describes the rotation filename
pattern used when files_rotation is enabled: replace the incorrect timestamp
token `YYYMMDD_HHMMSS` with the correct `YYYYMMDD_HHMMSS` in the sentence that
explains how files are renamed (the block referencing `files_rotation` and
`max_size` and the pattern `<filename>.<YYYYMMDD_HHMMSS_XXXXXXXX>`).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
SagiROosto
force-pushed
the
aws-iot-creds-docs
branch
from
2fc7093 to
3f1a696
Compare
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
eschabell
left a comment
eschabell
left a comment
There was a problem hiding this comment.
@SagiROosto see changes to be made first before review passes.
|
|
||
| The plugin retains up to `max_files` rotated files per output file. When the limit is reached, the oldest rotated files are deleted automatically. | ||
|
|
||
| Log rotation works with all supported output [formats](#format), including `out_file`, plain, CSV, LTSV, template, and msgpack. File operations are thread-safe, so rotation can be used alongside multiple [workers](../../administration/multithreading.md#outputs). |
There was a problem hiding this comment.
@SagiROosto can you put back-ticks around all the formats listed here:
- plain
- CSV
- LTSV
- template
- msgpack.
There was a problem hiding this comment.
🧹 Nitpick comments (1)
pipeline/outputs/file.md (1)
123-123: ⚡ Quick winImprove grammar and consistency for the timestamp description.
The phrase "The YYYYMMDD_HHMMSS is timestamp of the time when the rotation occurred" is missing an article and should use backticks for consistency with other technical terms in the sentence.
📝 Suggested improvement
-When `files_rotation` is enabled, the plugin monitors the size of each output file. Once a file exceeds `max_size`, the plugin rotates the file by renaming it with a timestamp suffix in the format `<filename>.<YYYYMMDD_HHMMSS_XXXXXXXX>`. The YYYYMMDD_HHMMSS is timestamp of the time when the rotation occurred, and `XXXXXXXX` is a random identifier to guarantee unique filenames if multiple rotations happen within the same second. +When `files_rotation` is enabled, the plugin monitors the size of each output file. Once a file exceeds `max_size`, the plugin rotates the file by renaming it with a timestamp suffix in the format `<filename>.<YYYYMMDD_HHMMSS_XXXXXXXX>`. The `YYYYMMDD_HHMMSS` part is the rotation timestamp, and `XXXXXXXX` is a random identifier to guarantee unique filenames if multiple rotations happen within the same second.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pipeline/outputs/file.md` at line 123, The sentence describing the rotation timestamp is missing an article and inconsistent formatting; update the text that explains the rotation suffix `<filename>.<YYYYMMDD_HHMMSS_XXXXXXXX>` (referencing `files_rotation` and `max_size`) to say: "The `YYYYMMDD_HHMMSS` is the timestamp of the time when the rotation occurred" (use backticks around `YYYYMMDD_HHMMSS` and `XXXXXXXX` for consistency) and ensure the full format is shown as `<filename>.<YYYYMMDD_HHMMSS_XXXXXXXX>` with `XXXXXXXX` described as a random identifier to guarantee unique filenames if multiple rotations happen within the same second.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@pipeline/outputs/file.md`:
- Line 123: The sentence describing the rotation timestamp is missing an article
and inconsistent formatting; update the text that explains the rotation suffix
`<filename>.<YYYYMMDD_HHMMSS_XXXXXXXX>` (referencing `files_rotation` and
`max_size`) to say: "The `YYYYMMDD_HHMMSS` is the timestamp of the time when the
rotation occurred" (use backticks around `YYYYMMDD_HHMMSS` and `XXXXXXXX` for
consistency) and ensure the full format is shown as
`<filename>.<YYYYMMDD_HHMMSS_XXXXXXXX>` with `XXXXXXXX` described as a random
identifier to guarantee unique filenames if multiple rotations happen within the
same second.
|
|
||
| The File output plugin supports size-based log rotation. | ||
|
|
||
| When `files_rotation` is enabled, the plugin monitors the size of each output file. Once a file exceeds `max_size`, the plugin rotates the file by renaming it with a timestamp suffix in the format `<filename>.<YYYYMMDD_HHMMSS_XXXXXXXX>`. The YYYYMMDD_HHMMSS is timestamp of the time when the rotation occurred, and `XXXXXXXX` is a random identifier to guarantee unique filenames if multiple rotations happen within the same second. |
There was a problem hiding this comment.
@SagiROosto also add back-ticks to the "YYYYMMDD_HHMMSS" for consistency.
eschabell
added
waiting-for-user
SagiROosto
force-pushed
the
aws-iot-creds-docs
branch
from
3f1a696 to
63477a5
Compare
patrick-stephens
left a comment
patrick-stephens
left a comment
There was a problem hiding this comment.
Some queries around bad-path behaviours but nothing blocking, it would be nice to resolve them though for the inevitable failures that people will have.
|
|
||
| The File output plugin supports size-based log rotation. | ||
|
|
||
| When `files_rotation` is enabled, the plugin monitors the size of each output file. Once a file exceeds `max_size`, the plugin rotates the file by renaming it with a timestamp suffix in the format `<filename>.<YYYYMMDD_HHMMSS_XXXXXXXX>`. The `YYYYMMDD_HHMMSS` is timestamp of the time when the rotation occurred, and `XXXXXXXX` is a random identifier to guarantee unique filenames if multiple rotations happen within the same second. |
There was a problem hiding this comment.
Where does the timestamp come from? Is it UTC or local timezone based?
Be nice to have a controllable format for the future too.
There was a problem hiding this comment.
@SagiROosto can you answer this for @patrick-stephens ?
There was a problem hiding this comment.
The suffix uses local wall-clock time, not UTC: time(NULL) → localtime_r() → strftime(..., "%Y%m%d_%H%M%S", ...), then _ plus 8 lowercase hex digits from flb_random_bytes() (fallback if RNG fails) so multiple rotations in the same second don’t collide.
Example: cpu.log.20260512_134500_a1b2c3d4 (and .gz when gzip is enabled).
There is no configurable timestamp format in the current implementation—fixed YYYYMMDD_HHMMSS + random tail. Agree a future option (e.g. UTC vs local, strftime template) would be useful; happy to track as a follow-up if we want it called out in the docs “limitations” section.
eschabell
left a comment
eschabell
left a comment
There was a problem hiding this comment.
@SagiROosto can you answer the tagged questions for @patrick-stephens ?
|
|
||
| The File output plugin supports size-based log rotation. | ||
|
|
||
| When `files_rotation` is enabled, the plugin monitors the size of each output file. Once a file exceeds `max_size`, the plugin rotates the file by renaming it with a timestamp suffix in the format `<filename>.<YYYYMMDD_HHMMSS_XXXXXXXX>`. The `YYYYMMDD_HHMMSS` is timestamp of the time when the rotation occurred, and `XXXXXXXX` is a random identifier to guarantee unique filenames if multiple rotations happen within the same second. |
There was a problem hiding this comment.
@SagiROosto can you answer this for @patrick-stephens ?
|
Done, see my replies. @eschabell @patrick-stephens |
|
Can we capture the clarifications in the actual docs as I know at least some of those questions will likely come up and better than tracking down this PR? |
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
SagiROosto
force-pushed
the
aws-iot-creds-docs
branch
from
63477a5 to
ed1836f
Compare
|
@patrick-stephens I've added a |
There was a problem hiding this comment.
🧹 Nitpick comments (2)
pipeline/outputs/file.md (2)
123-124: ⚡ Quick winAdd clarification about flush-driven rotation behavior.
In the PR discussion, you explained that rotation is flush-driven (size check occurs at the start of each flush, not periodically), and
@patrick-stephensrequested this be captured in the documentation. Consider adding a sentence clarifying that the size check happens when Fluent Bit is about to write new data to the file, and that the active file may temporarily exceedmax_sizeuntil the next flush triggers rotation.For example: "The size check occurs before each flush operation. The active file may temporarily exceed
max_sizebetween flushes, but will be rotated before the next chunk is written."Based on learnings, this clarification addresses a reviewer's explicit request to document the flush-driven rotation behavior explained in the PR discussion.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pipeline/outputs/file.md` around lines 123 - 124, Add a short clarifying sentence about flush-driven rotation to the documentation near files_rotation/max_size: state that the size check is performed at the start of each flush (when Fluent Bit is about to write a new chunk), so the active file can temporarily exceed max_size between flushes but will be rotated before the next write; update the paragraph that describes the timestamped rename behavior to include this flush-driven detail and use the terms "flush", "files_rotation", and "max_size" so it's discoverable.
17-18: ⚡ Quick winConsider documenting configuration validation behavior.
Based on the PR discussion, invalid
max_filesvalues (<= 0) cause startup failures, and invalidmax_sizestrings may not be rejected but result in rotation never triggering. Adding a brief note about validation behavior would help users troubleshoot configuration errors.For example, you could add a sentence to the
max_filesdescription: "If set to0or negative, Fluent Bit will fail to start."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pipeline/outputs/file.md` around lines 17 - 18, Update the descriptions for the `max_files` and `max_size` configuration entries (both under `files_rotation`) to document validation behavior: for `max_files` add a sentence such as "If set to 0 or negative, Fluent Bit will fail to start." for `max_size` add a sentence noting accepted suffixes (`k`, `m`, `g`) and that malformed values may be ignored causing rotation to never trigger; keep both notes scoped with the existing "Only applies when `files_rotation` is enabled" wording so users know when validation applies.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@pipeline/outputs/file.md`:
- Around line 123-124: Add a short clarifying sentence about flush-driven
rotation to the documentation near files_rotation/max_size: state that the size
check is performed at the start of each flush (when Fluent Bit is about to write
a new chunk), so the active file can temporarily exceed max_size between flushes
but will be rotated before the next write; update the paragraph that describes
the timestamped rename behavior to include this flush-driven detail and use the
terms "flush", "files_rotation", and "max_size" so it's discoverable.
- Around line 17-18: Update the descriptions for the `max_files` and `max_size`
configuration entries (both under `files_rotation`) to document validation
behavior: for `max_files` add a sentence such as "If set to 0 or negative,
Fluent Bit will fail to start." for `max_size` add a sentence noting accepted
suffixes (`k`, `m`, `g`) and that malformed values may be ignored causing
rotation to never trigger; keep both notes scoped with the existing "Only
applies when `files_rotation` is enabled" wording so users know when validation
applies.
eschabell
left a comment
eschabell
left a comment
There was a problem hiding this comment.
@SagiROosto thanks for the updates, good to go now once the code PR merges.
eschabell
removed
waiting-for-user
3 participants
This is the docs PR for fluent/fluent-bit#11352 about
out_filelogrotate featureSummary by CodeRabbit