feat(aggregate transform): Add support for event timestamp-based aggregation by kaarolch · Pull Request #24421 · vectordotdev/vector

kaarolch · 2025-12-30T09:07:12Z

Summary

This PR adds event-time aggregation support to the aggregate transform, addressing issues where metrics with different source timestamps but the same processing time are incorrectly aggregated together.

I made this PR to address some gaps in #23694. Thank you @adiwab for providing initial implementation.

Problem

Currently, the aggregate transform uses system processing time to bucket metrics. This causes issues when:

Multiple events have different source timestamps but arrive at the same processing time, or have the same source timestamp and after processing have different tag series.
In the first case they get aggregated into the same bucket and sent downstream with identical timestamps and in the second one they could be shipped in different batch.
Systems like Datadog overwrite the earlier values, resulting in data loss

Solution

Introduced an optional event_time configuration block:

Omitted (default): existing system-time behavior, maintains backward compatibility
Present: metrics are grouped into buckets based on their timestamps rather than when they're processed, with watermark-based out-of-order event rejection

Key Changes

Configuration Options:

Event-time aggregation is enabled by presence of the optional event_time block (no separate on/off flag). All sub-fields are optional with sensible defaults:

event_time.allowed_lateness_ms: Grace period for accepting late-arriving events, in milliseconds (default: 0)
event_time.missing_timestamp: How to handle metrics without a timestamp — drop (default) or use_system_time
event_time.max_future_ms: Maximum allowed future timestamp drift before an event is rejected as clock-skewed (default: 10000ms)

Implementation:

Event-time bucketing based on metric timestamps rounded down to interval_ms boundaries (Euclidean division, correct for pre-epoch timestamps too)
Watermark tracking to identify and reject out-of-order events past the grace period, enforced at record time (not just at flush)
Support for all aggregation modes
Separate bucket storage (event_time_buckets, event_time_prev_buckets, event_time_multi_buckets) for event-time mode, with event_time_prev_buckets bounded to a small rolling window and holding only MetricData (no EventMetadata) so Diff mode's delta retention doesn't hold finalizers/acknowledgements past emission
EventMetadata (including finalizers) is merged — not dropped — whenever a sample is superseded by timestamp selection in Auto/Latest/Diff modes
Dropped event metrics via AggregateEventDropped internal event, counted through component_discarded_events_total

Vector configuration

api:
  enabled: true
log_schema:
  level: debug
sources:
  http_metrics:
    type: http_server
    address: "0.0.0.0:8080"
    decoding:
      codec: influxdb
    path: "/api/v1/write"
transforms:
  aggregate_metrics:
    type: aggregate
    inputs:
      - http_metrics
    interval_ms: 5000
    event_time:
      allowed_lateness_ms: 5000
      missing_timestamp: use_system_time
      max_future_ms: 60000
sinks:
  console_debug:
    type: console
    inputs:
      - aggregate_metrics
    encoding:
      codec: json
    buffer:
      max_events: 10000
      type: memory
      when_full: drop_newest

How did you test this PR?

The aggregate transform's unit test suite covers 33 tests: the original 17 system-time tests (unchanged, ensuring no regression to default behavior) plus 16 event-time-specific tests, including:

Bucket boundary math (interval alignment, pre-epoch/negative timestamps via Euclidean division)
Watermark-based out-of-order/late-event rejection, both at record time and at flush time, including once a bucket has already been emitted
Multiple open time buckets flushing independently
Missing-timestamp handling for both drop and use_system_time
Future-timestamp rejection and clock-skew edge cases (including i64/u64 boundary values that previously risked overflow or panics)
Absolute "latest by event timestamp" selection (including mixed incremental/absolute kind changes) in Auto/Latest modes
Diff mode previous-bucket retention (bounded rolling window) vs. non-Diff modes retaining nothing
EventMetadata/finalizer merging on latest-selection, and finalizer release after Diff bucket retention
Config parsing of the event_time block and its documented literals
Graceful draining of open event-time buckets on shutdown/topology reload

I've used Sonnet 4.5 to create some scripts that push influxdb metrics to vector with multiple values:

[TEST 1] Basic Aggregation - 3 buckets
============================================================

Bucket 1 (timestamp 07:29:11):
  ✓ test_counter=10.0 @ 07:29:11
  ✓ test_counter=20.0 @ 07:29:12
  ✓ test_counter=30.0 @ 07:29:13
  Expected aggregate: 60.0

Bucket 2 (timestamp 07:29:21):
  ✓ test_counter=10.0 @ 07:29:21
  ✓ test_counter=20.0 @ 07:29:22
  ✓ test_counter=30.0 @ 07:29:23
  Expected aggregate: 60.0

Bucket 3 (timestamp 07:29:31):
  ✓ test_counter=10.0 @ 07:29:31
  ✓ test_counter=20.0 @ 07:29:32
  ✓ test_counter=30.0 @ 07:29:33
  Expected aggregate: 60.0

[TEST 2] Out-of-Order Rejection
============================================================

Bucket 1 (timestamp 07:29:31):
  ✓ test_ooo=10.0 @ 07:29:31
  ✓ test_ooo=20.0 @ 07:29:32
  Expected aggregate: 30.0

Bucket 2 (timestamp 07:29:41):
  ✓ test_ooo=15.0 @ 07:29:41
  ✓ test_ooo=25.0 @ 07:29:42
  Expected aggregate: 40.0

Out-of-order event (timestamp 07:29:33):
  Sending to already-flushed bucket - should be DROPPED
  ✓ test_ooo=999.0 @ 07:29:33

  Check Vector output - 999.0 should NOT appear!

[TEST 3] Multiple Buckets (10s span = 2 buckets)
============================================================

Sending events across 10 seconds:
  ✓ test_multi=100.0 @ 07:29:46
  ✓ test_multi=200.0 @ 07:29:48
  ✓ test_multi=300.0 @ 07:29:51
  ✓ test_multi=400.0 @ 07:29:53

  Expected:
    Bucket 1 (0-5s): 300.0
    Bucket 2 (5-10s): 700.0

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

Resolves conflict in src/transforms/aggregate.rs by integrating upstream's InnerMode refactor (prev_map/multi_map moved into enum variants) with the event-time aggregation feature (TimeSource, event_time_buckets, watermark-based flushing). Made-with: Cursor

kaarolch · 2026-03-20T12:55:02Z

I've added more test around event based timestamp.

kaarolch · 2026-03-21T16:47:47Z

@pront I saw you were active in the related issue Can you look on above PR?

kaarolch · 2026-04-27T06:33:10Z

Is there any recommendation for this PR, I know PR has 1,2k new lines but it's hard to split them to smaller changes.

syedg1 · 2026-04-30T19:04:05Z

@kaarolch — heads up, while running this PR's build in production we hit a memory leak in event_time_prev_buckets. It's read in only one place (the flush loop, gated on AggregationMode::Diff), but on flush it's populated unconditionally, and the eviction (retain) is also Diff-only. So every non-Diff aggregator (including the default Auto mode for counters/gauges) moves a HashMap<MetricSeries, MetricEntry> into event_time_prev_buckets per flush interval and never removes it.

Production impact we measured: ~140 MB/hour of growth per pod with mode: Auto, time_source: EventTime, interval_ms: 10000 at a few hundred counters/sec — ~2.3 GB per pod over 16 hours, on a trajectory toward OOM.

I drafted a fix that scopes both the insert and the retain to Diff mode, plus two regression tests (one verifying event_time_prev_buckets stays empty in non-Diff modes after 50 flushes, one verifying Diff mode still retains the small rolling window it needs). All 31 tests in transforms::aggregate pass.

PR is here: kaarolch#1 — opened against your events_based_aggr branch so it can land inside this PR before merge upstream. Happy to iterate on it if you'd prefer a different shape; just wanted to get it in front of you given the production signal.

pront · 2026-05-01T13:54:33Z

@codex review

pront · 2026-05-01T13:55:46Z

Is there any recommendation for this PR, I know PR has 1,2k new lines but it's hard to split them to smaller changes.

Hey @kaarolch, we have a pretty big backlog. So large PRs take even longer to be review and approved. I kicked off a codex review for now, please fix any issues that may arise there.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8574e57ba1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Addresses several bugs surfaced during review of the event-time aggregation feature: * Stop populating `event_time_prev_buckets` in non-`Diff` modes. The map is read only by `Diff` and was previously inserted (and not evicted) on every flush in `Auto`/`Sum`/`Latest`/etc., growing memory linearly with (unique series in interval) x (intervals since startup). * Drain remaining event-time buckets when the input stream closes. `flush_event_time_buckets` now accepts a `force` flag and a new `flush_final` entry-point is wired into the input-closed arm so in-flight metrics in still-open windows are emitted on shutdown or topology reload, matching system-time semantics. * Reject events for already-emitted windows. Watermark now records the exclusive end (`bucket_key + interval_ms`) of the highest flushed bucket, and `is_too_late` no longer subtracts `allowed_lateness_ms`. `allowed_lateness_ms` keeps its role of delaying bucket close at flush time; once a window is emitted it stays closed. This prevents late events from re-creating closed buckets and emitting duplicate partial aggregates. * Drop events whose (kind, value) is incompatible with the configured mode (for example an `Incremental` event arriving at a `Mean` aggregator) without materialising a bucket. The previous code path always created an empty `event_time_buckets` entry, which then flushed and advanced the watermark, silently rejecting valid in-order events for earlier buckets. Compatibility is now decided up front by `will_be_stored`, dropped events emit `AggregateEventDropped` (rather than the misleading `AggregateEventRecorded`), and `event_time_multi_buckets` is only touched in `Mean`/`Stdev` mode. Adds tests for edge-case behaviour, event-time `Mean` and `Stdev` happy-path flushing, plus an updated changelog entry describing shipped behaviour. Co-authored-by: Cursor <cursoragent@cursor.com>

kaarolch · 2026-05-04T08:29:26Z

@syedg1 I saw your PR to my branch was closed? I've check your PR and try to address extra edge cases in the last commit.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0942121d68

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ca2c616d24

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

… event-time mode Codex flagged that when event-time Auto mode receives an Absolute sample and then an Incremental sample for the same series/bucket, the Incremental arm's kind-mismatch branch in record_sum_in_map replaced the stored tuple outright, discarding the Absolute sample's EventMetadata. With acknowledgements enabled, that sample's finalizers resolved before the emitted aggregate was actually finalized. Merge metadata unconditionally before conditionally replacing the stored value, mirroring the reverse (Incremental-to-Absolute) path, which already merged correctly. Fold the new regression coverage into the existing event_time_auto_replaces_on_kind_change_before_timestamp_compare test (renamed to event_time_auto_handles_kind_switch_in_both_directions) rather than adding a new test function, keeping the event-time suite at 33 tests. Also trim a few comments in event_time.rs that either narrated the following line or restated the code below them, tightening style parity with transform.rs's terser precedent for equivalent logic. Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 12e5179630

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Codex flagged that record_count/record_count_in_map gated the counter increment on `existing.kind == data.kind`, but Count is documented to count both Absolute and Incremental metrics for a series. When a series mixed kinds within one window (system-time) or one bucket (event-time), the second sample's kind mismatch only emitted AggregateUpdateFailed and was silently dropped from the total instead of counted, undercounting the result. `MetricData::update` never inspects kind — only the stored Counter/Counter value types must match, which they always do here — so drop the kind gate and always attempt the update; metadata is now always merged too, matching the other modes' merge-first pattern. Fix applied to both transform.rs (system-time) and event_time.rs, since the two are identical implementations of the same documented mode. Extend count_agg with a mixed-kind case and add event_time_count_counts_mixed_kinds_in_same_bucket to cover event-time. Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fa3dd16f91

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Clippy's missing_const_for_fn lint is denied in vector and would fail Check clippy once the fork-PR Test Suite workflow is approved. Co-authored-by: Cursor <cursoragent@cursor.com>

…latest selection When select_latest_by_event_timestamp replaces MetricData with a newer sample, the previous EventMetadata was left as the merge base. Because EventMetadata::merge never overwrites attribution fields such as source_type or datadog_origin_metadata, Latest/Diff/absolute Auto could emit the winning value attributed to the losing sample. On replacement, use the incoming metadata as the base and merge the retained sample's metadata into it so finalizers are still preserved. Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9a0027ca80

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…time replacement EventMetadata::merge never overwrites attribution fields such as source_type or datadog_origin_metadata. Any path that replaces retained MetricData must therefore also make the incoming metadata the merge base, or the emitted value is attributed to the superseded sample. Introduce replace_entry and use it for Latest/Diff timestamp selection, Auto kind switches, failed same-kind updates, and Max/Min winners / kind-mismatch replacements. Consolidate the Auto kind-switch coverage into the existing metadata/finalizer test so the suite stays at 33 tests. Co-authored-by: Cursor <cursoragent@cursor.com>

…d helpers Reduce overlapping cases and LoC while keeping the behavioral checks, including both Latest arrival orders. Co-authored-by: Cursor <cursoragent@cursor.com>

kaarolch · 2026-08-19T05:56:47Z

@pront PR should looks better now.

ArunPiduguDD

@kaarolch @pront asked me to help review this PR as we also want this feature in Vector. I've went over / ran some local tests and the functionality looks good, I have a few minor suggestions

…cket Eligible event-time windows are a prefix of the BTreeMap, so flush with take_while instead of visiting every open bucket. Also rename is_too_late to was_bucket_flushed. Co-authored-by: Cursor <cursoragent@cursor.com>

…vents_based_aggr

kaarolch · 2026-09-03T12:14:10Z

@ArunPiduguDD @pront added recommendation, The rest I would like to apply as separate follow up small PRs.

pront

Thanks!

Master no longer accepts #[configurable(derived)]; field rustdoc already drives the generated config schema. Co-authored-by: Cursor <cursoragent@cursor.com>

kaarolch · 2026-09-03T17:17:27Z

@pront small commit to remove deprecated section from #26262

kaarolch added 2 commits December 30, 2025 09:32

Add event's timestamp aggregation option

aaece39

Update documentation

09a626a

kaarolch requested review from a team as code owners December 30, 2025 09:07

github-actions Bot added domain: transforms Anything related to Vector's transform components domain: external docs Anything related to Vector's external, public documentation labels Dec 30, 2025

Add Changelog.d file

e994ec1

kaarolch mentioned this pull request Dec 30, 2025

chore(ci): Add aggregate transform to semantic PR scope list #24422

Merged

8 tasks

adiwab mentioned this pull request Dec 30, 2025

Support metrics aggregation windowing by event timestamp #14015

Open

git-thuerk-done approved these changes Dec 30, 2025

View reviewed changes

kaarolch and others added 5 commits December 30, 2025 19:59

Merge branch 'master' into events_based_aggr

f8c6109

Merge branch 'master' into events_based_aggr

a3b5502

chore(vdev): apply vdev rust check fixes

bcc47f0

Fix some edge cases and add test

ea41697

Merge branch 'master' into events_based_aggr

6783f87

Merge branch 'master' into events_based_aggr

c077672

github-actions Bot added the work in progress label Apr 27, 2026

Merge branch 'master' into events_based_aggr

8574e57

chatgpt-codex-connector Bot reviewed May 1, 2026

View reviewed changes

Comment thread src/transforms/aggregate.rs Outdated

Comment thread src/transforms/aggregate.rs Outdated

Comment thread src/transforms/aggregate.rs Outdated

chatgpt-codex-connector Bot reviewed May 4, 2026

View reviewed changes

Comment thread src/transforms/aggregate.rs Outdated

Merge branch 'master' into events_based_aggr

718b502

chatgpt-codex-connector Bot reviewed Aug 17, 2026

View reviewed changes

Comment thread src/transforms/aggregate/event_time.rs Outdated

kaarolch and others added 2 commits August 17, 2026 10:20

Merge branch 'master' into events_based_aggr

12e5179

chatgpt-codex-connector Bot reviewed Aug 17, 2026

View reviewed changes

Comment thread src/transforms/aggregate/event_time.rs Outdated

kaarolch and others added 2 commits August 17, 2026 10:37

Merge branch 'master' into events_based_aggr

fa3dd16

chatgpt-codex-connector Bot reviewed Aug 18, 2026

View reviewed changes

Comment thread src/transforms/aggregate/event_time.rs Outdated

kaarolch and others added 2 commits August 18, 2026 08:30

chore(aggregate transform): make event_time accessor const for clippy

1eba22a

Clippy's missing_const_for_fn lint is denied in vector and would fail Check clippy once the fork-PR Test Suite workflow is approved. Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector Bot reviewed Aug 18, 2026

View reviewed changes

Comment thread src/transforms/aggregate/event_time.rs Outdated

Comment thread src/transforms/aggregate/event_time.rs Outdated

kaarolch and others added 3 commits August 18, 2026 09:31

test(aggregate transform): consolidate event-time coverage with share…

96a8f73

…d helpers Reduce overlapping cases and LoC while keeping the behavioral checks, including both Latest arrival orders. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'master' into events_based_aggr

ff76393

kaarolch added 2 commits August 19, 2026 07:56

Merge branch 'master' into events_based_aggr

dfc1299

Merge branch 'master' into events_based_aggr

128a609

ArunPiduguDD reviewed Aug 31, 2026

View reviewed changes

Comment thread src/transforms/aggregate/transform.rs

Comment thread src/transforms/aggregate/transform.rs

Comment thread src/transforms/aggregate/event_time.rs

Comment thread src/transforms/aggregate/event_time.rs Outdated

Comment thread src/transforms/aggregate/event_time.rs

ArunPiduguDD requested a review from pront August 31, 2026 17:30

kaarolch and others added 3 commits September 3, 2026 10:56

Merge branch 'events_based_aggr' of github.com:kaarolch/vector into e…

31f0178

…vents_based_aggr

Merge branch 'master' into events_based_aggr

3baba88

pront approved these changes Sep 3, 2026

View reviewed changes

pront enabled auto-merge September 3, 2026 16:55

github-actions Bot removed the docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. label Sep 3, 2026

fix(aggregate transform): drop removed configurable(derived) attributes

fc1c786

Master no longer accepts #[configurable(derived)]; field rustdoc already drives the generated config schema. Co-authored-by: Cursor <cursoragent@cursor.com>

auto-merge was automatically disabled September 3, 2026 17:16
Head branch was pushed to by a user without write access

Merge branch 'master' into events_based_aggr

746f977

Merge branch 'master' into events_based_aggr

9a2165e

Uh oh!

Conversation

kaarolch commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Solution

Key Changes

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

kaarolch commented Mar 20, 2026

Uh oh!

kaarolch commented Mar 21, 2026

Uh oh!

kaarolch commented Apr 27, 2026

Uh oh!

syedg1 commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pront commented May 1, 2026

Uh oh!

pront commented May 1, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaarolch commented May 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

kaarolch commented Aug 19, 2026

Uh oh!

ArunPiduguDD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaarolch commented Sep 3, 2026

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

kaarolch commented Sep 3, 2026

kaarolch commented Dec 30, 2025 •

edited

Loading

syedg1 commented Apr 30, 2026 •

edited

Loading