Skip to content

feat(tracing): stamp sgp evals run and row ids on SGP spans - #545

Open
mohammadatallah-scale wants to merge 8 commits into
nextfrom
mohammad/ove-1253-sgp-evals-span-metadata
Open

mohammadatallah-scale wants to merge 8 commits into
nextfrom
mohammad/ove-1253-sgp-evals-span-metadata

Conversation

@mohammadatallah-scale

@mohammadatallah-scale mohammadatallah-scale commented Oct 7, 2026 •

Copy link
Copy Markdown

OVE-1253. Pairs with an sgp-evaluations PR that tags each generation-unit task.

The problem. Spans of an evals generation-unit task carry no run or row id, so a spans search cannot find them.

The fix.

  • Stamp: the SGP copy of each span of an eval task gets flat run, row and attempt keys. A task that loses the eval marker stops being stamped.
  • Sync and async agents: the ACP server reads the task it receives.
  • Temporal agents: the ids travel through the workflow memo to the worker's activities. A span keeps the ids its task had when it started, so a reused task id cannot strip a queued span. Not covered: spans whose trace_id and task_id are both not the task id.

Test plan

  • Real Temporal dev server run: eval workflow stamps the three keys, a plain one stamps nothing

Changes since review

  • A span's run and row ids are fixed when it starts, so a task id reused before export cannot change them.
  • A span that started with no eval task can no longer pick up a later run's ids.
  • A failed upload keeps the ids, so a retry sends the same span.

RetriggerConfidence Score: 5/5

This update appears safe to merge; no new issue remains to address.

What we checked:

  • Plain spans evict saved eval IDs: No. Plain captures use a separate map, and the test checks that the eval capture remains after the plain map exceeds its limit.

Summary

SGP eval task run, row, and attempt ids now appear as searchable metadata on the task’s spans. The PR passes those ids through both in-process ACP handlers and Temporal workers, and keeps each span’s attribution fixed from start through upload.

  • SGP eval spans carry searchable run, row, and attempt ids.
  • Temporal activities carry eval ids into their spans.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Task metadata] --> B[Task ID registry]
    B --> C[Capture IDs when span starts]
    C --> D{Captured IDs?}
    D -->|Yes| E[Eval span map]
    D -->|No| F[Plain span map]
    E --> G[SGP export]
    F --> G
    G --> H[Release capture after export]
Loading

Reviews (5) · Last reviewed commit: "fix(tracing): keep eval span id captures..." · Reviewed by Greptile

mohammadatallah-scale and others added 2 commits October 7, 2026 08:04
Spans of a task whose task_metadata carries sgp_evals now get flat
sgp_evals_generation_run_id, sgp_evals_row_id and sgp_evals_attempt_idx keys on
the SGP copy. Sync and async agents read the task in the ACP server. Temporal
agents carry it in the workflow memo and activity headers to the worker.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…val attrs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread src/agentex/lib/core/tracing/sgp_evals.py Outdated
Comment thread src/agentex/lib/core/tracing/sgp_evals_interceptor.py
Comment thread src/agentex/lib/core/tracing/sgp_evals_interceptor.py
@mohammadatallah-scale

Copy link
Copy Markdown
Author

P3 sgp_evals_interceptor.py - activities of a child workflow get no header (memo is not inherited and their workflow id is not the task id), so their spans stay unstamped even when trace_id is the task id.
P3 worker.py:278 - no test pins SGPEvalsInterceptor in the worker's interceptor list, so dropping it keeps the suite green.

Refuted: continue-as-new keeps the memo (real dev-server run stamps the post-CAN run), ContextInterceptor headers coexist with this one in either order, and task_metadata does reach the agent inside params.task, so the event/send pop never drops a live eval task.

mohammadatallah-scale and others added 2 commits October 7, 2026 08:19
Local activities go through start_local_activity, which the outbound
interceptor did not override, so their spans carried no run or row id.
Also pin the worker's interceptor wiring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ry none

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread src/agentex/lib/core/tracing/sgp_evals_interceptor.py
…id cannot strip a queued span

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread src/agentex/lib/core/tracing/sgp_evals.py Outdated
Comment thread src/agentex/lib/core/tracing/processors/sgp_tracing_processor.py Outdated
mohammadatallah-scale and others added 2 commits October 7, 2026 08:56
…load succeeds

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread src/agentex/lib/core/tracing/sgp_evals.py Outdated
…o a span flood cannot evict them

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant