You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add trace and session annotation as a first-class Beacon feature for humans and agents: users can annotate a whole session and any transcript message/event, agents can annotate sessions/events through MCP, and the resulting annotated traces are easy to review, query, and export for self-improvement, skill development, evaluation, and fine-tuning datasets. Work lands on the feature branch feature/trace-annotations; implementation PRs should target that branch, not main.
Source Of Truth
Governing sources, in order, when instructions conflict:
Active user goal and explicit user corrections.
This progress tracker.
The active implementation issue or PR scope.
Named plan/spec/design docs.
Repo docs, local skills, and historical behavior.
No material conflicts are currently known. Backward compatibility is not required unless a later user correction says otherwise.
Shared Context
Beacon is a local AI-agent session dashboard backed by ClickHouse. Session/event capture writes to activity_events, projections live in session_projection and analytics_projection, web JSON handlers live in internal/web/api.go, transcript pages are templ views under internal/views/pages and internal/views/components, transcript behavior is in internal/assets/static/js/transcript.js, and MCP tools are registered in internal/mcp/tools.go.
Session detail currently renders /sessions/{id} and lazy-loads /sessions/{id}/conversation; JSON access includes /api/sessions/{id}, /api/sessions/{id}/events, and /api/events/{event_id}. MCP includes search/open/list/usage tools plus annotation read/write tools. Annotation write tools must be clearly marked non-read-only in MCP tool annotations.
Annotation records should be extensible from the start. Useful fields should include at least: target type (session, message, or event), session_id, optional event_uid, author type (human or agent), author identity/display name when available, source (ui, mcp, or future clients), note text, structured labels/tags, task/outcome/quality/confidence/follow-up fields where practical, metadata JSON for future extension, create/update timestamps, and soft deletion or status semantics. Message annotations are backed by activity_events.event_uid and must validate that the event row has event_kind = 'message'. Do not bake annotation semantics into one fixed fine-tuning format; keep the domain model broad enough for future evaluation, dataset, skill-development, and review workflows.
Definition of Done
A feature branch named feature/trace-annotations exists and contains the full annotation implementation.
Humans can annotate the session as a whole from the session detail page.
Humans can annotate individual transcript messages/events from the session chat and timeline views.
Agents can create/update/read annotation records through Beacon MCP tools.
Annotation data persists in Beacon storage and is available through typed JSON APIs.
Annotated traces can be discovered and exported or otherwise gathered into dataset-friendly JSON from Beacon without scraping the UI.
The UI exposes annotation state as a first-class part of session review, with badges/counts/panels that work in desktop and mobile transcript layouts.
In-depth QA exists for storage, API, MCP, UI behavior, accessibility-relevant paths, and visual states.
A committed QA report and a substantial set of UI screenshots/images show annotation flows and record results.
make install-local INSTALL_DIR="$HOME/.local/bin" succeeds, the persistent local review server is restarted from the installed binary, and the review URL is reported.
Release/publish is out of scope. README/user-facing docs are in scope where needed to make annotation workflows discoverable.
Validation Strategy
Every implementation issue should run the relevant subset and record exact results. The final pass should run the broad set.
generation: make generate-check
formatting: make fmt-check
go tests: make test
build: make build
go lint: make lint
frontend unit/lint: npm run test:frontend
e2e: npm run test:e2e
accessibility: npm run test:a11y
visual: npm run test:visual
local install: make install-local INSTALL_DIR="$HOME/.local/bin"
Annotation contract drift: UI, MCP, JSON APIs, and export paths must share one model instead of parallel ad hoc shapes.
Dataset usefulness: free-form notes alone are insufficient. Add structured fields while leaving room for future metadata.
Write security and accidental data loss: write APIs/tools need validation, bounded payload sizes, target existence checks where feasible, and soft delete/update semantics.
UI crowding: transcript pages are already dense. Annotation controls need to be discoverable without destabilizing chat/timeline layout.
QA artifact churn: screenshots should be stable, named, and tied to an explicit QA report rather than incidental Playwright output.
All implementation PRs target feature/trace-annotations. The storage/domain/API issue defines the canonical annotation schema and validation rules. UI, MCP, and export paths must consume that shared contract rather than duplicating request/response structs unless a transport-specific wrapper is necessary. Dataset export should reuse the same annotation query paths and include enough session/event context to make exported traces useful without a live ClickHouse instance.
Current State
Active issue / branch / PR: none; all planned work plus final-pass follow-up is merged into feature/trace-annotations.
Complete on feature/trace-annotations at b18c57fe56bad1959e0bba8ba96e5bcd3ee093f7.
Delivered:
First-class annotation storage/domain model and typed REST APIs for session, message, and event annotations.
Transcript UI for session-level and message/event-level annotation, including desktop/mobile drawer behavior, counts, edit/delete, and timeline/event controls.
MCP annotation tools for agents: create_annotation, update_annotation, list_annotations, get_annotation, and delete_annotation.
Annotated-trace discovery and dataset export APIs with session/event context and filter support.
Documentation in README.md and docs/mcp.md.
QA report and screenshots under docs/qa/trace-annotations/, with workflow evidence JSON for MCP and dataset APIs.
Final validation evidence:
make generate-check -> passed
make fmt-check -> passed
npm run test:frontend -> passed
make test -> passed
make build -> passed
make lint -> passed
npm run test:e2e -> passed, 53/53
npm run test:a11y -> passed, 10/10
npm run test:visual -> passed, 8/8
BEACON_QA_CAPTURE=1 npx playwright test tests/e2e/trace-annotations-qa.spec.ts --reporter=line -> passed, 3/3
npx playwright test tests/e2e/trace-annotations-qa.spec.ts --reporter=line -> passed, 3 skipped as designed
git diff --check -> passed
make install-local INSTALL_DIR="$HOME/.local/bin" -> passed
Persistent review server restarted from installed binary with /tmp/beacon-trace-annotations-dev.toml; curl -fsS http://localhost:4600/ >/dev/null -> passed
Served UI visual smoke passed for dashboard and transcript; transcript smoke found session, message, and event annotation controls.
Final holistic review found correctness/security gaps; PR Fix final annotation review gaps #313 fixed them and passed four independent clean reviewers.
Known residual risks:
Scoped annotation list pagination intentionally scans raw annotation pages until the visible page is filled; very heavily filtered sessions or large visible offsets can do extra work, but correctness and authorization behavior are covered by tests.
Live ClickHouse performance remains environment-dependent beyond local and CI coverage.
Goal
Add trace and session annotation as a first-class Beacon feature for humans and agents: users can annotate a whole session and any transcript message/event, agents can annotate sessions/events through MCP, and the resulting annotated traces are easy to review, query, and export for self-improvement, skill development, evaluation, and fine-tuning datasets. Work lands on the feature branch
feature/trace-annotations; implementation PRs should target that branch, notmain.Source Of Truth
Governing sources, in order, when instructions conflict:
No material conflicts are currently known. Backward compatibility is not required unless a later user correction says otherwise.
Shared Context
Beacon is a local AI-agent session dashboard backed by ClickHouse. Session/event capture writes to
activity_events, projections live insession_projectionandanalytics_projection, web JSON handlers live ininternal/web/api.go, transcript pages are templ views underinternal/views/pagesandinternal/views/components, transcript behavior is ininternal/assets/static/js/transcript.js, and MCP tools are registered ininternal/mcp/tools.go.Session detail currently renders
/sessions/{id}and lazy-loads/sessions/{id}/conversation; JSON access includes/api/sessions/{id},/api/sessions/{id}/events, and/api/events/{event_id}. MCP includes search/open/list/usage tools plus annotation read/write tools. Annotation write tools must be clearly marked non-read-only in MCP tool annotations.Annotation records should be extensible from the start. Useful fields should include at least: target type (
session,message, orevent),session_id, optionalevent_uid, author type (humanoragent), author identity/display name when available, source (ui,mcp, or future clients), note text, structured labels/tags, task/outcome/quality/confidence/follow-up fields where practical, metadata JSON for future extension, create/update timestamps, and soft deletion or status semantics. Message annotations are backed byactivity_events.event_uidand must validate that the event row hasevent_kind = 'message'. Do not bake annotation semantics into one fixed fine-tuning format; keep the domain model broad enough for future evaluation, dataset, skill-development, and review workflows.Definition of Done
feature/trace-annotationsexists and contains the full annotation implementation.make install-local INSTALL_DIR="$HOME/.local/bin"succeeds, the persistent local review server is restarted from the installed binary, and the review URL is reported.Validation Strategy
Every implementation issue should run the relevant subset and record exact results. The final pass should run the broad set.
make generate-checkmake fmt-checkmake testmake buildmake lintnpm run test:frontendnpm run test:e2enpm run test:a11ynpm run test:visualmake install-local INSTALL_DIR="$HOME/.local/bin"tmux kill-session -t beacon-up 2>/dev/null || true;tmux new-session -d -s beacon-up "$HOME/.local/bin/beacon up";tmux capture-pane -pt beacon-up -S -80;curl -fsS http://localhost:4600/ >/dev/nullRisks
Issue Checklist (dependency order)
Integration Notes
All implementation PRs target
feature/trace-annotations. The storage/domain/API issue defines the canonical annotation schema and validation rules. UI, MCP, and export paths must consume that shared contract rather than duplicating request/response structs unless a transport-specific wrapper is necessary. Dataset export should reuse the same annotation query paths and include enough session/event context to make exported traces useful without a live ClickHouse instance.Current State
feature/trace-annotations.b18c57fe56bad1959e0bba8ba96e5bcd3ee093f723a5aaee4436dd46bdac4802403ba4fd6be4d602; current feature branch SHA:b18c57fe56bad1959e0bba8ba96e5bcd3ee093f7Final Status
Complete on
feature/trace-annotationsatb18c57fe56bad1959e0bba8ba96e5bcd3ee093f7.Delivered:
create_annotation,update_annotation,list_annotations,get_annotation, anddelete_annotation.README.mdanddocs/mcp.md.docs/qa/trace-annotations/, with workflow evidence JSON for MCP and dataset APIs.Final validation evidence:
make generate-check-> passedmake fmt-check-> passednpm run test:frontend-> passedmake test-> passedmake build-> passedmake lint-> passednpm run test:e2e-> passed, 53/53npm run test:a11y-> passed, 10/10npm run test:visual-> passed, 8/8BEACON_QA_CAPTURE=1 npx playwright test tests/e2e/trace-annotations-qa.spec.ts --reporter=line-> passed, 3/3npx playwright test tests/e2e/trace-annotations-qa.spec.ts --reporter=line-> passed, 3 skipped as designedgit diff --check-> passedmake install-local INSTALL_DIR="$HOME/.local/bin"-> passed/tmp/beacon-trace-annotations-dev.toml;curl -fsS http://localhost:4600/ >/dev/null-> passedReview evidence:
Known residual risks:
Review URL: http://localhost:4600/