competitive_monitoring

Example: competitive monitoring — a REAL run

this is a real, paid pipeline run, not a fixture — real Anthropic + Tavily calls over real web results. It also exercises a different output format (structured_report) than the lead-scoring run, so the two together show the same pipeline producing two genuinely different real artifacts.

Date	2026-07-25
Subject	Notion (monitored as a competitor)
LLM	anthropic/claude-opus-5
Retrieval	Tavily web search
Cost	~$0.844 (78,864 tokens, 13 LLM calls, 179s — as reported by the run)
Grounding	87 of 93 claims (~94%) cite a retrieved source; the rest are flagged [UNCITED]
Config	config.yaml — the exact config used, reproducible

What's here

• output.json — the structured_report: title, executive summary, one finding per expert (summary + supporting points + concerns), and a red_flags list. This is the monitoring digest a product/marketing team would read, or diff month-over-month.
• research_log.json — the grounded evidence trail: every expert's claims with the source URL(s) behind each one.
• run_record.json / trace.jsonl — per-call tokens and cost, phase timings, and every retrieval query issued.

What this run actually surfaced (real signals)

• Product: the April 2026 "Workers for Agents" developer preview — custom code executed by AI agents inside Notion, shipped with eight new API endpoints — read by the panel as a platform-layer expansion rather than a feature drop, alongside 3.2/3.3 changelog activity.
• Pricing: a deliberate move upmarket — four seat-based tiers plus the first meaningful usage-based layer, with AI capability gated to higher tiers.
• Hiring: job postings up 79.3% in 2026 to 414, concentrated in enterprise-facing roles.

The 11 red_flags are the model's synthesis of those findings — a plausible read, not verified analysis.

The interesting part: it audits its own evidence

Four of the eleven red flags are about the research itself, not about Notion:

"Headcount figures do not reconcile: Revelio ~2,919 (March 2026) vs. Unify GTM 1,920 (April 2026), a ~34% spread."

"Posting-growth baselines conflict within the same attributed provider."

"One research query on posting locations returned entirely off-topic sources about the UK consumer organisation Which?"

"LinkedIn's 6,061 'open jobs' figure is contaminated by generic role-category counts and cannot be reconciled with Notion's own 109-position Ashby board."

A monitoring digest that reports "headcount grew 48%" and stops is worth very little; one that says "two vendors disagree by 34% and here's a query that retrieved the wrong company entirely" tells you which numbers to trust. That contradiction-surfacing falls out of keeping every claim attached to its source instead of averaging findings into a summary.

Grounding, honestly

94% of claims are tied to a retrieved source, spanning 21 distinct URLs across 19 domains (notion.com, reveliolabs.com, jobs.ashbyhq.com, and others). The six [UNCITED] claims are assertions the pipeline could not tie back to a Tavily-retrieved document, so it refuses to credit them rather than faking a citation.

The earlier version of this example ran on a web-connected model (perplexity/sonar) and reached 67%; that README predicted a non-web model over the same Tavily results would ground a higher fraction, since it has no independent web knowledge to fall back on. Swapping to Opus 5 moved it 67% → 94%, which is that prediction holding up.

Reproduce it

# needs real ANTHROPIC_API_KEY and TAVILY_API_KEY in .env
uv run panelist run examples/competitive_monitoring/config.yaml

Expect roughly $0.85 and three minutes on claude-opus-5 — close enough to the config's max_cost_usd: 1.00 stop that a longer run could trip it, which is the cap doing its job. Swap llm.model to claude-haiku-4-5 for a cheaper run.