diff --git a/examples/agentic-data-creation/README.md b/examples/agentic-data-creation/README.md index f0c5e57a..26542e9d 100644 --- a/examples/agentic-data-creation/README.md +++ b/examples/agentic-data-creation/README.md @@ -1,12 +1,12 @@ # agentic-data-creation -**An agent manufactures its own hard training data.** This is the INNER loop of Autodata / -Agentic Self-Instruct (Meta FAIR, arXiv 2606.25996): instead of hand-writing examples, an agent +**An agent manufactures its own hard training data.** This is the INNER loop of **agentic +self-instruct** (the self-instruct pattern, Wang et al. 2022, taken agentic): instead of hand-writing examples, an agent *writes* candidate {context, question, reference, rubric} examples from a grounding doc and keeps only the ones that are **hard for a weak solver but doable for a strong one**. The hard ones are exactly the examples worth training on. -> This example builds only the paper's **data-creation** half (the inner loop). The RL-training +> This example builds only the **data-creation** half (the inner loop). The RL-training > outer half needs a trainer this repo does not have, so it is out of scope here. Runs fully offline (scripted solvers + a mocked judge, no credentials): @@ -47,7 +47,7 @@ flowchart TD ## The one new piece — `discriminativeAcceptRule` Everything else is composed from primitives this repo already ships. The genuinely new piece is the -paper's reward, written as a small, Validator-shaped accept/reject: +accept rule, written as a small, Validator-shaped accept/reject: ```ts discriminativeAcceptRule({ strongScore, weakScore, minStrong = 0.65, maxWeak = 0.5, minGap = 0.2 }) @@ -86,7 +86,7 @@ this before trusting it (the `calibrate-before-measure` discipline): it measures challenger's **first (un-refined) draft** — plain generation — and on the **loop-accepted** example, and shows the accept rule separates them. **Offline the solvers are scripted, so this proves the wiring + that the rule discriminates by construction — it is NOT an empirical reproduction of the -paper's Table 1.** Reproducing Table 1 for real (the loop actually producing harder data) needs the +illustrative target.** Reproducing that separation for real (the loop actually producing harder data) needs the live run below, with real two-tier solver models: ``` @@ -101,7 +101,7 @@ examples would be uninformative, and the loop would be optimizing noise. `offline-fixtures.ts` is the credentialless stand-in (the same pattern `examples/driver-loop` and `examples/self-improving-loop` use): deterministic scripted challenger/solvers and a **mocked judge -transport** bound into a *real* `llmJudge` `JudgeConfig`, tuned to reproduce Table 1. The judge, +transport** bound into a *real* `llmJudge` `JudgeConfig`, tuned to the illustrative target separation. The judge, sampler, fold, cost ledger, and corpus are all the real primitives — only the LLM responses are scripted. To run live, swap the mock transport for `createChatClient({ transport: 'router', apiKey })` (glm-5.2) and the scripted workers for real sandbox/cli-bridge clients; the loop is unchanged. diff --git a/examples/agentic-data-creation/agentic-data-creation.ts b/examples/agentic-data-creation/agentic-data-creation.ts index 6288be2e..46821925 100644 --- a/examples/agentic-data-creation/agentic-data-creation.ts +++ b/examples/agentic-data-creation/agentic-data-creation.ts @@ -1,7 +1,7 @@ /** - * agentic-data-creation — the INNER loop of Autodata / Agentic Self-Instruct (Meta FAIR, - * arXiv 2606.25996): an agent MANUFACTURES hard training examples from a grounding doc, and keeps - * only the ones that DISCRIMINATE a strong solver from a weak one. + * agentic-data-creation — the INNER loop of AGENTIC SELF-INSTRUCT (the self-instruct pattern, + * Wang et al. 2022, taken agentic): an agent MANUFACTURES hard training examples from a grounding doc, + * and keeps only the ones that DISCRIMINATE a strong solver from a weak one. * * This file is the SUBJECT. The whole method is four roles + one accept rule, composed from * primitives this package already ships — nothing here re-implements a judge, a sampler, a cost @@ -78,7 +78,7 @@ export interface AcceptDecision { // THE ONE NEW PIECE — the paper's discriminative reward, as a small Validator-shaped rule. // ═══════════════════════════════════════════════════════════════════════════════════════════ // -// Autodata keeps an example ONLY IF it separates a strong solver from a weak one: the strong +// The accept rule keeps an example ONLY IF it separates a strong solver from a weak one: the strong // solver should mostly get it (>= minStrong), the weak solver should mostly miss it (< maxWeak), // and the margin between them (the "gap") must clear minGap. That is the whole objective — make // examples too hard for the weak solver — so the rule is the LITERAL accept criterion, never @@ -340,7 +340,7 @@ export interface DataCreationResult { } /** - * Run the Autodata inner loop: manufacture `target` discriminating examples from `doc`, refining + * Run the self-instruct inner loop: manufacture `target` discriminating examples from `doc`, refining * each via the challenger fold until it is accepted (or its retry budget runs out). Returns the * accepted set, the per-example gap for the accepted (agentic) AND the first-draft (plain) examples * for calibration, the corpus they accreted into, and the cost ledger. diff --git a/examples/agentic-data-creation/offline-fixtures.ts b/examples/agentic-data-creation/offline-fixtures.ts index ca670909..8a7fb691 100644 --- a/examples/agentic-data-creation/offline-fixtures.ts +++ b/examples/agentic-data-creation/offline-fixtures.ts @@ -13,8 +13,9 @@ * [0,1] rubric score from (strength, difficulty) with a small per-sample jitter so the N× mean * is a genuine average. LIVE mode (glm-5.2) instead reads the real answer text against the rubric. * - * The scores are tuned to reproduce the paper's Table 1 separation: an EASY (plain) example barely - * separates the two solvers (gap ≈ 0.02); a HARD (agentic) one separates them widely (gap ≈ 0.31). + * The scores are tuned to an ILLUSTRATIVE target separation: an EASY (plain) example barely separates + * the two solvers (gap ≈ 0.02); a HARD (agentic) one separates them widely (gap ≈ 0.31). These numbers are + * by construction here — a live run produces the real ones. */ import { createChatClient, llmJudge } from '@tangle-network/agent-eval' @@ -157,7 +158,7 @@ export function solverClient(strength: 'weak' | 'strong'): SandboxClient { // `llmJudge` builds the system+user messages, makes ONE chat() call, and parses the model's // `{ dimensions, notes }` JSON into a canonical [0,1] `JudgeScore` (real composite math). Offline, // the transport returns a scripted score from the answer's grade marker; live, a real model scores -// the prose. Tuned so EASY → gap ≈ 0.02, HARD → gap ≈ 0.31 (the paper's Table 1). +// the prose. Tuned so EASY → gap ≈ 0.02, HARD → gap ≈ 0.31 (illustrative targets, by construction). export function buildRubricJudge(): JudgeConfig { const chat = createChatClient({ transport: 'mock', diff --git a/examples/agentic-data-creation/run.ts b/examples/agentic-data-creation/run.ts index 0f8494d9..92fa0456 100644 --- a/examples/agentic-data-creation/run.ts +++ b/examples/agentic-data-creation/run.ts @@ -5,7 +5,7 @@ * training examples from one grounding doc, and prints: * 1. each accepted example with its weak/strong solver scores and the gap, * 2. the CALIBRATION — does the gap metric actually separate? A plainly-generated (first-draft) - * example should show a SMALL gap; an agentic-loop-accepted one a LARGE gap (the paper's Table 1), + * example should show a SMALL gap; an agentic-loop-accepted one a LARGE gap (the illustrative target), * 3. the cost ledger, split by role (challenger vs each solver) — composed, never hand-counted. * * Run: pnpm tsx examples/agentic-data-creation/run.ts @@ -55,9 +55,9 @@ async function main(): Promise { const plain = mean(result.plainGaps) const agentic = mean(result.agenticGaps) console.log('\n— Calibration: does the gap metric discriminate? —') - console.log(` plain (first-draft examples) mean gap = ${plain.toFixed(2)} (paper ≈ 0.02)`) + console.log(` plain (first-draft examples) mean gap = ${plain.toFixed(2)} (target ≈ 0.02)`) console.log( - ` agentic (loop-accepted examples) mean gap = ${agentic.toFixed(2)} (paper ≈ 0.31)`, + ` agentic (loop-accepted examples) mean gap = ${agentic.toFixed(2)} (target ≈ 0.31)`, ) const separates = Number.isFinite(plain) && Number.isFinite(agentic) && agentic - plain >= 0.15 console.log( diff --git a/examples/agents-of-all-shapes/README.md b/examples/agents-of-all-shapes/README.md index 4ab68f2b..d6ef59e6 100644 --- a/examples/agents-of-all-shapes/README.md +++ b/examples/agents-of-all-shapes/README.md @@ -20,7 +20,7 @@ No sandbox. No deploy. No server. The analysis runs **in-process**. ```bash # Verified QA path — in-process, no key, no infra: -npx tsx examples/agents-of-all-shapes/run.ts +pnpm tsx examples/agents-of-all-shapes/run.ts # CI verification (what proves it): pnpm test -- tests/agents-of-all-shapes.test.ts diff --git a/examples/agents-of-all-shapes/shared/intelligence.ts b/examples/agents-of-all-shapes/shared/intelligence.ts index 8163194d..4ac8c265 100644 --- a/examples/agents-of-all-shapes/shared/intelligence.ts +++ b/examples/agents-of-all-shapes/shared/intelligence.ts @@ -17,6 +17,7 @@ import { analyzeRuns, fromOtelSpans, type InsightReport } from '@tangle-network/agent-eval/contract' import type { TraceSpanEvent } from '@tangle-network/agent-eval/hosted' +import { createOtelExporter } from '@tangle-network/agent-runtime' export type { InsightReport, TraceSpanEvent } @@ -101,51 +102,33 @@ export interface ShipOptions { serviceName?: string } -/** Optional hosted path: POST the same OTel spans to Tangle Intelligence's - * OTLP/HTTP ingest. Identical analysis runs server-side. */ +/** Optional hosted path: POST the same OTel spans to Tangle Intelligence's OTLP/HTTP ingest via the + * runtime's OWN exporter — `createOtelExporter` builds the resourceSpans envelope, appends `/v1/traces`, + * and batches the POST. No hand-rolled wire format; the same primitive the runtime uses in production. */ export async function shipToTangleOtlp(spans: TraceSpanEvent[], opts: ShipOptions): Promise { - const res = await fetch(`${opts.endpoint}/v1/traces`, { - method: 'POST', - headers: { - 'content-type': 'application/json', - authorization: `Bearer ${opts.apiKey}`, - }, - body: JSON.stringify({ - resourceSpans: [ - { - resource: { - attributes: [ - { - key: 'service.name', - value: { stringValue: opts.serviceName ?? 'agents-of-all-shapes' }, - }, - ], - }, - scopeSpans: [ - { - scope: { name: 'agents-of-all-shapes' }, - spans: spans.map((s) => ({ - traceId: s.traceId, - spanId: s.spanId, - name: s.name, - startTimeUnixNano: String(s.startTimeUnixNano), - endTimeUnixNano: String(s.endTimeUnixNano), - attributes: Object.entries(s.attributes).map(([key, value]) => ({ - key, - value: - typeof value === 'number' - ? { doubleValue: value } - : { stringValue: String(value) }, - })), - status: s.status, - })), - }, - ], - }, - ], - }), + const exporter = createOtelExporter({ + endpoint: opts.endpoint, + headers: { authorization: `Bearer ${opts.apiKey}` }, + serviceName: opts.serviceName ?? 'agents-of-all-shapes', }) - if (!res.ok) { - throw new Error(`intelligence ingest failed: ${res.status} ${await res.text()}`) + if (!exporter) throw new Error('shipToTangleOtlp: no OTLP endpoint configured') + // OTLP status.code is numeric (UNSET=0, OK=1, ERROR=2); TraceSpanEvent carries the string enum. + const statusCode = { UNSET: 0, OK: 1, ERROR: 2 } as const + for (const s of spans) { + exporter.exportSpan({ + traceId: s.traceId, + spanId: s.spanId, + name: s.name, + startTimeUnixNano: String(s.startTimeUnixNano), + endTimeUnixNano: String(s.endTimeUnixNano), + attributes: Object.entries(s.attributes).map(([key, value]) => ({ + key, + value: typeof value === 'number' ? { doubleValue: value } : { stringValue: String(value) }, + })), + ...(s.status + ? { status: { code: statusCode[s.status.code], message: s.status.message } } + : {}), + }) } + await exporter.flush() } diff --git a/examples/intelligence-drop-in/README.md b/examples/intelligence-drop-in/README.md index c096ad3c..6eebe54b 100644 --- a/examples/intelligence-drop-in/README.md +++ b/examples/intelligence-drop-in/README.md @@ -1,30 +1,38 @@ # intelligence-drop-in -The Observe + Mode-0 slice of the Tangle Intelligence SDK: wrap an existing -agent, ship one trace per call, and pay only inference at the OFF tier. The -wrapper is best-effort — a live agent never fails because Intelligence is down. +The Observe + Mode-0 slice of the Tangle Intelligence SDK: wrap an existing agent, ship one trace per +call, and pay **only inference** (the base model stream) at the OFF tier. **Why it matters:** you get +per-call observability + billing with a one-line wrapper, and you can prove — not just assert — that +turning intelligence *off* charges nothing extra. The wrapper is best-effort: a live agent never fails +because Intelligence is down. + +> **Mode 0** = the OFF tier: telemetry stays on, but intelligence spend (analysts, corpus, extra spawns) +> is clamped to 0. **Inference spend** = the base model stream you'd pay anyway; **intelligence spend** = +> what the SDK's extra reasoning adds on top. ## Run ```bash +# $0, no creds — stands up a throwaway local OTLP collector so the trace is visible without a key. pnpm tsx examples/intelligence-drop-in/intelligence-drop-in.ts ``` -The example stands up a throwaway local OTLP collector, so it runs with no -credentials. +It prints three proofs and **asserts** the last two (throws if they don't hold): +1. wrap any `(input) => Promise` in one line and it ships a trace; +2. point it at a dead endpoint — the agent still answers (export failure swallowed); +3. at `effort: 'off'`, read the exported span BACK off the collector and confirm `intelligence_usd = 0`. ## What it shows -- `withTangleIntelligence(agent, { project, apiKey, endpoint })` — wrap any - `(input) => Promise` agent; the shape is preserved and one trace span - is exported per call. -- `createIntelligenceClient(...).traceRun(meta, fn)` — the explicit-trace API: - `trace.recordOutput` / `trace.recordOutcome` inside the body. -- **Best-effort export** — pointed at a dead endpoint, the agent still returns - its answer; the export failure is swallowed. -- **Mode 0 / OFF** (`effort: 'off'`) — pure passthrough, zero intelligence - spawns. The exported trace carries `{ inferenceUsd, intelligenceUsd }` and - `intelligenceUsd` is clamped to `0` — the mechanism that proves an OFF - customer paid inference-only. -- `client.doctor()` — network-free readiness: Observe is always reachable; - Recommend and Gated-PR report the inputs they still need. +- `withTangleIntelligence(agent, { project, apiKey, endpoint })` — wrap any agent; the call shape is + preserved and one trace span is exported per call, fire-and-forget. +- `createIntelligenceClient(...).traceRun(meta, fn)` — the explicit-trace API (`trace.recordOutput` / + `trace.recordOutcome`), used here so we can `flush()` and read the span back. +- **The OFF proof, by execution** — at OFF there is no intelligence spawn, so the exported span's + `intelligence_usd` is `0` by construction. The example digs it out of the OTLP payload and asserts it. + +## Going live + +Drop the local collector: set `TANGLE_API_KEY` and point `endpoint` at your real OTLP/HTTP collector +(or omit `endpoint` to use `OTEL_EXPORTER_OTLP_ENDPOINT`). Raise `effort` from `off` to `standard`/`max` +to enable the intelligence tiers — the same wrapper, one field changed. diff --git a/examples/intelligence-drop-in/intelligence-drop-in.ts b/examples/intelligence-drop-in/intelligence-drop-in.ts index 9cd3518f..168632a8 100644 --- a/examples/intelligence-drop-in/intelligence-drop-in.ts +++ b/examples/intelligence-drop-in/intelligence-drop-in.ts @@ -1,17 +1,18 @@ /** * Tangle Intelligence — the Observe + Mode-0 drop-in. * - * Wrap any `(input) => Promise` agent, ship one trace per call to - * Tangle Intelligence, and pay only inference at the OFF tier. The wrapper is - * best-effort: a live agent never fails because Intelligence is down. + * Wrap any `(input) => Promise` agent, ship one trace per call to Tangle Intelligence, and pay + * only INFERENCE (the base model stream) at the OFF tier. "Mode 0" = the OFF tier: telemetry stays on, + * but intelligence spend (analysts, corpus, extra spawns) is clamped to 0. The wrapper is best-effort: + * a live agent never fails because Intelligence is down. * - * This example stands up a throwaway local OTLP collector so the trace is - * visible with no credentials, then shows the three load-bearing guarantees: + * This stands up a throwaway local OTLP collector so the trace is visible with NO credentials, then + * proves the three load-bearing guarantees BY READING THE EXPORTED SPAN (not by asserting a label): * 1. a trace lands on a successful turn, * 2. the agent still answers when the trace endpoint is dead, - * 3. effort:'off' runs as pure passthrough with intelligenceUsd = 0. + * 3. at effort:'off' the exported span carries `intelligence_usd = 0` — the billing floor. * - * Run: pnpm tsx examples/intelligence-drop-in/intelligence-drop-in.ts + * Run ($0, no creds): pnpm tsx examples/intelligence-drop-in/intelligence-drop-in.ts */ import { createServer } from 'node:http' import { @@ -19,13 +20,38 @@ import { withTangleIntelligence, } from '@tangle-network/agent-runtime/intelligence' -// A trivial agent — any async input → output works. async function supportAgent(input: { question: string }): Promise<{ answer: string }> { return { answer: `You asked: ${input.question}. Here is a helpful reply.` } } +/** Dig the intelligence-usd attribute out of an OTLP payload. The exporter emits it loop-prefixed as + * `loop.tangle.usage.intelligence_usd`, with the value under `doubleValue` or (for 0) `intValue: "0"` — + * so we suffix-match the key and coerce either numeric form. Walks the whole tree, robust to nesting. */ +function intelligenceUsd(otlpPayload: unknown): number | undefined { + let found: number | undefined + const visit = (node: unknown): void => { + if (!node || typeof node !== 'object') return + if (Array.isArray(node)) return void node.forEach(visit) + const o = node as Record + if ( + typeof o.key === 'string' && + o.key.endsWith('tangle.usage.intelligence_usd') && + o.value && + typeof o.value === 'object' + ) { + const v = o.value as Record + const raw = v.doubleValue ?? v.intValue + if (raw !== undefined) found = Number(raw) + } + for (const child of Object.values(o)) visit(child) + } + visit(otlpPayload) + return found +} + +const settle = () => new Promise((r) => setTimeout(r, 150)) + async function main() { - // ── A local OTLP collector so the trace is visible with no creds ── const received: unknown[] = [] const server = createServer((req, res) => { let body = '' @@ -40,49 +66,46 @@ async function main() { await new Promise((r) => server.listen(0, r)) const port = (server.address() as { port: number }).port const endpoint = `http://127.0.0.1:${port}/v1/otlp` + const project = 'support-agent' - // ── 1. Observe — wrap the agent, run a turn, see the trace ──────── + // 1) THE ERGONOMICS — wrap any `(input) => Promise` in one line; it ships a trace best-effort + // (fire-and-forget: the call returns as soon as the agent does, the span flushes in the background). const wrapped = withTangleIntelligence(supportAgent, { - project: 'support-agent', + project, apiKey: process.env.TANGLE_API_KEY ?? 'sk-tan-demo', endpoint, }) const out = await wrapped({ question: 'how do I reset my password?' }) - console.log('agent output:', out.answer) - - // Force the best-effort batch to flush so we can read it back. - const observeClient = createIntelligenceClient({ project: 'support-agent', endpoint }) - await observeClient.traceRun({ input: { ping: 1 } }, async (t) => { - t.recordOutcome({ success: true, costUsd: 0.0012 }) - return 'pong' - }) - await observeClient.flush() - await new Promise((r) => setTimeout(r, 50)) - console.log(`traces received by collector: ${received.length}`) - console.log(`doctor:`, JSON.stringify(observeClient.doctor().modes, null, 0)) + console.log(`1) wrapped an agent, got its answer: ${JSON.stringify(out.answer).slice(0, 48)}…`) - // ── 2. Survives a dead endpoint — the agent still answers ───────── - const onDeadEndpoint = withTangleIntelligence(supportAgent, { - project: 'support-agent', + // 2) The agent still answers when the trace endpoint is dead — best-effort telemetry never breaks the app. + const onDead = withTangleIntelligence(supportAgent, { + project, endpoint: 'http://127.0.0.1:1/v1/otlp', // nothing listening }) - const stillWorks = await onDeadEndpoint({ question: 'is intelligence down?' }) - console.log('survived dead endpoint:', stillWorks.answer.length > 0) + const stillWorks = await onDead({ question: 'is intelligence down?' }) + console.log(`2) survived a dead endpoint: ${stillWorks.answer.length > 0}`) - // ── 3. Mode 0 / OFF — pure passthrough, intelligenceUsd = 0 ─────── - const offClient = createIntelligenceClient({ - project: 'support-agent', - endpoint, - effort: 'off', + // 3) THE HEADLINE — a trace lands, AND at OFF the exported span carries intelligence_usd = 0. We use an + // explicit client so we can `flush()` and READ THE SPAN BACK off the collector. At OFF there is no + // intelligence spawn, so only the base inference stream costs anything (we record just that); the + // span's intelligence_usd is 0 by construction — a `standard`-tier run would fill it itself. + received.length = 0 + const off = createIntelligenceClient({ project, endpoint, effort: 'off' }) + await off.traceRun({ input: { q: 'cheap turn' } }, async (trace) => { + trace.recordOutcome({ usage: { inferenceUsd: 0.0008 } }) + return supportAgent({ question: 'cheap turn' }) }) - await offClient.traceRun({ input: { q: 'cheap turn' } }, async (trace) => { - // Even if a caller mis-reports intelligence spend, OFF clamps it to 0. - trace.recordOutcome({ usage: { inferenceUsd: 0.0008, intelligenceUsd: 0.05 } }) - return await supportAgent({ question: 'cheap turn' }) - }) - await offClient.flush() - await new Promise((r) => setTimeout(r, 50)) - console.log(`OFF tier intelligence_off:`, offClient.effort) + await off.flush() + await settle() + + console.log(`3a) a trace landed on the collector: ${received.length > 0}`) + const usd = received.map(intelligenceUsd).find((v) => v !== undefined) + console.log( + `3b) OFF tier — intelligence_usd on the exported span: ${usd} (0 = the billing floor)`, + ) + if (received.length === 0) throw new Error('expected a trace to land on the collector') + if (usd !== 0) throw new Error(`OFF tier must export intelligence_usd = 0, got ${usd}`) server.close() } diff --git a/examples/self-improving-loop/self-improving-loop.ts b/examples/self-improving-loop/self-improving-loop.ts index be5ca519..36ceb8dc 100644 --- a/examples/self-improving-loop/self-improving-loop.ts +++ b/examples/self-improving-loop/self-improving-loop.ts @@ -11,9 +11,8 @@ // 5. apply mutation → new AgentProfile variant // 6. re-run multishot with v1 profile // 7. gate pairs v1 vs v0 per persona and ships only if the `pairedBootstrap` CI lower bound -// clears 0 — the production held-out gate's statistical core. NOTE the gate runs at n=3 here -// for a runnable demo; the production gate ALSO enforces a minimum-evidence floor this omits -// (see the ⚠️ at the gate, § 6). Never ship a real change on n=3. +// clears 0 — the production held-out gate's statistical core. (This demo gates at n=3 for a +// runnable example, BELOW the production floor — the one authoritative caveat is the ⚠️ at § 6.) // // The finding type and the gate statistic are the real substrate primitives (not a local one-off); // the analyst body, the proposer, and the LLM are scripted ONLY so the demo runs offline and diff --git a/examples/strategy-suite/README.md b/examples/strategy-suite/README.md index 04a82e50..b7a3d047 100644 --- a/examples/strategy-suite/README.md +++ b/examples/strategy-suite/README.md @@ -12,14 +12,17 @@ free. 1. **Just run it** — `runBenchmark({ environment, tasks, worker })` compares strategies at equal budget and reports the paired lift. -2. **Pick built-ins** — `sample` (N independent attempts, keep the - best-verifying), `refine` (attempt → critic reads the trace → steer the - next → repeat), `adaptiveRefine` (refine, but abandon-and-restart a line - that stops improving), `sampleThenRefine`. -3. **Author your own** — `defineStrategy(name, body)`. A body composes two - steps — `shot()` (one worker attempt over the artifact) and `critique()` - (the firewalled analyst reads the trace → a steer) — with zero - Supervisor/Scope ceremony. The example authors `doubleCheck` inline. +2. **Pick built-ins** — this run compares two: `sample` (N independent attempts, keep the + best-verifying) and `refine` (attempt → critic reads the trace → steer the next → repeat). Two more + ship and swap in the same way: `adaptiveRefine` (refine, but abandon-and-restart a line that stops + improving) and `sampleThenRefine`. +3. **Author your own** — `defineStrategy(name, body)`. A body composes two steps — `shot()` (one worker + attempt over the artifact) and `critique()` (the firewalled analyst reads the trace → a steer) — with + zero Supervisor/Scope ceremony. The example authors **`doubleCheck`**: a policy the built-ins *don't* + have — it never trusts a single passing shot, requiring the solution to pass **twice in a row** before + it stops (a flake/luck guard). `refine` ships on the first pass; `doubleCheck` re-verifies once more. + The payoff is real on a non-deterministic surface (flaky tools/tests) — the whole point of authoring a + stop-condition the library doesn't ship, in ~10 lines. ## Run diff --git a/examples/strategy-suite/strategy-suite.ts b/examples/strategy-suite/strategy-suite.ts index 1f50f062..19a9658b 100644 --- a/examples/strategy-suite/strategy-suite.ts +++ b/examples/strategy-suite/strategy-suite.ts @@ -36,6 +36,12 @@ const task = counterTask('counter-to-5') // shot() = one worker attempt over the artifact; critique() = the firewalled // analyst reads the trace and returns a steer for the next shot. +// An AUTHORED policy the built-ins DON'T have: never trust a single passing shot — require the solution +// to pass TWICE IN A ROW before declaring done (a flake/luck guard). `refine` stops the instant a shot +// passes; `doubleCheck` runs ONE more verification shot and only accepts if that also passes. On a +// deterministic surface this is `refine` + one confirm shot; its real payoff is on a NON-deterministic +// surface (real tools / flaky tests), where a lucky single pass is caught before it ships. That is the +// point of `defineStrategy`: a stop-condition the library doesn't ship, in ~10 lines. const doubleCheck = defineStrategy( 'doubleCheck', async ({ surface, task: t, budget, shot, critique }) => { @@ -44,21 +50,30 @@ const doubleCheck = defineStrategy( let messages: Record[] | undefined let steer: string | undefined let completions = 0 + let consecutivePasses = 0 try { for (let i = 0; i < budget; i += 1) { const out = await shot({ handle, messages, steer }) if (!out) break completions += out.completions progression.push(out.score) - if (out.score >= 1) break messages = out.messages + if (out.score >= 1) { + consecutivePasses += 1 + if (consecutivePasses >= 2) break // passed twice in a row → trust it, stop + steer = 'That passed. Run the checks ONE more time to confirm it is stable, not a fluke.' + continue + } + // A failing shot resets the streak — a single pass is never enough. + consecutivePasses = 0 const findings = await critique(out.messages) completions += 1 if (!findings) break steer = `Not done yet. ${findings}` } + const resolved = consecutivePasses >= 2 const score = progression.length ? Math.max(...progression) : 0 - return { score, resolved: score >= 1, completions, progression, shots: progression.length } + return { score, resolved, completions, progression, shots: progression.length } } finally { await surface.close(handle) }