feat(routing): add a TypeSafe-backed classifier router

[pst2154]

Summary

Explore and implement a TypeSafe System One routing option for Switchyard. TypeSafe would make a typed Choice over configured target labels, while Switchyard remains responsible for executing the selected target.

Motivation

The existing LLM-classifier path uses a generative model for the routing decision. A small live smoke comparison found that TypeSafe made the same 10/10 clear routing decisions with materially lower observed decision latency:

Router	Correct	Mean	Median	Observed p95
TypeSafe jev-latest	10/10	281 ms	278 ms	375 ms
azure/openai/gpt-5.6-sol	10/10	1,653 ms	1,575 ms	2,576 ms

This is exploratory evidence, not a production-quality accuracy result. The cases were deliberately separable and each request ran once.

Proposed behavior

• Send normalized request state to TypeSafe as one typed Choice.
• Define each configured target with a stable label and semantic criterion.
• Route high-confidence choices to the selected target.
• Route low-confidence choices and TypeSafe service failures to a configured capable fallback.
• Load the TypeSafe credential from an environment variable; never place it in TOML or logs.
• Preserve the full returned probability distribution for telemetry and later threshold calibration.

Architecture question

switchyard-libsy is intentionally I/O-free, while TypeSafe is an external HTTP judgment service. The implementation should preserve that boundary. Likely options are a generic external-decision step served by the runner, or a runner-owned classifier provider that returns a routing decision to libsy. Direct HTTP calls from a libsy algorithm would violate the current architecture.

Acceptance criteria

• Agree on the I/O boundary and configuration shape.
• Add a TypeSafe-backed routing implementation with environment-based authentication.
• Validate target labels, fallback target, and confidence threshold.
• Fail open to the configured capable target on low confidence or service failure.
• Add mocked contract, configuration, fallback, and server integration tests.
• Record decision latency, selected label, confidence, and failure/fallback reason without logging secrets.
• Evaluate on representative labeled Switchyard traffic before making production quality claims.
• Document setup and operational behavior.

[cm-morrissey]

This ended up being exactly what I was already building for myself, so I went ahead and put together a PR for it: #739

Would love any feedback!

[lmvdz]

I opened #762 as a benchmark-informed alternative built on #739's runner-owned client, secret handling, and fail-open design.

The main contract difference is that type_safe_classifier now compares the user's configured target names directly. Targets can carry reusable routing descriptions, with per-route overrides. The client also averages up to three fixed candidate orders and records the full probability distribution plus decision latency. This keeps the data needed to calibrate the policy later instead of retaining only the winning label and confidence.

The change came from a paid 20-task Harbor/Terminal-Bench Lite run on Daytona. Estimated model API spend across all four arms was $86.24:

Arm	Solved	Estimated API cost
All Opus 4.7	17/20	$40.49
All Kimi K2.7 Code	14/20	$4.79
Existing Gemini classifier	8/20	$7.96
Prototype Jev Choice + Noul router	14/20	$33.00

The prototype Jev policy chose Opus for 13 tasks and Kimi for 7. It saved 18% against all-Opus, but lost three completions and tied all-Kimi at almost seven times its cost. It did uniquely solve one task missed by all three other arms, so the result supports routing as a concept while showing that this policy was not calibrated well enough.

We then reused the paid fixed-arm outcomes and made TypeSafe-only calls:

• A feat(routing): add type_safe_classifier backed by TypeSafe's Jev #739-style generic direct selector at threshold 0.6 chose Opus for 16/20 tasks and projected 16/20 completions. Its strong-only AUC was 0.328.
• On 80 additional task openings with three repeats, it chose the capable tier 203/240 times and was route-unanimous on 79/80 tasks. It was stable, but strongly biased toward the expensive tier.
• A separate 20-task option-order probe saw a median maximum probability movement of 9.5 percentage points and 2/20 top-choice flips.
• A 900-request TypeSafe-only stability study completed without errors.

Those results motivated direct configured candidates, explicit descriptions, deterministic order averaging, full probabilities, latency evidence, and guidance to pin the Jev model. They do not establish a calibrated production policy; #762 says that directly and recommends conservative fallback plus held-out threshold validation.

[lmvdz]

The current evidence also suggests a concrete next phase before treating TypeSafe routing as calibrated.

Define the decision we want to calibrate

The router should estimate the incremental value of each configured target for the current request. For two models, the useful quantity is not “does this look difficult?” or even “will the strong model succeed?” It is the probability that the strong model succeeds and the weak model fails, offset by the reverse case, serving cost, and latency.

A simple policy can choose the strong model when:

expected success uplift × value of one success > incremental model cost + latency penalty

That policy should also support a hard budget ceiling and a conservative fallback. The value and cost terms must be deployment settings rather than properties baked into the classifier.

For more than two candidates, a 2^N joint-outcome label does not scale. The research should compare two practical alternatives:

1. Estimate success probability for each candidate, then select by expected utility.
2. Estimate pairwise candidate uplift and combine the comparisons into a ranking.

Directly asking Jev which model is “best” remains a useful baseline, but our results show that its absolute probabilities cannot yet be treated as calibrated utility estimates.

Collect held-out serving outcomes

The existing 20 tasks have already influenced the policy and calibration work, so they cannot validate it. The next paid experiment should:

• Freeze the router prompt, candidate descriptions, option orders, Jev version, serving model versions, agent version, and task harness before any outcomes are observed.
• Start with the prepared 18-task stratified pilot (6 high, 6 middle, 6 low according to the saved TypeSafe ranking) to test the protocol and estimate variance.
• Run every candidate model on every selected task so routing policies can be replayed against paired outcomes.
• Use deterministic task verifiers, keep every valid first attempt, and report infrastructure failures separately.
• Repeat serving runs where the pilot shows enough model variance to change the conclusion. A single run per model/task is too noisy to estimate small uplifts reliably.
• Choose the size of the larger held-out set from a power or precision calculation based on the pilot. Do not select it by looking at which tasks favor a policy.

The 18-task set is a protocol pilot, not enough evidence by itself for a production claim.

Fit calibration without leaking the test set

For each task, save the full TypeSafe distributions for every fixed option order and repeat. Split tasks by task family, rather than random rows from the same family, to reduce leakage.

Fit only on the calibration partition. With the expected small sample, start with a low-parameter recalibration such as logistic or isotonic calibration and compare it with the uncalibrated score. Keep the final evaluation partition untouched until the policy, threshold, and cost function are frozen.

Report:

• Brier score and log loss for probability quality.
• Reliability plots and expected calibration error, with confidence intervals.
• Ranking metrics such as ROC AUC for strong_only, while recognizing that AUC alone does not establish useful probabilities.
• Route share, accuracy, cost, latency, cost per solved task, and regret against fixed weak, fixed strong, and per-task oracle arms.
• Sensitivity to thresholds, success value, cost assumptions, and option order.

The main acceptance test should be held-out decision value: a policy should improve on the chosen fixed-model baseline at a predeclared cost or quality constraint.

Research questions still open

• Order effects: Is averaging two orders sufficient, or does a three-order ensemble materially improve held-out decisions? Duplicate questions in one request may have correlated errors, so same-request and separate-request repeats should be compared.
• Target descriptions: How much do model identity, benchmark facts, price, context length, tool support, and latency each affect the distribution? Descriptions should be tested for robustness and prompt-injection resistance.
• Confidence semantics: The API confidence field is a concentration transform of the returned distribution. Research should use the full probabilities and test whether either signal predicts correctness or uplift after recalibration.
• Model-pair transfer: Does a calibration fitted for Opus/Kimi transfer to other pairs or to a model upgrade? The default assumption should be no until measured.
• Domain transfer: Evaluate coding, retrieval, long-context, tool-heavy, and non-coding tasks separately. The present Harbor sample covers only one agentic coding distribution.
• Drift: Pin Jev and serving versions for evaluation, record them in telemetry, and define when model, prompt, agent, or harness changes require recalibration.
• Operational behavior: Measure TypeSafe p50/p95 latency, rate limits, 429/5xx behavior, malformed responses, and fail-open route share under load.
• Abstention: Test whether low-margin or out-of-distribution requests should fall back to a configured default rather than forcing a target choice.

Before spending on another serving run, the experiment manifest, frozen policy, budget, primary metric, and stopping rule should be posted for review. That will make the next dollars answer a calibration question instead of producing another policy-shaped in-sample comparison.

[lmvdz]

One follow-up after reading Jev’s Architecture Unmasked: its black-box results support the shape of the classifier in #762, while also narrowing what the next experiment needs to establish.

The useful findings are behavioral. Jev returns a predictive distribution, while its confidence field is derived from that distribution’s concentration. Questions share state but do not condition on sibling questions. Choices are sensitive to option order, and adding an option can change the odds between existing options. Together, those findings support three decisions already made here:

• keep the full probability distribution and use confidence only as diagnostic evidence;
• represent both_succeed, strong_only, weak_only, and neither_succeeds as one mutually exclusive Choice; and
• treat the candidate set, descriptions, and fixed order ensemble as part of the calibrated contract.

The article’s proposed causal backbone, shared KV state, listwise readout, and sparse experts are plausible explanations, not published TypeSafe guarantees. None of the routing code needs to depend on them.

It did expose one concrete bug in my benchmark prototype. The calibration snapshot named jev-1.13.0, but the runtime still requested the moving jev-latest alias. I fixed that locally: calibrated runs now default to jev-1.13.0, reject snapshots for a different Jev version, and use the pinned model in both benchmark configurations. A live request to the explicit version returned HTTP 200 and identified the model as jev-1.13.0.

Calibration is now the main open problem. The current 20-task set has already influenced the prompt, order ensemble, prior correction, and cost policy, so it is development data. It cannot provide an unbiased estimate of the resulting router.

For the next paid run, I propose freezing a manifest before looking at any new outcomes. It should pin the Jev version, question text, candidate labels and descriptions, order ensemble, serving-model versions, agent and harness versions, retry policy, verifier, and routing utility. The utility must state the value of a successful task, incremental model cost, latency penalty, budget constraint, and fallback behavior.

The experiment should then:

1. Select a new task-family holdout without using Jev scores or serving outcomes.
2. Run every candidate model on every task. Repeat first attempts where pilot variance could change the conclusion.
3. Save every raw Jev distribution for each fixed option order and repeat.
4. Split by task family into calibration and final evaluation partitions. Related tasks and repeated runs must stay in the same partition.
5. Fit the calibration and routing policy without opening the final partition, then evaluate once under the frozen acceptance rule.

For two candidates, the four-outcome distribution gives the decision-relevant uplift directly:

P(strong_only) - P(weak_only)

The policy can compare that uplift with incremental cost and latency. This keeps probability estimation separate from the deployment decision.

The calibration comparison should include the raw Jev distribution, the current global prior-shift correction, and a regularized temperature or multiclass logistic model. A more flexible method such as Dirichlet calibration only makes sense if the new sample is large enough. Method selection belongs inside the calibration partition or nested cross-validation.

Probability quality should be reported with multiclass log loss, Brier score, classwise reliability, and uncertainty intervals. The routing result should report success, route share, total cost, latency, cost per solved task, and regret against fixed-weak, fixed-strong, and per-task-oracle policies. Repeated calls should be clustered by task when estimating uncertainty.

The acceptance test should be held-out decision value under a predeclared cost or quality constraint. A better AUC or calibration plot is useful diagnosis, but it is not enough if the resulting routes do not improve the deployment objective.

Several research questions remain:

• Order and candidate-set effects: compare one, two, and three fixed orders, and compare same-request permutations with separate requests because sibling outputs may have correlated errors. Test whether adding, removing, or renaming a candidate changes existing pairwise odds.
• Description content: ablate model identity, benchmark evidence, price, context length, tool support, and latency. Aggregate benchmark scores should remain prior evidence rather than calibration labels.
• Transfer and drift: assume calibration does not transfer across Jev releases, serving-model upgrades, agent changes, harness changes, candidate sets, or task domains until measured.
• Abstention: define behavior for low-margin and out-of-distribution requests instead of forcing every request to a candidate.
• More than two candidates: compare per-candidate success estimates, pairwise uplift, and direct selection. A joint 2^N outcome space will not scale.
• Operations: measure p50/p95 latency, rate limits, transient failures, malformed responses, and fail-open route share under realistic concurrency.

The next experiment should spend its budget on held-out workflow outcomes and repeated evaluation under shift, which will now teach us more than another aggregate benchmark score.

[cm-morrissey]

This is a nice follow-up — thanks for putting real benchmark work behind it.

The empirical case for moving off a single threshold/category selection is convincing on its own. A few things I liked in the implementation specifically:

• Getting the multi-order ensemble into one /v1/systemone call instead of three separate requests is a nice trick — no extra latency or cost for the debiasing.
• Keeping the full probability distribution and decision latency is exactly what's missing from feat(routing): add type_safe_classifier backed by TypeSafe's Jev #739 to make the calibration work in feat(routing): add a TypeSafe-backed classifier router #723's acceptance criteria actually possible later.
• Reusing Category::Named for direct target comparison instead of adding new plumbing keeps this consistent with how the custom classifier already works.

Happy to see this take over #739's contract if that's the direction maintainers want — the goal there was just to get a working baseline down, and this clearly goes further with better evidence behind it.

[x-itm]

One implementation detail worth making explicit in the contract: treat the candidate-order ensemble as versioned routing policy, not just a client-side optimization. Persist the ordered label list for each sub-query, ensemble size/order seed (or fixed permutation set), raw per-order distributions, and the aggregation rule alongside the selected label. That makes later calibration reproducible and lets telemetry distinguish model drift from a changed candidate set or ordering policy.

I’d also keep low_confidence, invalid_response, timeout, and transport/auth failure as distinct fallback reasons. They imply different operational actions and should not be collapsed into a single “fallback” metric.

[lmvdz]

I do not have the required funds to do a lot of benchmarks, hopefully some of you do

[x-itm]

That constraint is reasonable. A useful next step need not be a large paid study: publish a small, versioned fixture set plus a runner that records model/version, candidate order, latency, distributions, aggregation, and cost when available. Contributors can then add runs incrementally, while keeping exploratory results clearly separated from production claims.

[simonbright]

The #739-style classifier, at threshold 0.6, was route-unanimous on 79 of 80 tasks and chose the capable tier 203 of 240 times. If the route is almost always the expensive model, that run measured the model, not the router.

The best model changes by task family. On my own scenarios — 24 models, 11 analytical families, 242 scorecards, $0.44 — a classifier calibrated on one family does not transfer to another. Those scenarios never touched this policy, so they are held out of it.

I'll run that family comparison on them this week.

[x-itm]

Agreed: a route-unanimous run that overwhelmingly selects the capable tier is primarily a safety/capability result, not evidence of useful routing discrimination or savings. The held-out family comparison should make that distinction explicit: report per-family calibration/selection behavior, quality versus an always-capable baseline, and cost/latency deltas. Keeping those scenarios out of policy design is important; otherwise apparent transfer can be policy leakage.

[x-itm]

Agreed: a 203/240 capable-tier selection rate makes aggregate quality/latency hard to attribute to the routing policy. The held-out family comparison should report per-family route mix, per-tier outcome, and the counterfactuals against fixed cheap/capable baselines—not only the aggregate score. That will distinguish calibration failure from a policy that simply defaults upward. Recording threshold and any abstain/escalate behavior per family would also make the transfer question testable.

[simonbright]

The run contradicts what I posted. 11 analytical families, 24 models, 242 ended scorecards, estimated $0.4405. One distinct winner across all 11: inclusionai/ling-2.6-flash, the cheapest passing model on every family. Of the 110 off-family rows — winner picked on family i, scored on family j — 110 pass and 110 also win. I said a classifier calibrated on one family would not transfer to another. On this freeze the winner is constant across all eleven, so a cheapest-pass router has nothing to switch among them. I conflated family with domain. To be precise about what this measures: winner stability, not classifier transfer. No classifier was trained or run on these families.

Two limits on my own result before anyone else raises them. The eleven families are mine and share a pass bar — ended, score ≥ 7.0, trust not low — though the rubrics differ by family. "One model clears all eleven" may reflect where I set that bar as much as task homogeneity; eighteen of the twenty-four models pass all eleven at it. And I cannot extend this across domains cleanly: my conversational rankings use an efficiency score, mean rubric ÷ (latency × cost factor), on different model slates rather than cheapest-pass on these 24, so the two are not comparable. The scenarios never touched your policy, so that leakage path is closed. These two are not.

On your list, what this freeze can and cannot give. No router ran, so route mix, per-family calibration and abstain/escalate are not available. Latency is out — this freeze predates metered turn latency and carries wall-clock only. The counterfactual is available by lookup, so here it is: always-cheap is inclusionai/ling-2.6-flash, 11/11 pass, estimated $0.001173. Always-capable is anthropic/claude-haiku-4.5, 11/11 pass, estimated $0.1495. Same pass outcome at roughly 127× the cost. Always-cheap is also the per-family oracle here, so regret is zero.

Freeze batch 20260711_215. 242 ended of 264 attempted; 22 errors (openai/gpt-5-nano 11, nex-agi/nex-n2-mini 10, qwen/qwen3-32b 1). Dollars estimated, not billed. https://www.vantageai.cc/runtimeai/benchmarks/expert-tasks

[simonbright]

Published: https://www.vantageai.cc/runtimeai/benchmarks/family-comparison

One thing I did not have when I posted the counterfactual above, and it changes what that freeze is worth. The per-family score delta between the cheapest passer and the priciest passer is zero — on all eleven families, not approximately. The entire 110-row off-family matrix contains three distinct scores: 7.6, 8.0, 8.4. With eighteen of twenty-four models clearing the bar, that slate has no headroom, so it cannot separate routing policies at all. A router measured on it would look perfect.

Also correcting my own framing above: "always-capable" there is just the priciest model that passes all eleven. The freeze records no capability, size or ranking field, so that axis is priced rather than measured, and I should not have implied a capability tier.

Where this bears on the fixture set @x-itm proposed: a fixture set assembled from tasks that everything passes will reproduce exactly this. Whatever goes into it needs cases where the cheap candidate is reliably wrong, otherwise the recorded distributions are measuring the fixtures rather than the router.

[IRONICBo]

One lesson from using this pattern in Jev Social: candidate-set drift should be recorded separately from model drift. Our action set changes after each browser observation, and attempted operations are removed before the next typed choice. For Switchyard, I would persist the candidate-set hash plus ordered labels/descriptions, selected target, full distribution, Jev model revision, and distinct fallback reason; only replay or compare records when the set and schema match. I would also evaluate an explicit abstain/fallback outcome rather than relying on a confidence threshold alone, because adding or renaming a target can move the whole distribution. The latest family comparison is a useful warning too: fixtures need paired cases where candidates actually differ in outcome, otherwise calibration cannot identify routing uplift.

[simonbright]

IRONICBo's point that adding or renaming a target moves the whole distribution is the one claim here that is asserted rather than measured. The order effect has a number in this thread — lmvdz's 20-task probe, 9.5 percentage points median movement in the maximum probability and 2 of 20 top-choice flips. Set changes have none.

That matters because the two need different fixes: a fixed-order ensemble averages out order and does nothing about a candidate being added or removed. If the candidate-set hash is being persisted anyway, comparing distributions across records that differ only in set size gets the set effect out of data you are already collecting.