Add Cursor (CLI, Kimi K3) to the evaluated harness set

[bompus]

Thanks for publishing this. Holding model, tasks, and runtime constant and varying only the harness is the comparison that has been missing, and the token-weighted cache finding on Claude Code alone justifies the exercise.

Since v1.1 is planned to add harnesses: could Cursor be included?

Why it belongs

• It is one of the most widely used coding agents, and its agent loop (tool set, context assembly, prompt caching, search) is its own harness, not a wrapper over one already in the table. It would be a distinct point on the pass-rate vs. cost-per-pass chart, and for many readers the first one they look for.
• Kimi K3 is an officially supported Cursor model (Settings → Models, and --model kimi-k3 in the CLI), so the benchmark's fixed-model constraint holds. No workaround or unsupported configuration is needed.
• The Cursor CLI runs headless (agent -p --output-format json --force --trust --api-key …), so it fits the checkpoint-restore flow and the Harbor/Pier split from Add evaluation workflow for third-party harnesses #1. If Harbor has no built-in adapter, the custom-agent registration path in Add evaluation workflow for third-party harnesses #1's reference.md should cover it.

Where it deviates from the other twelve, so the row is honest

1. Provider. Cursor supports bring-your-own keys and an OpenAI base URL override in the IDE (Settings → API Keys), so the desktop Agent could in principle hit the same Fireworks gateway as the other rows. The headless CLI, which is the surface a Harbor/Pier checkpoint run can drive, does not honor those settings per Cursor's docs; its calls go through Cursor's backend to Cursor's inference partner for K3 (Moonshot on the pricing page). Same weights, so pass rate is comparable. Cache hit rate may shift with that provider's prefix-cache behaviour, which Add evaluation workflow for third-party harnesses #1's reference doc already treats as a caveat category, and egress would need to allow Cursor's hosts instead of api.fireworks.ai. If Cursor exposes the base-URL override in the CLI, or the maintainers have a way to drive the IDE headlessly, the caveat disappears and the row is a straight peer of the Fireworks rows.
2. Cost accounting. Cursor's listed K3 prices match the Fireworks/Moonshot list prices used for the baselines ($3 / $0.30 cached / $15 per M), plus Cursor's $0.25/M third-party token rate. Since the model calls are not visible on the wire from the runtime, tokens and cached tokens would have to come from Cursor's usage export rather than trajectory wire logs. Whether that export exposes cached-token counts per request at the granularity cache_hit_rate_normalized needs is the one thing I could not confirm from the docs.
3. Versioning. The CLI has a version to pin, but part of Cursor's context assembly runs on its backend, which has none. harness-versions.json would carry the CLI version plus a run date, which is weaker than the other rows but still worth recording.

None of these block a run. They are the same class of caveat the third-party workflow in #1 already reports (non-baseline provider, model mismatch), so a Cursor row would land with a marked provider caveat rather than as an unqualified peer of the Fireworks rows.

If you would rather this arrive as a profile under skills/frontierharness-eval/ in the style of #2, say so and I can look at putting one together once #1 lands.

[SASUKE40]

Thanks for the thoughtful write-up and for laying out the integration details and caveats. We’ll include Cursor in the next version of the benchmark, with any provider, cost-accounting, and versioning differences clearly documented alongside the results. Appreciate the suggestion!