Any plans to support Claw-Eval benchmark?

[passionate11]

Hi team! First of all, thank you so much for EcoClaw — the idea of routing to the cheapest capable model based on real benchmark data is brilliant, and the cost savings are very impressive. Your research work on LLM routing (Avengers, AvengersPro, LLMRouterBench, etc.) is also incredibly valuable to the community. Great job! 🙌

I noticed that EcoClaw currently uses PinchBench data for model selection and scoring. I'm curious whether there are any plans to support Claw-Eval as an additional benchmark source?

Claw-Eval is an end-to-end benchmark specifically designed for AI agents acting as personal assistants, with 104 tasks, 15 mock enterprise services, Docker sandboxes, and deterministic grading. It could be a great complement to PinchBench for evaluating model capabilities in agentic scenarios.

Specifically, I'm wondering:

1. Would it be feasible to incorporate Claw-Eval scores into EcoClaw's routing decisions (e.g., as an additional signal alongside PinchBench)?
2. Or has there been any testing of EcoClaw's routing accuracy against the Claw-Eval benchmark?

Thanks again for your amazing work, and looking forward to hearing your thoughts!

[shuyuehu]

Good to know about your work. Please allow us a few days to take a look at your work.

[passionate11]

Hi @shuyuehu, thanks for the reply!
Just to clarify — Claw-Eval is not my work. It's an independent, more comprehensive agent benchmark (think of it as a broader "Claw Bench") developed by others in the community.
I brought it up because I think running EcoClaw's routing against this benchmark could be a great way to further validate the effectiveness of your eco-routing approach. The diverse task coverage and deterministic grading in Claw-Eval would provide additional evidence beyond PinchBench, especially for agentic scenarios.
Would love to see some results if you get a chance to test it out!