[GSoC 2026] Chatbot: warm KV-cache tool-selection flip picks the wrong tool ('show my recent jobs' → list_investigations)

[berardifra]

Summary

On the LangChain 1.x chatbot, the query "show my recent jobs" picks the wrong tool when the
model is warm and returns a wrong answer. Warm, qwen2.5:3b calls only list_investigations
(empty for the test user) and replies "There are no investigations that match your filters." — even
though the user has 3 jobs. Cold (model just loaded), the same query runs the correct
list_investigations → search_jobs path and returns the 3 real jobs.

This is a correctness / reliability issue, not a latency one (see below). It is a new
manifestation of the tool-calling brittleness first documented in the 2026-06-21 latency benchmark
(Finding 1): at temperature=0 the only difference between warm and cold is Ollama's
KV-prefix-cache state vs a full recompute, which tips a borderline tool-selection argmax. PR #3809
(handle_validation_error / on_invalid_tool_args) made schema-invalid tool arguments
recoverable, but it does not touch which tool the model selects — so this failure mode is still
open.

Reproduction (from the 2026-07-03 warm-latency benchmark, qwen2.5:3b, temperature=0)

Query: "show my recent jobs" · user admin (3 seeded jobs, 0 investigations).

condition	tool rounds	tools called	total	answer
warm (model resident)	1	list_investigations	~6.6 s	❌ "There are no investigations that match your filters." (17 chunks) — wrong
cold (model just loaded)	2	list_investigations, search_jobs	~114.7 s	✅ the 3 real jobs (174 chunks) — correct

• Deterministic across all reps: warm 5/5 → 1 round, wrong answer; cold 3/3 → 2 rounds,
  correct answer. temperature=0, so this is reproducible, not a sampling fluke.
• The mechanism is the same as the 2026-06-21 Finding 1 (there, warm tipped summarize_job to emit
  the literal '<job_id>'; here, warm tips "show my jobs" onto list_investigations instead of
  search_jobs).
• The system prompt already disambiguates the two tools
  (api_app/chatbot_manager/agent/system_prompt.txt: search_jobs — Use for "show me jobs";
  list_investigations — Use for "show my investigations"), so this is not a missing-instruction
  bug — it is a model-robustness / prompt-hardening problem.

This is not a latency problem

The same benchmark showed warm latency on 1.x is good: every warm turn completes in ≤ ~31 s
(no-tool ~6 s, tool-backed answers ~27 s), first token in 0.3 s (no-tool) to ~12 s (tool), streaming
steadily at ~8.8 tok/s. A num_predict output cap was explicitly evaluated and rejected by that
benchmark (it would truncate legitimate answers with zero warm benefit). So the fix here is about
tool selection correctness, and it must not be conflated with a latency/output-length change. The
warm 6.6 s for this query is fast only because it gives up early with a wrong answer — it must not be
read as a latency win.

Proposed direction (needs a dedicated PR, not a blind one-liner)

• Harden tool selection for the jobs-vs-investigations distinction — e.g. a sharper contrast in the
  tool descriptions and/or a small few-shot exemplar in
  api_app/chatbot_manager/agent/system_prompt.txt steering "show/list my (recent) jobs" →
  search_jobs.
• Gate the change on a dedicated reliability harness, not a single manual check: run each candidate
  prompt across warm and cold with n ≫ 1 and several phrasings ("show my recent jobs",
  "list my jobs", "what jobs do I have?", …), and assert the correct tool is selected in both cache
  states. Prompt nudges are KV-cache-state sensitive, so a one-off "it works now" is not evidence.
• Keep it a separate PR with its own tests; do not bundle it into any latency/num_predict work.

References

• Benchmark report (GSoC work-product): gsoc/benchmarks/chatbot-latency-2026-07-03-langchain-1x.md
  (warm vs cold numbers, the tool-selection flip finding, and the rejected num_predict cap).
• Prior art: the 2026-06-21 latency benchmark Finding 1 (warm/cold tool-calling flip) and PR [GSoC 2026] chatbot: recover from invalid tool arguments #3809
  (handle_validation_error, which covered invalid tool arguments but not tool selection).

Related (out of scope for this issue)

• Cold-load adds a flat ~+70 s on the first query after the model goes idle. That is an ops
  concern (Ollama keep_alive / model residency / a periodic warmup ping), not a code fix, and is
  intentionally not part of this reliability issue.

[berardifra]

Fix PR: #3844.

Root-caused and validated with a same-session A/B/C re-run of a tool-selection reliability harness
(harness byte-identical across runs; only the prompt/tools changed). The defect had two layers:
(1) tool selection — "what jobs do I have?" routed to list_investigations 0/10; (2) once routing
was fixed, the model called search_jobs(query=None), which the str-typed optional args rejected, so
it still returned no jobs.

Combined fix result:

• "what jobs do I have?": selection 0/10 → 10/10 and job delivery 0/10 → 10/10
• "show my recent jobs" / "show me my jobs" also recovered (delivery 30/50 → 50/50 across the 5 jobs phrasings)
• investigations + analyze_observable (M-1, action_required preserved) + summarize_job controls unchanged
• jobs warm error rate 0.4 → 0.0; 171 chatbot unit tests green

The list_investigations null-arg fix is included proactively (it shared the identical latent gap).

[berardifra]

Fixed by #3844.