Skip to content

[GSoC 2026] chatbot: harden jobs-vs-investigations routing + accept null filter args (Refs #3843)#3844

Merged
mlodic merged 1 commit into
developfrom
gsoc-2026/llm-chatbot-toolsel-fix
Jul 7, 2026
Merged

[GSoC 2026] chatbot: harden jobs-vs-investigations routing + accept null filter args (Refs #3843)#3844
mlodic merged 1 commit into
developfrom
gsoc-2026/llm-chatbot-toolsel-fix

Conversation

@berardifra

Copy link
Copy Markdown
Contributor

Refs #3843.

What

Two coupled changes; the second is the null-arg gap the first exposed.

  1. Routing (prompt). The local 3B agent (qwen2.5:3b, temp=0) mis-routed several "jobs" phrasings
    to list_investigations. Strengthen the search_jobs side of system_prompt.txt only (broaden the
    cue with the failing phrasings incl. the verbatim "what jobs do I have?"; one [Rules]
    disambiguation line). list_investigations cue untouched, so its routing controls are unaffected.
  2. Null filter args (tools). With routing fixed, the model calls search_jobs(query=None, ...);
    the query: str = "" / status: str = "" signatures rejected the explicit None, so the answer
    was still empty. Widen query/status to Optional[str] = None on search_jobs and
    list_investigations (signature only — the bodies already guard with if query: / if status:,
    and visible_for_user(user) scoping is unchanged).

Evidence (same-session A/B/C, tool-selection reliability harness)

Correct = expected tool called; "delivered" = the actual job list was returned. Warm n=8, cold n=2.

phrasing before: tool / jobs after: tool / jobs
"what jobs do I have?" 0/10 / 0/10 10/10 / 10/10
"show my recent jobs" 2/10 / 2/10 10/10 / 10/10
"show me my jobs" 8/10 / 8/10 10/10 / 10/10
"list my jobs" / "find my recent jobs" 10/10 / 10/10 10/10 / 10/10
investigations controls; analyze_observable (M-1) / summarize_job controls pass pass (unchanged)

Zero regressions; all controls hold; analyze_observable keeps its M-1 action_required preview.
171 chatbot unit tests green.

Data-isolation note

The tool change is a type-only widening of already-optional filter args; the querysets stay scoped by
visible_for_user(user) and the bodies are unchanged, so a null filter behaves exactly like an
empty one (no widening). New tool tests assert isolation still holds under query=None / status=None
(another user's RED job / a non-shared investigation stay hidden). agent-data-isolation-reviewer was
not spawnable in the authoring session; the review was done inline.

@berardifra

Copy link
Copy Markdown
Contributor Author

Thanks for the review! CI is green and the branch is up to date with develop. I don't have merge rights on develop — could you squash-merge when you have a moment?

@mlodic mlodic merged commit 2ff9022 into develop Jul 7, 2026
10 checks passed
@berardifra berardifra deleted the gsoc-2026/llm-chatbot-toolsel-fix branch July 7, 2026 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gsoc-2026 GSoC 2026 - LLM Chatbot project (Francesco Berardi)

Development

Successfully merging this pull request may close these issues.

2 participants