Streamed tool-call arguments are truncated to the first fragment (breaks llama.cpp + any server that streams tool_calls deltas incrementally; poisons session history)

[regmibishal1]

Environment

• echokit_server v0.3.2 (release binary), [llm] platform = "openai_chat"
• LLM backend: llama.cpp server (b-series, mid-2026) with --jinja, model Qwen3-30B-A3B-Instruct (also reproduces with Llama-3.1-8B)
• MCP tools via [[llm.mcp_server]] (Docker MCP gateway, http_streamable)
• Device: EchoKit DIY kit over /ws/{id}

Symptom

Ask the assistant anything that triggers a tool call. It speaks the
call_mcp_message ("give me a second...") and then goes silent and every
subsequent turn in the session also fails silently. Server log shows:

INFO  ... tool get_current_time call message: ...
ERROR ... Tool call get_current_time failed
ERROR ... error during llm or tts handling: llm failed, status:500 Internal Server Error,
     body:{"error":{"code":500,"message":"Failed to parse tool call arguments as JSON:
     [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while
     parsing object key - unexpected end of input; expected string literal", ...}}

The echokit log of the parsed tool call shows the truncation directly:

name: "get_current_time",
arguments: "{",

Root cause (two defects in src/ai/mod.rs)

Per the OpenAI streaming spec, tool_calls arrive as delta fragments: the
first fragment carries id/name and an arguments prefix, later fragments
carry only {"index": 0, "function": {"arguments": "<fragment>"}}, and
finish_reason: "tool_calls" marks completion. llama.cpp streams exactly this
way. Cloud providers (e.g. Groq) often emit the whole call in a single delta,
which is why this wasn't visible before.

1. llm::ToolCall / ToolFunction have no serde defaults and no index
   field, so continuation fragments (no id, no type, no name) fail to
   deserialize entirely and fall into the partial-JSON buffer.
2. StableLlmResponse::next_chunk swaps out the first tool_calls delta and
   returns immediately (return Ok(...Functions(tools))), so the call is
   emitted with arguments: "{".
   The mangled call is then stored in the session history; since the backend
   validates history on every request, all following turns 500 until the
   server is restarted — one truncated fragment permanently wedges the session.

Suggested fix (patch attached, happy to turn into a PR)

• Add lenient serde defaults + a deserialize-only index: Option<u32> to
  ToolCall/ToolFunction (index is skip_serializing so outgoing
  requests stay spec-clean).
• Accumulate fragments in a pending_tools: Vec<ToolCall> on
  StableLlmResponse, matched by index (fallback heuristic for
  index-less providers), concatenating arguments.
• Only return Functions(...) on finish_reason == "tool_calls", with an
  end-of-stream flush as a safety net.
  Single-delta providers are unaffected (their one fragment has id+name and
  finish_reason follows immediately). Running this patch locally: tool calls
  from llama.cpp now assemble correctly and sessions no longer wedge.

diff --git a/src/ai/mod.rs b/src/ai/mod.rs
index 6dc700b..c34c32f 100644
--- a/src/ai/mod.rs
+++ b/src/ai/mod.rs
@@ -161,6 +161,9 @@ pub struct StableLlmResponse {
     response: reqwest::Response,
     text_splitter: TextSplitter,
     first_chunk: bool,
+    /// streamed tool-call fragments assembled across chunks (fix for
+    /// servers like llama.cpp that spread arguments over many deltas)
+    pending_tools: Vec<llm::ToolCall>,
 }
 
 impl StableLlmResponse {
@@ -216,6 +219,11 @@ impl StableLlmResponse {
 
             let body = self.response.chunk().await?;
             if body.is_none() {
+                if !self.pending_tools.is_empty() {
+                    let tools = std::mem::take(&mut self.pending_tools);
+                    log::info!("llm streamed tool calls assembled at EOS: {:#?}", tools);
+                    return Ok(StableLLMResponseChunk::Functions(tools));
+                }
                 return self.return_string_buffer();
             }
             let body = body.unwrap();
@@ -231,7 +239,8 @@ impl StableLlmResponse {
             log::trace!("llm response chunk body: {body}");
 
             let mut chunks = String::new();
-            let mut tools = Vec::new();
+            let mut tools_done = false;
+            let pending = &mut self.pending_tools;
             body.split("data: ").for_each(|s| {
                 if s.is_empty() || s.starts_with("[DONE]") {
                     return;
@@ -247,8 +256,35 @@ impl StableLlmResponse {
                         log::trace!("llm response content: {content}");
                         chunks.push_str(&content);
                     }
-                    if !chunk.choices[0].delta.tool_calls.is_empty() {
-                        std::mem::swap(&mut chunk.choices[0].delta.tool_calls, &mut tools);
+                    // Accumulate streamed tool-call deltas. Servers like llama.cpp
+                    // send id/name in the first fragment and spread the JSON
+                    // arguments across later fragments, matched by `index`.
+                    for tc in chunk.choices[0].delta.tool_calls.drain(..) {
+                        let slot = match tc.index {
+                            Some(i) => pending.iter_mut().find(|p| p.index == Some(i)),
+                            None => {
+                                if !tc.id.is_empty() || !tc.function.name.is_empty() {
+                                    None // looks like a new complete call
+                                } else {
+                                    pending.last_mut() // bare fragment: append to last
+                                }
+                            }
+                        };
+                        match slot {
+                            Some(p) => {
+                                if !tc.id.is_empty() {
+                                    p.id = tc.id;
+                                }
+                                if !tc.function.name.is_empty() {
+                                    p.function.name = tc.function.name;
+                                }
+                                p.function.arguments.push_str(&tc.function.arguments);
+                            }
+                            None => pending.push(tc),
+                        }
+                    }
+                    if chunk.choices[0].finish_reason.as_deref() == Some("tool_calls") {
+                        tools_done = true;
                     }
                 } else {
                     chunk_ret.push_str(s);
@@ -256,17 +292,17 @@ impl StableLlmResponse {
             });
 
             log::trace!("llm response chunks: {chunks}");
-            log::trace!("llm response tools: {:#?}", tools);
 
-            if tools.is_empty() {
-                if let Some(new_str) = self.push_str(&chunks) {
-                    log::trace!("llm response text: {new_str}");
-                    return Ok(StableLLMResponseChunk::Text(new_str));
-                }
-            } else {
-                log::trace!("llm response tools: {:#?}", tools);
+            if tools_done && !self.pending_tools.is_empty() {
+                let tools = std::mem::take(&mut self.pending_tools);
+                log::info!("llm streamed tool calls assembled: {:#?}", tools);
                 return Ok(StableLLMResponseChunk::Functions(tools));
             }
+
+            if let Some(new_str) = self.push_str(&chunks) {
+                log::trace!("llm response text: {new_str}");
+                return Ok(StableLLMResponseChunk::Text(new_str));
+            }
         }
     }
 }
@@ -381,15 +417,29 @@ pub mod llm {
 
     #[derive(Debug, serde::Serialize, serde::Deserialize, Clone)]
     pub struct ToolCall {
+        // Streamed deltas may omit id/type/name and carry only an `index`
+        // plus an arguments fragment — all fields need lenient defaults.
+        #[serde(default)]
         pub id: String,
-        #[serde(rename = "type")]
+        #[serde(rename = "type", default = "ToolCall::default_type")]
         pub type_: String,
+        #[serde(default)]
         pub function: ToolFunction,
+        #[serde(default, skip_serializing)]
+        pub index: Option<u32>,
     }
 
-    #[derive(Debug, serde::Serialize, serde::Deserialize, Clone)]
+    impl ToolCall {
+        fn default_type() -> String {
+            "function".to_string()
+        }
+    }
+
+    #[derive(Debug, serde::Serialize, serde::Deserialize, Clone, Default)]
     pub struct ToolFunction {
+        #[serde(default)]
         pub name: String,
+        #[serde(default)]
         pub arguments: String,
     }
 
@@ -515,6 +565,7 @@ pub async fn llm_stable<'p, I: IntoIterator<Item = C>, C: AsRef<llm::Content>>(
         first_chunk: true,
         response,
         text_splitter: TextSplitter::new(),
+        pending_tools: Vec::new(),
     })
 }
 
@@ -1535,6 +1586,7 @@ impl ResponsesLLmResponse {
                                         id: call_id,
                                         type_: "function".to_string(),
                                         function: llm::ToolFunction { name, arguments },
+                                        index: None,
                                     });
                                 }
                                 ResponsesOutputItem::Message { id, role, content } => {
diff --git a/src/services/realtime_ws.rs b/src/services/realtime_ws.rs
index 282730f..0a206f8 100644
--- a/src/services/realtime_ws.rs
+++ b/src/services/realtime_ws.rs
@@ -561,6 +561,7 @@ async fn handle_client_message(
                                                 name: item.name.clone().unwrap_or_default(),
                                                 arguments: arguments.clone(),
                                             },
+                                            index: None,
                                         }]),
                                         tool_call_id: None,
                                     });