server: keep SSE streams alive while tool-call output is withheld

[sscarduzio]

Summary

• while a stream withholds output, write an SSE comment : generating every 5 s on the OpenAI chat, Responses and Anthropic paths. Withheld means what the parser holds: suppress/tool mode, a tool call that opened inside the think block (mode stays THINKING until it completes), and Responses reasoning the client did not opt into with reasoning.summary
• one shared helper next to the existing : prefill keepalive; same assumption (SSE parsers drop comment lines), same send_all failure handling
• five no-model regressions over a socketpair: one per stream path, plus the tool-inside-thinking and unemitted-reasoning cases

Why

A multi-thousand-token tool call sends nothing for minutes: the raw text is withheld until the call parses. Clients with an HTTP idle timeout abort at that point (pi and Node's undici both default to 300 s), then retry the identical prompt. On a model whose recurrent state cannot rewind (Qwen 3.8 Flash Next on the qwen branch) the retry is a full re-prefill. Observed on an M1 Max: one 33K-token agent turn cost three 9-minute prefills plus the abandoned generations before the client gave up. The same silence exists for DeepSeek and GLM tool calls on main; those retries are only cheaper, not absent.

Validation

Machine: MacBook Pro M1 Max 32-core GPU, 64 GB, macOS 26.6.2. Backend: Metal.

Unit tests on this branch (main + this commit):

make clean && make -j8            # 0 warnings
make ds4_test && ./ds4_test --server
server: OK
ds4 tests: ok

Live A/B. The change is server-only, so it was exercised through the same diff applied on top of ivanfioravanti/ds4-metal@qwen3.8-flash-next (18ca8ec) serving Qwen3.8-Flash-Next-IQ2XXSImatrix-Q2KDownPad768-MTP.gguf + PLE-Q4_1, --ctx 65536 --prefill-chunk 1024 --mtp. "Keepalive off" is the same build with the helper compiled to a no-op (an env toggle used only for this A/B, since removed). The client is curl -N --speed-limit 1 --speed-time 20: it aborts when fewer than 1 byte/s arrives for 20 s, i.e. a 20-second idle timeout. Each request asks for a write_file tool call carrying a ~900-word essay (~60 s of generation).

endpoint	keepalive off	keepalive on
/v1/chat/completions	last byte at 3 s, aborted at 20 s idle (curl 28)	10 comments, completed in 55 s
/v1/messages	last byte at 3 s, aborted	11 comments, completed in 59 s
/v1/responses	last byte at 3 s, aborted	11 comments, completed in 63 s

Same setup with the real client that hit this: pi (@earendil-works/pi-coding-agent 0.85.1) with its idle timeout set to 20 s and its write tool, keepalive off: terminated, no file, one retry then give-up. Keepalive on: 1,543-word file written, exit 0.

Not run: --logprob-vectors, --long-context and --tool-call-quality need ds4flash.gguf, which is not on this machine. The change does not touch inference, prompt rendering or the transcript; it only writes comment lines to the client socket while nothing else is being written.

🤖 Generated with Claude Code