server: opt-in auto-reduction of over-budget image histories (DS4_VISION_KEEP_IMAGES) - #1022
Open
nazerim wants to merge 1 commit into
Open
server: opt-in auto-reduction of over-budget image histories (DS4_VISION_KEEP_IMAGES)#1022nazerim wants to merge 1 commit into
nazerim wants to merge 1 commit into
Conversation
Agent loops that attach a screenshot per tool round accumulate images without bound and today hard-fail past the limit. Dropping older images is intentionally lossy API policy, so this is opt-in: with the environment variable DS4_VISION_KEEP_IMAGES=N (1 <= N <= 1024) set at server start, a request whose history carries more than N images is served by keeping the last N (the freshest views matter most to the agent) and replacing each dropped image sentinel in the rendered transcript with a fixed text note, so the model still sees an honest history, marker/span counts stay consistent for the vision sync, and the reduced text remains a deterministic cache key. Unset preserves current behavior exactly: requests over 16 images are rejected, and the error message now names the opt-in. A 1024 hard guard still fails absurd requests before any work. Sliding-window note: as the keep-window advances, the newly omitted image's text changes, invalidating cached prefixes only back to the previous retained image.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #971 (rebased onto current main as
fa0af86; the encode-loop hunk auto-merged with 81a7658, only the doc moved from README to docs/SERVER.md).Why. Coding agents that screenshot per tool round accumulate image histories past the 16-image default and get hard 400s at exactly the moments the context matters most. Dropping the oldest images is what the model's own context window would do anyway - but that's a lossy policy change, so it stays strictly opt-in: unset env = today's reject-400 behavior verbatim (with an error message that now hints at the knob).
What.
DS4_VISION_KEEP_IMAGES=N(1..1024, validated once; invalid values log a warning and fall back to reject): keep the last N images, splice a fixed visible note over each dropped sentinel in the prompt text so the transcript stays coherent ([oldest image omitted to fit the vision budget]), and serve. Cache identity is unaffected (markers are canonicalized, and the reduction is deterministic for a given history length, so disk tiers key consistently).Validation (this head): clean optimized build, ds4-server unit suite green; live on M5 Max: keep=16 with a 20-image request served with kept 16 / omitted 4, invalid env value warned and rejected, 17 rejected by default with the hint, keep=2 worked, and the production session ran 128-image-budget turns for a week (130 -> kept 128 / omitted 2 on the largest replay).