server: opt-in auto-reduction of over-budget image histories (DS4_VISION_KEEP_IMAGES) - #971
Closed
nazerim wants to merge 1 commit into
Closed
server: opt-in auto-reduction of over-budget image histories (DS4_VISION_KEEP_IMAGES)#971nazerim wants to merge 1 commit into
nazerim wants to merge 1 commit into
Conversation
Agent loops that attach a screenshot per tool round accumulate images without bound and today hard-fail past the limit. Dropping older images is intentionally lossy API policy, so this is opt-in: with the environment variable DS4_VISION_KEEP_IMAGES=N (1 <= N <= 1024) set at server start, a request whose history carries more than N images is served by keeping the last N (the freshest views matter most to the agent) and replacing each dropped image sentinel in the rendered transcript with a fixed text note, so the model still sees an honest history, marker/span counts stay consistent for the vision sync, and the reduced text remains a deterministic cache key. Unset preserves current behavior exactly: requests over 16 images are rejected, and the error message now names the opt-in. A 1024 hard guard still fails absurd requests before any work. Sliding-window note: as the keep-window advances, the newly omitted image's text changes, invalidating cached prefixes only back to the previous retained image.
This was referenced Sep 4, 2026
cropduster
added a commit
to cropduster/ds4
that referenced
this pull request
Sep 9, 2026
…compaction, hints label Resolution: - ds4_server.c = upstream side wholesale: upstream's newer vision-cache architecture (server_image_cache, visible_image_key, rebase_vision_state, canonicalized tool checkpoints) functionally subsumes our fork's vembed_cache/raw_visible_len vision-cluster commits (upstream PRs antirez#969-antirez#971 lineage, since evolved). Our antirez#924-equivalent was implemented upstream by antirez. - ds4.h = union (keep fork's vision-identity prototypes + upstream rebase_vision_state) - ds4.c = keep fork's sync_impl/preflight structure + upstream checkpoint_valid guard
Author
|
Closing for rebase, not abandoning: the opt-in DS4_VISION_KEEP_images budget remains upstream-absent after 81a7658. Superseded-tree conflicts + our fork just merged upstream through 6289c51, so I'll re-validate on the merged tree and resubmit clean if it's still additive. Live-validated at keep=16 (20-image request -> kept 16/omitted 4 with drop-oldest note), invalid-env rejection, and default reject>16 with hint. |
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up split from #968 per @JordiPosthumus's scope review: automatically
discarding older images is an intentionally lossy API policy, so it lives in
its own PR and is explicitly opt-in.
Behavior:
rejected; the error message now names the opt-in.
history carries more than N images is SERVED by keeping the last N (the
freshest views matter most to agent loops) and replacing each dropped image
sentinel in the rendered transcript with a fixed text note, so the model
still sees an honest history, marker/span counts stay consistent for the
vision sync, and the reduced text remains a deterministic cache key.
values are ignored with a one-time warning and the reject policy stays.
Why: screenshot-per-tool-round agent sessions accumulate images without
bound. The current alternative for such clients is a session that stops
working at the 17th screenshot with no graceful path; truncation with an
honest transcript note keeps them running and cache-friendly.
Sliding-window note: as the keep-window advances, the newly omitted image's
text changes, invalidating cached prefixes only back to the previous
retained image.
Verification (on this exact submitted code):
newer markers survive verbatim, drop-all keeps surrounding text, lost
sentinel reported instead of misrendered). ds4-server tests: ok.
request served; log: "multimodal image budget: 20 images, kept last 16,
omitted 4 oldest".
rejected with HTTP 400 and the hint message naming the opt-in.
reject policy retained.