feat(conversation): materialize non-inline multipart content to the workspace#4113
Draft
jpshackelford wants to merge 1 commit into
Draft
feat(conversation): materialize non-inline multipart content to the workspace#4113jpshackelford wants to merge 1 commit into
jpshackelford wants to merge 1 commit into
Conversation
Add a polymorphic materialization contract on BaseContent (mirroring to_llm_dict) and a single conversation-layer interceptor that writes non-inline multipart content (images today; files/audio/blobs later) to the workspace and injects a path pointer into the event's extended_content. - BaseContent.materialize(workspace) -> list[MaterializedRef], default no-op - ImageContent.materialize(): data-URL decode and size-capped http(s) download, written content-addressed for idempotency - LocalConversation._materializing wraps the _on_event chain, mirroring _rules_injecting: source-agnostic (user messages + tool/MCP observations) and type-agnostic (dispatches on materialize()) Closes #4111 Co-authored-by: openhands <openhands@all-hands.dev>
Contributor
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
Contributor
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Contributor
Coverage Report •
|
||||||||||||||||||||||||||||||||||||||||||||||||||
This was referenced Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #4111.
When the agent is handed multipart content it cannot act on inline (an image today; files/audio/blobs later), the bytes never land on the sandbox filesystem, so file-oriented tasks (e.g. re-uploading an attachment via a REST API) fail — the agent hunts for a file that doesn't exist, or fabricates a placeholder.
This PR solves the retrieval half of the problem once, at the conversation layer, in a way that is agnostic to both the source and the type of the content, exactly as proposed in the issue.
What changed
1. A polymorphic materialization contract on
BaseContent(mirrors the existingto_llm_dict()contract):ImageContent.materialize()decodes adata:URL or downloads anhttp(s)://URL (size-capped), writes it into the workspace, and returns the path(s).MaterializedRefmodel (path + optional source URL / mime type / size) is exported fromopenhands.sdk.<workspace>/.materialized/with a content-addressed filename (SHA-256 prefix), so identical content always maps to the same file — making writes naturally idempotent. Writing goes throughworkspace.file_upload, so it works for both local and remote workspaces.2. A single conversation-layer
_on_eventinterceptor (LocalConversation._materializing), composed into the callback chain right alongside the existing_rules_injectingwrapper. As each event enters, it:materialize()polymorphically on the event's content parts,TextContentpointer (the workspace path) into the event'sextended_content, under the state lock, before persistence.Because it dispatches on the
BaseContentabstraction and runs for every event flowing through_on_event, it is:materialize()on that type; the hook is never revisited.Per-event materialization is deduped in memory by event id, so re-emitting an event (e.g. replay) never re-runs writes.
Acceptance criteria
materialize()contract onBaseContentwith a workingImageContentimplementation (data-URL decode + size-cappedhttp(s)download)._on_eventinterceptor materializes content into the workspace and injects path pointers intoextended_content, before persistence, under the state lock.materialize()polymorphically; a new content type needs no change here.http-URL download → file (size cap enforced, both viaContent-Lengthand streamed bytes), aTextContentno-op, an event with no materializable parts, and idempotency across replay (no duplicate writes).Out of scope (as noted in the issue)
Testing
tests/sdk/llm/test_content_materialize.py(content-level) andtests/sdk/conversation/test_content_materialization.py(hook-level) — all pass.tests/sdk/conversation/(741) andtests/sdk/llm/(881) suites pass.ruff check,ruff format, andpyrightclean on changed files.This PR was created by an AI agent (OpenHands) on behalf of the user.
@jpshackelford can click here to continue refining the PR
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:2886882-pythonRun
All tags pushed for this build
About Multi-Architecture Support
2886882-python) is a multi-arch manifest supporting both amd64 and arm642886882-python-amd64) are also available if needed