Skip to content

feat(ask): render tool questions as a native form - #854

Open
snowboardit wants to merge 7 commits into
agegr:mainfrom
snowboardit:feat/structured-ask
Open

snowboardit wants to merge 7 commits into
agegr:mainfrom
snowboardit:feat/structured-ask

Conversation

@snowboardit

@snowboardit snowboardit commented Sep 14, 2026

Copy link
Copy Markdown

Problem

Pi has no wire protocol for asking the user a question. Extensions that ask one, such as the ask_user tool from pi-ask-user, build a terminal component and pass it to ctx.ui.custom().

Pi Web runs that component headless, renders it to text lines, and streams the lines to the browser with raw keystrokes forwarded back. That bridge is correct for arbitrary extension UIs, but a text terminal cannot be used with a pointer, and it is close to unusable on a phone. In practice the user reads a fake terminal in an "Extension panel" and then types the answer into the composer by hand.

Answering a question is a core part of an agent session, so it gets a real interface.

Approach

A small adapter layer in lib/structured-ask.ts maps one known question-asking tool onto a structured description (question, context, options, multi-select, freeform, comment), and maps a submitted answer back onto the value that the extension's ctx.ui.custom() promise must resolve with.

The server correlates a custom UI request with the tool call that is running when the request arrives. The extension API supplies no tool identity with ctx.ui.custom(), and a question-asking tool blocks while its question is open, so the running tool calls are both the only correlation available and a sufficient one.

When an adapter matches, the extension_ui_request event carries an ask field and the browser renders a native form. The browser answers with an extension_ui_ask_response command. The server checks the submission against the question it answers, then resolves the extension's promise. A submission that does not fit leaves the question open. No keystrokes are synthesized anywhere.

UI

  • Single select submits on tap, with no extra confirm step.
  • Multi-select uses checkboxes and a Submit button that shows the count.
  • A freeform row accepts a typed answer. In a multi-select, a typed answer travels beside the checked options, so writing your own answer does not cost you your checks.
  • An optional comment field appears once an answer is chosen, when the question allows a comment.
  • The header chevron minimizes the card instead of dismissing it, so the conversation stays readable while the user decides. Only Skip cancels.
  • Keyboard support (digits, arrows, Enter, Escape) sits on top of real buttons rather than replacing them.

Answered questions render as decision cards in the transcript, rebuilt from the tool result details. They are also kept out of the collapsed process-details group, which previously hid every answered question behind a disclosure.

Example (single selection)

Asked question

asked-question

Answered question

answered-question

Fallbacks

  • Any unrecognized custom UI keeps the existing terminal panel, unchanged.
  • A recognized question can still be answered in the terminal view through a header button, so a shape change in the extension cannot lock a user out of answering.

Known limits

  • Pi Web now holds knowledge of one extension's argument and result shapes. That knowledge is confined to one adapter entry, and adding another question-asking extension means adding another entry. docs/adr/0004-structured-ask-adapter.md records the tradeoff and the alternative of upstreaming a descriptor protocol to the extension.
  • The correlation assumes one asking tool blocks at a time. That holds for ask_user. If two asks were ever in flight in one session, the newest match wins.

Testing

  • 12 unit tests for the adapter (parsing, aliases, validation, result rebuilding).
  • 4 protocol tests for the server path (question attached, answer resolves, cancel resolves, mismatched submission leaves the question open, unknown tools stay on the terminal path).
  • 9 source-level UI tests for the card, the transcript card, and the chat wiring.
  • Full suite green. Every commit in this branch passes tsc --noEmit, eslint, and npm test on its own.
  • Exercised manually in a live session on desktop and mobile: single select, multi-select with a typed entry, comment, freeform, dismiss, minimize, and the terminal escape hatch.

Docs

docs/adr/0004-structured-ask-adapter.md plus a file-map entry and a design note in AGENTS.md.

Pi has no wire protocol for "ask the user a question". Extensions build a
terminal component and hand it to ctx.ui.custom(), which leaves the host
with rendered text and no idea what was asked.

Add a registry that maps one known tool call onto a question description,
checks a submitted answer against that question, and produces the value
the extension expects back. The first entry covers the ask_user tool from
pi-ask-user. Option parsing accepts the key aliases models fall back to
when a proxy mangles the schema.

Nothing uses this yet.
Correlate a custom UI request with the tool call that runs when the
request arrives. The extension API supplies no tool identity with
ctx.ui.custom(), and an asking tool blocks while its question is open, so
the running tool calls are both the only correlation available and a
sufficient one.

A recognized question rides along on the extension_ui_request event, and
a new extension_ui_ask_response command answers it. The submission is
checked against the question before the extension's promise resolves; one
that does not fit leaves the question open. Unrecognized custom UIs are
untouched and keep streaming terminal lines.
A terminal panel cannot be used with a pointer and is close to unusable
on a phone, so a recognized question now gets a real form: single select
that submits on tap, multi-select with a count, a freeform row, an
optional comment, and Skip. Multi-select keeps a typed answer beside the
checked options, because a custom answer should not cost the user their
checks. Keyboard support sits on top of real buttons, not in place of
them.

The header chevron minimizes the card instead of dismissing it, so the
conversation stays readable while the user decides. Only Skip cancels.

Unrecognized custom UIs keep the terminal panel, and a recognized
question can still be answered there through the terminal-view button.
A question the user answered is a decision, not tool output. Render a
finished ask_user call as a decision card built from the tool result
details: the question, the options offered, which ones were chosen, any
comment, and whether the question was dismissed. A question still waiting
on an answer renders the same card from the tool arguments.
A turn's tool calls are folded into the process-details group, which
collapses once the turn has a final answer. That hid every decision card
behind a disclosure, so an answered question looked lost.

Pull question tool calls out of that group and render them in the turn.
A blocked custom UI notified with generic text. A recognized question
carries its own words, so show them and let the user decide from the
notification whether to come back now.
Write down why Pi Web holds knowledge of one extension's shapes, how a
custom UI is correlated with a tool call, and what keeps that coupling
contained.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant