Add an inline SVG presentation tool with durable chat replay #80

New Issue

Closed

opened 2026-08-30 19:13:56 +00:00 by hugo · 1 comment

hugo commented 2026-08-30 19:13:56 +00:00

Owner

Copy Link

Goal

Let the local model deliberately present a static SVG inline at the point of a chat response. This is a chat content type, not an A2UI surface: it has no controls, client events, separate tab, surface lifecycle, or A2UI state.

The SVG must remain visible after switching sessions and after restarting DS4Server.

Research

• Claude supports SVG as an Artifact, with versions and conversation association, but renders it in a dedicated pane. That persistence model is useful; the separate-surface UI is explicitly not what this feature needs: https://support.claude.com/en/articles/9487310-what-are-artifacts-and-how-do-i-use-them
• VS Code chat models rich output as typed response parts (Markdown, images, references, progress, buttons, and trees) and exposes response turns in chat history. Its API notes that tool results which must survive later turns need explicit persisted metadata: https://code.visualstudio.com/api/extension-guides/ai/chat and https://code.visualstudio.com/api/references/vscode-api
• MCP likewise treats tools as model-controlled and tool results as typed content rather than asking the model to smuggle every visual through prose. It supports image and embedded-resource result blocks, while requiring tool input/output validation: https://modelcontextprotocol.io/specification/2025-11-25/server/tools
• Codex's open image-generation implementation follows the same separation: a model tool emits an image-specific turn item for the UI and saves the generated artifact, rather than treating the payload as ordinary assistant Markdown: https://github.com/openai/codex/blob/main/codex-rs/ext/image-generation/src/tool.rs
• DS4Server already enables Iced's svg feature. iced::widget::svg::Handle::from_memory renders SVG bytes directly, so no browser/WebView or A2UI renderer is needed: https://docs.rs/iced/0.14.0/iced/widget/svg/struct.Handle.html
• Iced's renderer already uses usvg/resvg. resvg intentionally renders only static SVG and does not implement scripts, events, links, or animation: https://github.com/linebender/resvg/blob/main/README.md

Current DS4Server path

The implementation should reuse the existing durable tool path:

1. src/agent.rs declares tools in TOOLS, parses model tool calls, executes them, and reconstructs ToolCards with stored_tool_cards.
2. The raw assistant response, including its tool call and arguments, is continuously saved to messages.content by src/app/generation.rs / src/database.rs.
3. Tool results are saved as the following tool = true message.
4. Session selection reloads all messages, converts them back to ChatMessage, and src/app/view/chat.rs rebuilds stored tool cards from the assistant message and following tool result.

Therefore the SVG source should have one durable source of truth: the present_svg tool argument already stored in the assistant message. Do not add an SVG table, duplicate the SVG into the tool result, or write an artifact file for this bounded text payload.

Proposed implementation

1. Add a model tool

Add a non-mutating present_svg tool to src/agent.rs:

{
  "svg": "<svg ...>...</svg>",
  "alt": "Concise description of the graphic"
}

• Both fields are required, non-empty strings.
• The description must tell the model to use it for a static inline diagram/illustration when SVG materially improves the answer.
• It must say not to use this tool for interactive controls or stateful UI; those remain A2UI.
• It must say the SVG is self-contained and alt describes the information conveyed by the graphic.
• Execution is local, read-only, requires no approval, touches no paths, and returns only a small result such as SVG presented inline: <alt> so the model can continue its response.

Add ToolHandler::PresentSvg and route it through the existing schema validation and execute_validated switch. Do not create a second tool execution mechanism.

2. Validate at the tool boundary

Treat model-produced SVG as untrusted input even though it is rendered by resvg rather than a browser.

• Define one named source-size limit (suggested: 64 KiB) and reject empty or oversized SVG with ordinary actionable Tool error: feedback.
• Parse it with the same usvg generation already present through Iced; declaring the already-resolved usvg 0.45 dependency directly is preferable to adding a second XML/SVG stack.
• Require a valid, non-empty SVG tree and sane intrinsic dimensions.
• Bound render complexity (node count and dimensions) with named constants and tests.
• Reject external resources and embedded raster images. They are unnecessary for model-authored diagrams and create file/network/decompression ambiguity. Static vector shapes, paths, gradients, masks, filters, and text are sufficient.
• Never evaluate script, animation, event handlers, links, foreignObject, HTML, or remote URLs. resvg already omits active browser behavior; keep validation explicit so a later renderer change cannot silently broaden the trust boundary.

Validation must happen before the tool reports success. Invalid SVG stays a normal failed tool card and is not rendered.

3. Render the successful call inline

In src/app/view/chat.rs, render successful present_svg cards as a quiet inline figure in call order instead of the generic tool-card row:

• Build an iced::widget::svg::Handle::from_memory from the stored svg argument.
• Fit it to the available chat width, preserve aspect ratio, prevent horizontal overflow, and cap its displayed height (suggested: 480 logical pixels).
• Use the shared raised-surface/border style already used by chat/tool content; do not introduce a separate tab or screen-specific visual language.
• Show alt as a muted visible caption so the information has a text equivalent and remains understandable if rendering fails.
• During a live tool run, show the existing lifecycle state until validation completes; replace it with the figure only after Completed.
• Preserve mixed-call order when present_svg is emitted in the same batch as ordinary tools.

Keep extraction of svg and alt from a ToolCard in one small pure helper so live and rehydrated rendering use the same rules.

4. Rehydrate without a migration

No database migration should be needed. The complete tool call is already persisted in messages.content; the following tool result records whether validation completed or failed. On reload, stored_tool_cards must reconstruct the call and the chat view must render the same successful SVG from its stored arguments.

This must also work for conversations that later compact: compaction may change model context, but it must not remove the persisted visible chat record.

Do not store a second base64 or SVG copy. This avoids the history bloat and duplication problems seen when agents persist large image payloads in multiple event/message records.

5. Preserve exports

Update Markdown chat export so a successful present_svg call is represented by its alt text and a fenced svg source block, rather than a generic tool-result dump. Failed calls remain ordinary tool failures. This keeps the durable chat portable without relying on unsafe raw HTML rendering in Markdown readers.

Likely files

• Cargo.toml / Cargo.lock only if usvg must be named directly for validation; reuse the version already selected by Iced.
• src/agent.rs for the tool schema, handler, limits, validation, and stored-card extraction.
• src/app/view/chat.rs for inline rendering.
• src/app.rs for Markdown export behavior.
• src/database.rs only for a reopen regression test; no schema or migration change is expected.

Acceptance criteria

• The model can call present_svg(svg, alt) and a valid static SVG appears inline in Chat, not in the A2UI tab.
• The tool is described as static presentation only and does not replace A2UI for interactive/stateful UI.
• The inline figure preserves aspect ratio, stays within chat width and the height cap, uses existing DS4Server surface styling, and includes a visible text alternative.
• Invalid, empty, oversized, over-complex, externally referenced, or raster-embedding SVG is rejected with actionable tool feedback and is not rendered.
• Script, events, links, animation, foreignObject, HTML, filesystem reads, and network access cannot execute through presented SVG.
• A successful SVG remains identical after switching to another session and back, after application restart/reopen, and after later context compaction.
• SVG source is stored once through the existing assistant tool-call message; there is no new SVG table, duplicate payload in the tool-result message, or artifact file.
• Mixed ordinary tools and present_svg calls preserve their original order and lifecycle states.
• Markdown export includes the alt text and SVG source without exposing DSML/GLM transport markup.
• Existing A2UI parsing, persistence, rendering, and dismissal behavior is unchanged.

Minimum verification

• Unit-test both supported DSML/GLM tool syntaxes with SVG markup inside the string argument and verify exact source recovery.
• Unit-test valid SVG plus each rejection class: empty, size limit, malformed/non-SVG, excessive complexity/dimensions, external reference, embedded raster, and active/foreign content.
• Test that successful and failed present_svg calls map to inline figure vs. ordinary failed card, including a mixed tool batch.
• Add a database reopen test that saves an assistant present_svg call and its tool result, reloads the session, rebuilds stored cards, and recovers byte-identical SVG plus alt text.
• Extend the Markdown export regression for a successful and a failed SVG call.
• Manually ask the local model for a small diagram, verify inline rendering, switch sessions, restart the app, and verify rehydration.
• Run the repository commit gates from AGENTS.md before committing the implementation.

Non-goals

• A2UI surfaces or changes to the A2UI protocol.
• Interactive SVG, JavaScript, animation, hyperlinks, HTML, or user-input handling.
• General image generation, raster images, arbitrary attachments, SVG editing/version history, or a new artifact browser.
• Remote SVG URLs or filesystem-backed SVG references.

## Goal Let the local model deliberately present a static SVG inline at the point of a chat response. This is a chat content type, not an A2UI surface: it has no controls, client events, separate tab, surface lifecycle, or A2UI state. The SVG must remain visible after switching sessions and after restarting DS4Server. ## Research - Claude supports SVG as an Artifact, with versions and conversation association, but renders it in a dedicated pane. That persistence model is useful; the separate-surface UI is explicitly not what this feature needs: https://support.claude.com/en/articles/9487310-what-are-artifacts-and-how-do-i-use-them - VS Code chat models rich output as typed response parts (Markdown, images, references, progress, buttons, and trees) and exposes response turns in chat history. Its API notes that tool results which must survive later turns need explicit persisted metadata: https://code.visualstudio.com/api/extension-guides/ai/chat and https://code.visualstudio.com/api/references/vscode-api - MCP likewise treats tools as model-controlled and tool results as typed content rather than asking the model to smuggle every visual through prose. It supports image and embedded-resource result blocks, while requiring tool input/output validation: https://modelcontextprotocol.io/specification/2025-11-25/server/tools - Codex's open image-generation implementation follows the same separation: a model tool emits an image-specific turn item for the UI and saves the generated artifact, rather than treating the payload as ordinary assistant Markdown: https://github.com/openai/codex/blob/main/codex-rs/ext/image-generation/src/tool.rs - DS4Server already enables Iced's `svg` feature. `iced::widget::svg::Handle::from_memory` renders SVG bytes directly, so no browser/WebView or A2UI renderer is needed: https://docs.rs/iced/0.14.0/iced/widget/svg/struct.Handle.html - Iced's renderer already uses `usvg`/`resvg`. `resvg` intentionally renders only static SVG and does not implement scripts, events, links, or animation: https://github.com/linebender/resvg/blob/main/README.md ## Current DS4Server path The implementation should reuse the existing durable tool path: 1. `src/agent.rs` declares tools in `TOOLS`, parses model tool calls, executes them, and reconstructs `ToolCard`s with `stored_tool_cards`. 2. The raw assistant response, including its tool call and arguments, is continuously saved to `messages.content` by `src/app/generation.rs` / `src/database.rs`. 3. Tool results are saved as the following `tool = true` message. 4. Session selection reloads all messages, converts them back to `ChatMessage`, and `src/app/view/chat.rs` rebuilds stored tool cards from the assistant message and following tool result. Therefore the SVG source should have one durable source of truth: the `present_svg` tool argument already stored in the assistant message. Do not add an SVG table, duplicate the SVG into the tool result, or write an artifact file for this bounded text payload. ## Proposed implementation ### 1. Add a model tool Add a non-mutating `present_svg` tool to `src/agent.rs`: ```json { "svg": "<svg ...>...</svg>", "alt": "Concise description of the graphic" } ``` - Both fields are required, non-empty strings. - The description must tell the model to use it for a static inline diagram/illustration when SVG materially improves the answer. - It must say not to use this tool for interactive controls or stateful UI; those remain A2UI. - It must say the SVG is self-contained and `alt` describes the information conveyed by the graphic. - Execution is local, read-only, requires no approval, touches no paths, and returns only a small result such as `SVG presented inline: <alt>` so the model can continue its response. Add `ToolHandler::PresentSvg` and route it through the existing schema validation and `execute_validated` switch. Do not create a second tool execution mechanism. ### 2. Validate at the tool boundary Treat model-produced SVG as untrusted input even though it is rendered by `resvg` rather than a browser. - Define one named source-size limit (suggested: 64 KiB) and reject empty or oversized SVG with ordinary actionable `Tool error:` feedback. - Parse it with the same `usvg` generation already present through Iced; declaring the already-resolved `usvg 0.45` dependency directly is preferable to adding a second XML/SVG stack. - Require a valid, non-empty SVG tree and sane intrinsic dimensions. - Bound render complexity (node count and dimensions) with named constants and tests. - Reject external resources and embedded raster images. They are unnecessary for model-authored diagrams and create file/network/decompression ambiguity. Static vector shapes, paths, gradients, masks, filters, and text are sufficient. - Never evaluate script, animation, event handlers, links, `foreignObject`, HTML, or remote URLs. `resvg` already omits active browser behavior; keep validation explicit so a later renderer change cannot silently broaden the trust boundary. Validation must happen before the tool reports success. Invalid SVG stays a normal failed tool card and is not rendered. ### 3. Render the successful call inline In `src/app/view/chat.rs`, render successful `present_svg` cards as a quiet inline figure in call order instead of the generic tool-card row: - Build an `iced::widget::svg::Handle::from_memory` from the stored `svg` argument. - Fit it to the available chat width, preserve aspect ratio, prevent horizontal overflow, and cap its displayed height (suggested: 480 logical pixels). - Use the shared raised-surface/border style already used by chat/tool content; do not introduce a separate tab or screen-specific visual language. - Show `alt` as a muted visible caption so the information has a text equivalent and remains understandable if rendering fails. - During a live tool run, show the existing lifecycle state until validation completes; replace it with the figure only after `Completed`. - Preserve mixed-call order when `present_svg` is emitted in the same batch as ordinary tools. Keep extraction of `svg` and `alt` from a `ToolCard` in one small pure helper so live and rehydrated rendering use the same rules. ### 4. Rehydrate without a migration No database migration should be needed. The complete tool call is already persisted in `messages.content`; the following tool result records whether validation completed or failed. On reload, `stored_tool_cards` must reconstruct the call and the chat view must render the same successful SVG from its stored arguments. This must also work for conversations that later compact: compaction may change model context, but it must not remove the persisted visible chat record. Do not store a second base64 or SVG copy. This avoids the history bloat and duplication problems seen when agents persist large image payloads in multiple event/message records. ### 5. Preserve exports Update Markdown chat export so a successful `present_svg` call is represented by its alt text and a fenced `svg` source block, rather than a generic tool-result dump. Failed calls remain ordinary tool failures. This keeps the durable chat portable without relying on unsafe raw HTML rendering in Markdown readers. ## Likely files - `Cargo.toml` / `Cargo.lock` only if `usvg` must be named directly for validation; reuse the version already selected by Iced. - `src/agent.rs` for the tool schema, handler, limits, validation, and stored-card extraction. - `src/app/view/chat.rs` for inline rendering. - `src/app.rs` for Markdown export behavior. - `src/database.rs` only for a reopen regression test; no schema or migration change is expected. ## Acceptance criteria - [ ] The model can call `present_svg(svg, alt)` and a valid static SVG appears inline in Chat, not in the A2UI tab. - [ ] The tool is described as static presentation only and does not replace A2UI for interactive/stateful UI. - [ ] The inline figure preserves aspect ratio, stays within chat width and the height cap, uses existing DS4Server surface styling, and includes a visible text alternative. - [ ] Invalid, empty, oversized, over-complex, externally referenced, or raster-embedding SVG is rejected with actionable tool feedback and is not rendered. - [ ] Script, events, links, animation, `foreignObject`, HTML, filesystem reads, and network access cannot execute through presented SVG. - [ ] A successful SVG remains identical after switching to another session and back, after application restart/reopen, and after later context compaction. - [ ] SVG source is stored once through the existing assistant tool-call message; there is no new SVG table, duplicate payload in the tool-result message, or artifact file. - [ ] Mixed ordinary tools and `present_svg` calls preserve their original order and lifecycle states. - [ ] Markdown export includes the alt text and SVG source without exposing DSML/GLM transport markup. - [ ] Existing A2UI parsing, persistence, rendering, and dismissal behavior is unchanged. ## Minimum verification - Unit-test both supported DSML/GLM tool syntaxes with SVG markup inside the string argument and verify exact source recovery. - Unit-test valid SVG plus each rejection class: empty, size limit, malformed/non-SVG, excessive complexity/dimensions, external reference, embedded raster, and active/foreign content. - Test that successful and failed `present_svg` calls map to inline figure vs. ordinary failed card, including a mixed tool batch. - Add a database reopen test that saves an assistant `present_svg` call and its tool result, reloads the session, rebuilds stored cards, and recovers byte-identical SVG plus alt text. - Extend the Markdown export regression for a successful and a failed SVG call. - Manually ask the local model for a small diagram, verify inline rendering, switch sessions, restart the app, and verify rehydration. - Run the repository commit gates from `AGENTS.md` before committing the implementation. ## Non-goals - A2UI surfaces or changes to the A2UI protocol. - Interactive SVG, JavaScript, animation, hyperlinks, HTML, or user-input handling. - General image generation, raster images, arbitrary attachments, SVG editing/version history, or a new artifact browser. - Remote SVG URLs or filesystem-backed SVG references.

hugo added the enhancement label 2026-08-30 19:13:56 +00:00

hugo commented 2026-08-31 06:29:54 +00:00

Author

Owner

Copy Link

Implemented and pushed as 8adac26.

• Added the non-mutating present_svg(svg, alt) tool through the existing tool schema/execution path, with a 64 KiB source limit, bounded dimensions/node complexity, direct usvg validation, and explicit rejection of external/raster/active/foreign content.
• Rendered only completed calls inline in original mixed-call order with aspect-fit sizing, a 480 px height cap, shared chat surface styling, and a visible alt caption. Running and failed calls keep the normal tool lifecycle UI.
• Reused the assistant tool-call message as the sole durable SVG source. Added rehydration coverage for session switching, compaction, and database reopen without a migration, artifact file, or duplicated result payload.
• Exported successful figures as alt text plus fenced SVG while failed calls remain ordinary tool failures.

Verification: both DSML and GLM transports recover the SVG byte-for-byte; validation tests cover empty, oversized, malformed/non-SVG, excessive dimensions/complexity, external references, raster content, and active/foreign content; mixed lifecycle, export, schema, and database reopen regressions pass. Manual local-model verification rendered a two-step diagram inline, preserved it across a session switch, and replayed it after restarting/reopening DS4Server. This is a new DS4Server presentation feature; DS4 has no corresponding SVG functionality, and the existing DS4 transport/A2UI regression suite remains green.

Commit gates passed: cargo fmt --all -- --check; cargo clippy --all-targets --all-features -- -D warnings; make bundle; cargo test --all-features (214 passed, 16 ignored).

Implemented and pushed as 8adac26. - Added the non-mutating present_svg(svg, alt) tool through the existing tool schema/execution path, with a 64 KiB source limit, bounded dimensions/node complexity, direct usvg validation, and explicit rejection of external/raster/active/foreign content. - Rendered only completed calls inline in original mixed-call order with aspect-fit sizing, a 480 px height cap, shared chat surface styling, and a visible alt caption. Running and failed calls keep the normal tool lifecycle UI. - Reused the assistant tool-call message as the sole durable SVG source. Added rehydration coverage for session switching, compaction, and database reopen without a migration, artifact file, or duplicated result payload. - Exported successful figures as alt text plus fenced SVG while failed calls remain ordinary tool failures. Verification: both DSML and GLM transports recover the SVG byte-for-byte; validation tests cover empty, oversized, malformed/non-SVG, excessive dimensions/complexity, external references, raster content, and active/foreign content; mixed lifecycle, export, schema, and database reopen regressions pass. Manual local-model verification rendered a two-step diagram inline, preserved it across a session switch, and replayed it after restarting/reopening DS4Server. This is a new DS4Server presentation feature; DS4 has no corresponding SVG functionality, and the existing DS4 transport/A2UI regression suite remains green. Commit gates passed: cargo fmt --all -- --check; cargo clippy --all-targets --all-features -- -D warnings; make bundle; cargo test --all-features (214 passed, 16 ignored).

hugo closed this issue 2026-08-31 06:30:00 +00:00

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

No results found.

Labels

Clear labels

cannot reproduce

bug

Something is not working

critical

issues that are more important than others of the same group

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

idea

just an idea

invalid

Something is wrong

question

More information is needed

wontfix

This won't be fixed

No Label enhancement

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

hugo

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/DS4Server#80