Skip to content

feat: propose chat-native UI artifacts via execute capability - #3054

Draft
reachjalil wants to merge 13 commits into
different-ai:devfrom
reachjalil:feature/ui-artifacts
Draft

feat: propose chat-native UI artifacts via execute capability#3054
reachjalil wants to merge 13 commits into
different-ai:devfrom
reachjalil:feature/ui-artifacts

Conversation

@reachjalil

@reachjalil reachjalil commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Proposal

Introduce alpha UI Artifacts: deterministic, chat-native answer cards that any compatible agent engine can discover and render through OpenWork Cloud's existing two-tool surface.

The public agent MCP remains exactly:

  • search_capabilities
  • execute_capability

Artifact discovery and use are virtual capabilities executed through execute_capability; this proposal does not add an OpenCode-only tool or expand the public MCP surface.

What this adds

  • Seven member-controlled standard artifact families:
    • workspace brief
    • calendar
    • widgets
    • conversation thread
    • priority inbox
    • attention queue
    • approval queue
  • One composable widgets.collection artifact that accepts metric, progress, status, balance, and date widgets in grid, strip, or stack layouts.
  • One calendar.view artifact with day, agenda, and week variants over the same event contract.
  • An alpha right-rail tile catalog with per-artifact enable/disable preferences persisted per Den membership.
  • Shared Zod contracts for strict search, use, action, narration, replacement, and suggestion receipts.
  • A startable stateful mock MCP exposing only search_artifacts and use_artifact.
  • Native desktop rendering for validated execute_capability receipts.
  • Revision-safe mock approval/rejection:
    • a button stages a visible user request instead of executing silently;
    • only the instance, item, decision, and expected revision enter the prompt;
    • a successful decision replaces the same artifact instance;
    • stale decisions fail with revision_conflict.

Hard opt-in boundary

UI Artifacts is a true member-level feature flag and defaults to false in both the desktop preference cache and the Den database.

When it is off:

  • agent MCP initialization contains no artifact-specific steering;
  • search_capabilities returns no artifact virtual capabilities;
  • direct artifact execution fails closed with artifact_disabled;
  • ordinary successful capability results receive no artifact suggestion content;
  • the right-rail entry and artifact catalog are absent;
  • artifact-shaped tool output stays in the ordinary tool renderer;
  • revision reconciliation and the native artifact renderer are bypassed.

The prompt design is also turn-scoped. Artifact instructions are not stored in MCP initialization even after opt-in. A bounded agentInstruction is carried only inside a successful enabled suggestion, expires at the end of that turn, allows at most one render, carries no provider result values, and forbids inferred approval decisions.

Disabling the flag therefore takes effect without depending on an OpenCode-specific restart or MCP reconnection.

Context and safety policy

  • Suggestions are attached only to successful execute_capability results.
  • At most one artifact may be suggested or rendered per turn.
  • Suggestions expire at the end of the current turn and carry argument shape only, never provider values.
  • The agent searches the catalog once and uses the returned schema digest and mock example.
  • Mock fixtures remain visibly labeled and may not substitute for provider data.
  • The agent narrates a compact summary and decision-relevant facts instead of repeating the card.
  • Approval or rejection is never inferred and never calls a live provider approval tool.
  • Stateful mock instances are isolated by membership plus a one-way hash of the MCP bearer token; raw tokens are not persisted or logged.

Full lifecycle proof

The coded Fraimz drives the local Den API and real Electron renderer with deterministic mock data:

  1. prove the disabled MCP and desktop experience is inert;
  2. opt in and inspect the seven artifact tiles;
  3. render a heterogeneous widget collection;
  4. render the calendar day variant;
  5. render the calendar agenda variant;
  6. render the calendar week variant;
  7. replace the screenshot-style dashboard with a workspace brief inside chat;
  8. render an untouched approval at revision 1;
  9. click Approve and inspect the minimal staged prompt;
  10. execute the decision, replace the card at revision 2, and reject a stale revision-1 replay.

Public Fraimz with ten validated screenshots:
https://openwork-ui-artifacts-fraimz.vercel.app

Run: 2026-07-23T13-42-20-726Z on commit 0d3286aa714aaf49711b30f4ee2636bba434958e.

Verification

  • pnpm --filter @openwork-ee/den-api exec bun test test/ui-artifacts-capability.test.ts test/mcp-agent-timeouts.test.ts
    • 18/18 passed
    • covers disabled discovery/execution/enrichment, prompt isolation, provider compatibility, and timeout behavior
  • pnpm --filter @openwork/ui-artifact-mcp check
    • typecheck passed
    • 12/12 tests passed, including a spawned stdio MCP and retained approval state
    • build passed
  • pnpm --filter @openwork/app exec bun test --isolate tests/ui-artifacts.test.ts
    • 5/5 passed
  • pnpm --filter @openwork/app typecheck
    • passed
  • pnpm fraimz --flow ui-artifacts-execute-capability --cdp-url http://127.0.0.1:28903
    • 1 flow passed, 0 failed, 0 skipped
    • ten narrated frames and voice-over coverage passed at the exact PR head
  • git diff --check
    • passed

The lifecycle was driven and visually inspected through Electron CDP. It has not yet been manually confirmed by the requester.

Risks and remaining work

  • All bundled artifact examples and approval state are intentionally mock data. Live provider normalization, provenance, and organization-level artifact policy are follow-up work.
  • The bounded in-memory approval store is demo infrastructure, not durable production approval state.
  • Full repository tests, packaged installers, other operating systems, broad accessibility review, and complete localization coverage were not run.

Rollback is immediate: disable the alpha UI Artifacts preference or revert these commits. With the preference off, existing OpenWork capability and chat behavior remains active and no live provider mutation is performed by the mock lifecycle.

@vercel

vercel Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-landing Ready Ready Preview, v0 Jul 30, 2026 2:58pm

@vercel

vercel Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@reachjalil is attempting to deploy a commit to the Different AI Team on Vercel.

A member of the Team first needs to authorize it.

@reachjalil

reachjalil commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

fraimz — ✅ PASSED

Five-frame mock lifecycle on 02a510024 · run 2026-07-23T12-49-16-052Z

Open the complete interactive fraimz

  1. The right rail offers eight member-controlled chat artifact tiles

    The member can inspect the standard patterns and decide which artifacts the agent may suggest and render.

    Eight configurable UI artifact tiles in the right rail
  2. A workspace brief replaces the screenshot-style home dashboard inside chat

    The agent kept the cloud surface to search_capabilities and execute_capability, searched for one matching artifact, and used the returned schema.

    Workspace brief rendered as a chat-native dashboard
  3. The approval artifact starts at revision one without making a decision

    Both choices are visible, the data is explicitly labeled as mock, and no action has occurred.

    Mock approval artifact awaiting a decision at revision one
  4. Approval stays user-controlled through a minimal revision-bound prompt

    Clicking Approve stages a visible request with only the artifact instance, selected item, decision, and expected revision. It explicitly forbids a live provider action.

    Minimal explicit approval prompt staged in the composer
  5. The execute response replaces the card with revision two

    The selected item becomes approved, the submitted prompt is cleared, one reconciled artifact remains, and a stale revision-one replay has been rejected by the mock MCP.

    Updated approval artifact at revision two

@reachjalil

Copy link
Copy Markdown
Collaborator Author

Updated the proposal at 0d3286aa714aaf49711b30f4ee2636bba434958e.

The latest revision makes the alpha a hard opt-in boundary:

  • defaults off in desktop and Den;
  • no artifact text in MCP initialization, even after opt-in;
  • enabled suggestion guidance is turn-scoped inside the suggestion receipt;
  • disabled search exposes no artifact virtual capabilities;
  • disabled direct calls return artifact_disabled;
  • disabled ordinary calls receive no suggestion enrichment;
  • the rail, reconciliation, and native renderer remain inactive while off.

It also consolidates Calendar into one day/agenda/week artifact and Widgets into one heterogeneous composable collection.

Updated ten-frame Fraimz: https://openwork-ui-artifacts-fraimz.vercel.app

Disabled boundary Composable widgets Calendar week
UI Artifacts disabled Composable widget collection Calendar week variant

Focused verification on this exact head:

  • Den artifact/agent MCP tests: 18/18
  • mock MCP package: typecheck, 12/12 tests, build
  • app artifact tests: 5/5
  • app typecheck: passed
  • Fraimz: 1 passed, 0 failed, 10 narrated frames
  • git diff --check: passed

@reachjalil

Copy link
Copy Markdown
Collaborator Author

CI follow-up for 0d3286aa714aaf49711b30f4ee2636bba434958e: all GitHub Actions checks passed, including Linux/macOS tests, contracts/security, schema+migrations, i18n, Den API/Web image builds, inference, and Helm validation.

The four remaining red Vercel statuses are external Authorization required to deploy gates for the Different AI team; they did not execute a build. The separately deployed public Fraimz is ready at https://openwork-ui-artifacts-fraimz.vercel.app.

# Conflicts:
#	apps/app/src/components/chat/message-list.tsx
#	apps/app/src/react-app/domains/session/chat/session-page.tsx
#	apps/app/src/react-app/domains/session/surface/session-surface.tsx
#	ee/apps/den-api/src/mcp/agent.ts
#	ee/packages/den-db/drizzle/meta/0048_snapshot.json
#	ee/packages/den-db/drizzle/meta/_journal.json
#	packaging/docker/Dockerfile.den
# Conflicts:
#	apps/app/src/components/chat/message-list.tsx
#	apps/app/src/react-app/domains/session/chat/session-page.tsx
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant