Skip to content

feat(agent): add GitHub Copilot as an in-app agent backend - #139

Merged
0xsline merged 21 commits into
0xsline:mainfrom
SSKUltra:feat/copilot-backend
Sep 8, 2026
Merged

feat(agent): add GitHub Copilot as an in-app agent backend#139
0xsline merged 21 commits into
0xsline:mainfrom
SSKUltra:feat/copilot-backend

Conversation

@SSKUltra

@SSKUltra SSKUltra commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

What this adds

A third agent backend — GitHub Copilot — alongside api and codex, driven in-app through the official @github/copilot-sdk.

image image

Today Copilot users can only reach OpenChatCut from the outside, over the external MCP endpoint. That path is deliberately narrow: only draft-safe read/edit tools are exposed, tool calls round-trip through the broker into the browser, and the conversation lives in the user's terminal rather than in the editor's chat. This gives Copilot the same first-class, in-app treatment Codex already has.

Why the SDK rather than MCP or --acp

@github/copilot-sdk is a close analogue of the codex app-server integration, so server/copilot/ mirrors server/codex/ deliberately:

Codex Copilot
Transport codex app-server --listen stdio:// SDK over JSON-RPC (spawns the CLI)
Host tools pushed per turn defineTool() with in-process handlers
Isolated home CODEX_HOME baseDirectoryCOPILOT_HOME
Approval host responds to tool RPCs onPermissionRequest
Streaming text/thinking/tool deltas same, mapped onto the existing event union

copilot --acp was considered and rejected: ACP has no per-turn tool injection, so it would have required running a separate MCP server for the editor's own tools — reintroducing exactly the indirection this removes.

Design decisions worth reviewing

Tools are the entire surface. availableTools is restricted to custom tools, so the agent has no shell, filesystem, web or MCP access, and any permission request is rejected as a filter escape rather than prompted. Verified by asking it to run ls, read ~/.zshrc and fetch a URL: zero tool calls, and it enumerated only the OpenChatCut tools it had.

Capabilities come from the runtime, not the bundled catalog. Copilot reports exact per-model limits, so resolveCopilotModelCapabilities() uses those and records them as exact rather than estimated. Models are attributed to their upstream provider (claude-* → anthropic, grok-* → xai) so capability overrides and vision-model selection keep working. Models without tool support are hidden, since every editing flow needs tool calls.

Tool failures return a typed ToolResultObject. Throwing collapses to a generic "Tool execution failed" and strips the diagnostic. With the typed failure the model receives the real reason verbatim — confirmed by returning Unknown clip: clip-zzz and seeing it echoed back.

__images are handed over as binary attachments, matching the Codex backend's inputImage handling, so view_asset_frames contact sheets stay viewable instead of being stringified into the prompt. Verified with a synthetic sheet (red left half, blue right half) that the model described correctly.

The turn uses an idle timeout, not a wall-clock cap. A long multi-shot edit streams continuously for many minutes and must not be killed for elapsed time. It is suspended while a host tool is in flight, bounded by the broker's own maximum so a never-settled call cannot hang the turn.

Auth stays with the CLI. Users run copilot login; OpenChatCut never sees the credential. The settings pane is read-only status plus model/effort selection, and session state is isolated under ~/.openchatcut/copilot so in-app runs never disturb the user's own ~/.copilot.

Second commit: durable agent edits

While building this I hit a failure worth fixing independently — it is backend-agnostic and affects api and codex equally.

A run applied a long sequence of edits, verified them by rendering frames, and wrote a confident summary while none of the work was durable. The server could not tell: the editor answered "success" and the tool result carried no durability information, unlike the external MCP path which does track baseRevision.

The editor now reports { revision, pending, failed } with each tool result, derived from state SaveCoordinator already keeps, and executeBrowserTool fails mutating tools when the editor's writes are settled-failed. The distinctions matter as much as the check:

  • reads never fail on a save error — the agent still has to read state to report the problem
  • a pending save is not a failure — autosave debounces 500ms and normally settles after the tool returns, so treating pending as loss would fail nearly every edit
  • a missing signal is unknown, not failed — an older editor keeps working

Project writes are also flushed at terminal settle, including failed and cancelled runs, which is exactly when the last edits used to be stranded inside the debounce window.

Happy to split this into its own PR if you'd prefer — it shares store-types.ts and routes.ts with the first commit, so it would need to land second.

Testing

server/copilot/copilot-agent.verify.ts and server/agent-runs/editor-persistence.verify.ts, both registered in verify:server-runs and git-tracked (the verify-gate-coverage and verify-registration invariants both catch omissions here — they caught mine).

Coverage: turn parsing (saved model/effort fallback, explicit-null suppression, tool-name validation, duplicate rejection), provider attribution incl. unknown-vendor fallback, capabilities from runtime vs. estimate vs. user override, the version gate, and the full mutating/read × durable/pending/failed/missing decision table.

Local gates on this branch, rebased onto current main:

npm run lint            ✅
npm test                ✅  171 segments
npm run verify:agent-skill  ✅
tsc -b + vite build     ✅

Acceptance evidence

Live against a real Copilot subscription:

GET /api/copilot/status  → installed, supported, authenticated
GET /api/copilot/models  → 19 models, e.g. claude-sonnet-5
                           ctx=1000000 in=936000 out=64000
                           efforts=[low, medium, high, xhigh, max]
POST /api/copilot/turn   → tool-start read_timeline {}
                           tool-start set_item_timing {"itemId":"clip-b","durationInFrames":540}
                           events: thinking-delta, context-usage, tool-start/end, text-delta, done

Driven against the real generated tool catalog, the model produced schema-valid arguments unaided (computing 600 − 60 = 540 frames at 30fps).

Notes for the maintainer

  • New dependency: @github/copilot-sdk. The CLI itself is discovered rather than bundled (OPENCHATCUT_COPILOT_PATH overrides), so it is not a hard install requirement.
  • Two new non-secret keystore names: COPILOT_MODEL, COPILOT_REASONING_EFFORT, registered in MODEL_ROUTING_NAMES in keystore.verify.ts.
  • English and Italian dictionary entries added; verify:i18n passes. Russian was not required by the coverage checks.

celeste1900 and others added 6 commits September 2, 2026 11:52
Add OFox (https://ofox.ai), a multi-model gateway, as:

- an OpenAI-compatible LLM preset (mirrors the openrouter entry; keystore
  whitelist, vendor icon monogram, vision label, verify assertions)
- a video generation provider (async task id + polling, same shape as
  grok-imagine-video, implemented in its own server/plugins module):
  text-to-video, first-frame / first-and-last-frame image-to-video, and
  up to 9 image references (frames and references are mutually exclusive
  at the API level, enforced before submission); generateAudio and seed
  pass through. Local project media rides as base64 data URLs, which the
  OFox gateway re-hosts on its own object storage (verified live).
  Validation, resumer registration, capabilities, tool schema, settings
  page with model discovery via the /models probe, agent skill reference,
  i18n (en/it/ru), .env.example, and verify coverage included.

refVideos/refAudios are supported by the OFox API but not wired yet;
per-model duration/resolution limits are enforced by the API with a clear
400 before any task is created, so no local per-model whitelist is kept.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mirror_urls are persistent signed CDN addresses returned when the
upstream has mirroring enabled; unsigned_urls are temporary upstream
links that may expire within 24 hours. Prefer the former and fall back
to the latter, matching the official retrieve-endpoint recommendation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drive Copilot in-app through @github/copilot-sdk, alongside the existing
api and codex backends, rather than only exposing OpenChatCut over MCP.

The SDK is a direct analogue of the codex app-server integration, so this
mirrors server/codex/ closely:

- server/copilot/installation.ts discovers the CLI (bundled platform
  package, PATH, common install roots, OPENCHATCUT_COPILOT_PATH)
- server/copilot/client.ts owns one runtime with an isolated
  COPILOT_HOME (~/.openchatcut/copilot) so in-app runs never disturb the
  user's own ~/.copilot, and caches the model catalog
- server/copilot/turn-manager.ts registers OpenChatCut's tool catalog as
  SDK tools with in-process handlers and maps session events onto the
  same stream-event union the codex backend emits

Host tools are the whole tool surface: availableTools is restricted to
custom tools, so the agent has no shell, filesystem, web or MCP access,
and any permission request is rejected as a filter escape.

Copilot reports exact per-model limits, so capabilities come from
resolveCopilotModelCapabilities rather than the bundled models.dev
catalog, and models are attributed to their upstream provider
(claude-* -> anthropic, grok-* -> xai) so capability overrides and vision
selection keep working. Models without tool support are hidden.

Two details worth recording:

- tool failures return a typed ToolResultObject; throwing collapses to a
  generic "Tool execution failed" and strips the diagnostic the agent
  needs to recover
- __images results are handed over as binary attachments, matching the
  codex backend's inputImage handling, so frame contact sheets remain
  viewable instead of being stringified into the prompt

The turn uses an idle timeout rather than a wall-clock cap: a long edit
streams continuously for many minutes and must not be killed for total
elapsed time. It is suspended while a host tool is in flight, bounded by
the broker's own maximum so a never-settled call cannot hang the turn.

Auth stays with the CLI (copilot login); OpenChatCut never sees the
credential and the settings pane is read-only status plus model choice.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
An agent run could apply a long sequence of edits, verify them by
rendering frames, and write a confident summary while none of the work
was durable. The server had no way to tell: the editor answered "success"
and the tool result carried no durability information, unlike the
external MCP path which does track baseRevision.

The editor now reports { revision, pending, failed } with each tool
result, derived from state SaveCoordinator already keeps, and the server
stores it on the run. executeBrowserTool then fails mutating tools when
the editor's writes are settled-failed, telling the agent to stop rather
than build further work on state that cannot be saved.

The distinctions matter as much as the check:

- reads never fail on a save error; the agent still has to read state in
  order to report the problem
- a pending save is not a failure. Autosave debounces by 500ms and
  normally settles after the tool returns, so treating pending as loss
  would fail nearly every edit
- a missing signal is unknown, not failed, so an older editor keeps
  working

Project writes are also flushed at terminal settle, including failed and
cancelled runs, which is exactly when the last edits used to be stranded
inside the autosave debounce window. A checkpoint flush every few tool
results bounds exposure further; neither is awaited during the run, so
tool throughput is unchanged.

editor-persistence.verify.ts pins the decision table.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributing requires an executable check for non-trivial logic, and the
repository enforces two further invariants: every verify file must be
reachable from `npm test`, and must be tracked by git.

copilot-agent.verify.ts covers the parts worth pinning:

- turn parsing: saved model/effort fallback, explicit-null suppressing a
  saved effort, tool-name validation, duplicate-tool rejection
- provider attribution for the multi-vendor catalog, including the
  fallback for unrecognised vendors
- capabilities sourced from the runtime rather than the bundled catalog,
  the estimate path when a model cannot be described, and a user override
  still outranking both
- the version gate treating an unreadable version as unsupported

Also registers editor-persistence.verify.ts, which was written earlier and
was likewise not in the chain.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Copilot vendor page and the composer both fell back to placeholders: a
"GH" monogram in settings, and the GitHub octocat in the model picker and
toolbar. The octocat is the wrong mark — it identifies GitHub, not Copilot.

Vendors the Copilot mark from @primer/octicons (MIT, published by GitHub),
normalised to this repo's icon shape: 24x24 viewBox, 1em sizing, and
fill="currentColor" so it follows the active skin like the OpenAI ring. The
header comment now records octicons alongside the existing lobehub/simple-icons
provenance, and the monogram fallback is dropped.

The composer imports the asset directly rather than reusing VendorIcon: that
component is settings-only today, and importing it would pull every vendored
SVG into the chat bundle for one icon. This mirrors how the codex entry already
imports its own PNG.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SSKUltra

SSKUltra commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Added the real Copilot mark, replacing the placeholders.

Settings previously fell back to a GH monogram, and the composer model picker and toolbar used the GitHub octocat — which identifies GitHub, not Copilot.

The mark is vendored from @primer/octicons (MIT, published by GitHub) and normalised to this repo's icon shape: 24x24 viewBox, 1em sizing, fill="currentColor" so it follows the active skin the same way the OpenAI ring does. The header comment in vendorIcons.tsx now records octicons alongside the existing lobehub / simple-icons provenance, and the monogram fallback is removed.

The composer imports the asset directly rather than reusing VendorIcon: that component is settings-only today, so importing it would pull every vendored SVG into the chat bundle for a single icon. This matches how the codex entry already imports its own PNG.

Gates re-run on the updated branch: lint ✅ · npm test (171 segments) ✅ · verify:agent-skill ✅ · tsc -b + vite build ✅. Confirmed the mark ships in the built bundle (dist/assets/copilot-*.js).

Keep explicit OFox requests on the OFox provider, expose it only for video, and fail before outbound requests when its key is missing. Reject malformed provider task IDs and preserve resume polling without resubmission.

Shared paths: the existing model fallback and other provider strategies remain unchanged; OFox capability is enabled only with its configured key. Regressions cover legacy defaults, OFox field preservation, capability isolation, credential guards, task IDs, and resume behavior.

Validation: forced TypeScript build, zero-warning lint, 88 affected verifies, focused provider checks, generated catalog check, and production build passed. A real DeepSeek browser run executed ToolSearch, submit_video(model=ofox), and track_progress; the actual video POST preserved ofox and reported the expected missing-key error with provider=ofox. No paid OFox generation was performed. Existing font warnings and duplicate post-settle draft cleanup 403 are outside this change.
Merge current main and repair bundled CLI resolution, SDK streaming, Auto model discovery, and optional-backend startup. Remove unrelated persistence changes so existing API and Codex save paths retain their behavior.

Validation: tsc -b, lint, verify:server-runs, appShell startup isolation regression, production build, and desktop main bundle passed. Real CLI 1.0.82 authenticated through gh-cli and streamed READY. Chrome selected Copilot Auto and executed set_aspect_ratio plus read_timeline through browser tool claims and results, returning the portrait canvas response.
Reuse the shared unpackedPath helper and explicitly unpack Copilot platform packages and Koffi. Cover archive paths with spaces and preserve normal development resolution.

Validation: Copilot verification, lint, TypeScript build and desktop main bundle passed. A real app.asar under a path with spaces was loaded by Electron; CLI 1.0.82 resolved to app.asar.unpacked and authenticated through gh-cli. npm regenerated the lockfile from the manifest with no changes.
Include the OFox credential, task ID, and resume regressions in the npm test gate. No production behavior changes.

Validation: verify-registration.verify.mjs and verify-gate-coverage.verify.mjs passed for all 572 verify files; ofox-video-provider.verify.ts passed.
Snapshot the complete transmitted tool payload once and use it for context preparation, request dispatch, and usage metadata. Reject Windows shell shims before selecting a CLI executable, so an override cannot mask a valid bundled native binary.

Validation: TypeScript build, lint, desktop main bundle, and the complete server-runs verification script passed. Added registered regressions for full-tool budget accounting and Windows shim/native candidate ordering. In Chrome, real Copilot Auto called manage_timelines and read_project, verified 1920x1080, and completed both browser tool results and run settlement with HTTP 200. Native tool permissions remain host-only.
# Conflicts:
#	src/components/settings/vendorIcons.tsx
Preserve both OFox provider checks and all reliability regression scripts while resolving the test:serial registration conflict.

Validation: both registration gates passed for 574 files, five focused OFox/provider/capability verifications passed, and the generated server tool catalog is current.
The combined Copilot and oFOX settings pane exceeded the enforced 500-line limit. Extract only Copilot rendering and reasoning options, passing existing fields and capability override callbacks unchanged.

Validation: TypeScript, lint, and verify-gate-coverage passed; all 576 verification files remain reachable from npm test.
@0xsline
0xsline merged commit ae82d39 into 0xsline:main Sep 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants