feat(agent): add GitHub Copilot as an in-app agent backend - #139
Conversation
Add OFox (https://ofox.ai), a multi-model gateway, as: - an OpenAI-compatible LLM preset (mirrors the openrouter entry; keystore whitelist, vendor icon monogram, vision label, verify assertions) - a video generation provider (async task id + polling, same shape as grok-imagine-video, implemented in its own server/plugins module): text-to-video, first-frame / first-and-last-frame image-to-video, and up to 9 image references (frames and references are mutually exclusive at the API level, enforced before submission); generateAudio and seed pass through. Local project media rides as base64 data URLs, which the OFox gateway re-hosts on its own object storage (verified live). Validation, resumer registration, capabilities, tool schema, settings page with model discovery via the /models probe, agent skill reference, i18n (en/it/ru), .env.example, and verify coverage included. refVideos/refAudios are supported by the OFox API but not wired yet; per-model duration/resolution limits are enforced by the API with a clear 400 before any task is created, so no local per-model whitelist is kept. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mirror_urls are persistent signed CDN addresses returned when the upstream has mirroring enabled; unsigned_urls are temporary upstream links that may expire within 24 hours. Prefer the former and fall back to the latter, matching the official retrieve-endpoint recommendation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drive Copilot in-app through @github/copilot-sdk, alongside the existing api and codex backends, rather than only exposing OpenChatCut over MCP. The SDK is a direct analogue of the codex app-server integration, so this mirrors server/codex/ closely: - server/copilot/installation.ts discovers the CLI (bundled platform package, PATH, common install roots, OPENCHATCUT_COPILOT_PATH) - server/copilot/client.ts owns one runtime with an isolated COPILOT_HOME (~/.openchatcut/copilot) so in-app runs never disturb the user's own ~/.copilot, and caches the model catalog - server/copilot/turn-manager.ts registers OpenChatCut's tool catalog as SDK tools with in-process handlers and maps session events onto the same stream-event union the codex backend emits Host tools are the whole tool surface: availableTools is restricted to custom tools, so the agent has no shell, filesystem, web or MCP access, and any permission request is rejected as a filter escape. Copilot reports exact per-model limits, so capabilities come from resolveCopilotModelCapabilities rather than the bundled models.dev catalog, and models are attributed to their upstream provider (claude-* -> anthropic, grok-* -> xai) so capability overrides and vision selection keep working. Models without tool support are hidden. Two details worth recording: - tool failures return a typed ToolResultObject; throwing collapses to a generic "Tool execution failed" and strips the diagnostic the agent needs to recover - __images results are handed over as binary attachments, matching the codex backend's inputImage handling, so frame contact sheets remain viewable instead of being stringified into the prompt The turn uses an idle timeout rather than a wall-clock cap: a long edit streams continuously for many minutes and must not be killed for total elapsed time. It is suspended while a host tool is in flight, bounded by the broker's own maximum so a never-settled call cannot hang the turn. Auth stays with the CLI (copilot login); OpenChatCut never sees the credential and the settings pane is read-only status plus model choice. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
An agent run could apply a long sequence of edits, verify them by
rendering frames, and write a confident summary while none of the work
was durable. The server had no way to tell: the editor answered "success"
and the tool result carried no durability information, unlike the
external MCP path which does track baseRevision.
The editor now reports { revision, pending, failed } with each tool
result, derived from state SaveCoordinator already keeps, and the server
stores it on the run. executeBrowserTool then fails mutating tools when
the editor's writes are settled-failed, telling the agent to stop rather
than build further work on state that cannot be saved.
The distinctions matter as much as the check:
- reads never fail on a save error; the agent still has to read state in
order to report the problem
- a pending save is not a failure. Autosave debounces by 500ms and
normally settles after the tool returns, so treating pending as loss
would fail nearly every edit
- a missing signal is unknown, not failed, so an older editor keeps
working
Project writes are also flushed at terminal settle, including failed and
cancelled runs, which is exactly when the last edits used to be stranded
inside the autosave debounce window. A checkpoint flush every few tool
results bounds exposure further; neither is awaited during the run, so
tool throughput is unchanged.
editor-persistence.verify.ts pins the decision table.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributing requires an executable check for non-trivial logic, and the repository enforces two further invariants: every verify file must be reachable from `npm test`, and must be tracked by git. copilot-agent.verify.ts covers the parts worth pinning: - turn parsing: saved model/effort fallback, explicit-null suppressing a saved effort, tool-name validation, duplicate-tool rejection - provider attribution for the multi-vendor catalog, including the fallback for unrecognised vendors - capabilities sourced from the runtime rather than the bundled catalog, the estimate path when a model cannot be described, and a user override still outranking both - the version gate treating an unreadable version as unsupported Also registers editor-persistence.verify.ts, which was written earlier and was likewise not in the chain. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Copilot vendor page and the composer both fell back to placeholders: a "GH" monogram in settings, and the GitHub octocat in the model picker and toolbar. The octocat is the wrong mark — it identifies GitHub, not Copilot. Vendors the Copilot mark from @primer/octicons (MIT, published by GitHub), normalised to this repo's icon shape: 24x24 viewBox, 1em sizing, and fill="currentColor" so it follows the active skin like the OpenAI ring. The header comment now records octicons alongside the existing lobehub/simple-icons provenance, and the monogram fallback is dropped. The composer imports the asset directly rather than reusing VendorIcon: that component is settings-only today, and importing it would pull every vendored SVG into the chat bundle for one icon. This mirrors how the codex entry already imports its own PNG. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Added the real Copilot mark, replacing the placeholders. Settings previously fell back to a The mark is vendored from The composer imports the asset directly rather than reusing Gates re-run on the updated branch: |
Keep explicit OFox requests on the OFox provider, expose it only for video, and fail before outbound requests when its key is missing. Reject malformed provider task IDs and preserve resume polling without resubmission. Shared paths: the existing model fallback and other provider strategies remain unchanged; OFox capability is enabled only with its configured key. Regressions cover legacy defaults, OFox field preservation, capability isolation, credential guards, task IDs, and resume behavior. Validation: forced TypeScript build, zero-warning lint, 88 affected verifies, focused provider checks, generated catalog check, and production build passed. A real DeepSeek browser run executed ToolSearch, submit_video(model=ofox), and track_progress; the actual video POST preserved ofox and reported the expected missing-key error with provider=ofox. No paid OFox generation was performed. Existing font warnings and duplicate post-settle draft cleanup 403 are outside this change.
Merge current main and repair bundled CLI resolution, SDK streaming, Auto model discovery, and optional-backend startup. Remove unrelated persistence changes so existing API and Codex save paths retain their behavior. Validation: tsc -b, lint, verify:server-runs, appShell startup isolation regression, production build, and desktop main bundle passed. Real CLI 1.0.82 authenticated through gh-cli and streamed READY. Chrome selected Copilot Auto and executed set_aspect_ratio plus read_timeline through browser tool claims and results, returning the portrait canvas response.
Reuse the shared unpackedPath helper and explicitly unpack Copilot platform packages and Koffi. Cover archive paths with spaces and preserve normal development resolution. Validation: Copilot verification, lint, TypeScript build and desktop main bundle passed. A real app.asar under a path with spaces was loaded by Electron; CLI 1.0.82 resolved to app.asar.unpacked and authenticated through gh-cli. npm regenerated the lockfile from the manifest with no changes.
Include the OFox credential, task ID, and resume regressions in the npm test gate. No production behavior changes. Validation: verify-registration.verify.mjs and verify-gate-coverage.verify.mjs passed for all 572 verify files; ofox-video-provider.verify.ts passed.
Snapshot the complete transmitted tool payload once and use it for context preparation, request dispatch, and usage metadata. Reject Windows shell shims before selecting a CLI executable, so an override cannot mask a valid bundled native binary. Validation: TypeScript build, lint, desktop main bundle, and the complete server-runs verification script passed. Added registered regressions for full-tool budget accounting and Windows shim/native candidate ordering. In Chrome, real Copilot Auto called manage_timelines and read_project, verified 1920x1080, and completed both browser tool results and run settlement with HTTP 200. Native tool permissions remain host-only.
# Conflicts: # src/components/settings/vendorIcons.tsx
# Conflicts: # package.json
Preserve both OFox provider checks and all reliability regression scripts while resolving the test:serial registration conflict. Validation: both registration gates passed for 574 files, five focused OFox/provider/capability verifications passed, and the generated server tool catalog is current.
The combined Copilot and oFOX settings pane exceeded the enforced 500-line limit. Extract only Copilot rendering and reasoning options, passing existing fields and capability override callbacks unchanged. Validation: TypeScript, lint, and verify-gate-coverage passed; all 576 verification files remain reachable from npm test.
# Conflicts: # package.json
What this adds
A third agent backend — GitHub Copilot — alongside
apiandcodex, driven in-app through the official@github/copilot-sdk.Today Copilot users can only reach OpenChatCut from the outside, over the external MCP endpoint. That path is deliberately narrow: only draft-safe read/edit tools are exposed, tool calls round-trip through the broker into the browser, and the conversation lives in the user's terminal rather than in the editor's chat. This gives Copilot the same first-class, in-app treatment Codex already has.
Why the SDK rather than MCP or
--acp@github/copilot-sdkis a close analogue of thecodex app-serverintegration, soserver/copilot/mirrorsserver/codex/deliberately:codex app-server --listen stdio://defineTool()with in-process handlersCODEX_HOMEbaseDirectory→COPILOT_HOMEonPermissionRequestcopilot --acpwas considered and rejected: ACP has no per-turn tool injection, so it would have required running a separate MCP server for the editor's own tools — reintroducing exactly the indirection this removes.Design decisions worth reviewing
Tools are the entire surface.
availableToolsis restricted to custom tools, so the agent has no shell, filesystem, web or MCP access, and any permission request is rejected as a filter escape rather than prompted. Verified by asking it to runls, read~/.zshrcand fetch a URL: zero tool calls, and it enumerated only the OpenChatCut tools it had.Capabilities come from the runtime, not the bundled catalog. Copilot reports exact per-model limits, so
resolveCopilotModelCapabilities()uses those and records them as exact rather than estimated. Models are attributed to their upstream provider (claude-*→ anthropic,grok-*→ xai) so capability overrides and vision-model selection keep working. Models without tool support are hidden, since every editing flow needs tool calls.Tool failures return a typed
ToolResultObject. Throwing collapses to a generic"Tool execution failed"and strips the diagnostic. With the typed failure the model receives the real reason verbatim — confirmed by returningUnknown clip: clip-zzzand seeing it echoed back.__imagesare handed over as binary attachments, matching the Codex backend'sinputImagehandling, soview_asset_framescontact sheets stay viewable instead of being stringified into the prompt. Verified with a synthetic sheet (red left half, blue right half) that the model described correctly.The turn uses an idle timeout, not a wall-clock cap. A long multi-shot edit streams continuously for many minutes and must not be killed for elapsed time. It is suspended while a host tool is in flight, bounded by the broker's own maximum so a never-settled call cannot hang the turn.
Auth stays with the CLI. Users run
copilot login; OpenChatCut never sees the credential. The settings pane is read-only status plus model/effort selection, and session state is isolated under~/.openchatcut/copilotso in-app runs never disturb the user's own~/.copilot.Second commit: durable agent edits
While building this I hit a failure worth fixing independently — it is backend-agnostic and affects
apiandcodexequally.A run applied a long sequence of edits, verified them by rendering frames, and wrote a confident summary while none of the work was durable. The server could not tell: the editor answered "success" and the tool result carried no durability information, unlike the external MCP path which does track
baseRevision.The editor now reports
{ revision, pending, failed }with each tool result, derived from stateSaveCoordinatoralready keeps, andexecuteBrowserToolfails mutating tools when the editor's writes are settled-failed. The distinctions matter as much as the check:Project writes are also flushed at terminal settle, including failed and cancelled runs, which is exactly when the last edits used to be stranded inside the debounce window.
Happy to split this into its own PR if you'd prefer — it shares
store-types.tsandroutes.tswith the first commit, so it would need to land second.Testing
server/copilot/copilot-agent.verify.tsandserver/agent-runs/editor-persistence.verify.ts, both registered inverify:server-runsand git-tracked (theverify-gate-coverageandverify-registrationinvariants both catch omissions here — they caught mine).Coverage: turn parsing (saved model/effort fallback, explicit-null suppression, tool-name validation, duplicate rejection), provider attribution incl. unknown-vendor fallback, capabilities from runtime vs. estimate vs. user override, the version gate, and the full mutating/read × durable/pending/failed/missing decision table.
Local gates on this branch, rebased onto current
main:Acceptance evidence
Live against a real Copilot subscription:
Driven against the real generated tool catalog, the model produced schema-valid arguments unaided (computing 600 − 60 = 540 frames at 30fps).
Notes for the maintainer
@github/copilot-sdk. The CLI itself is discovered rather than bundled (OPENCHATCUT_COPILOT_PATHoverrides), so it is not a hard install requirement.COPILOT_MODEL,COPILOT_REASONING_EFFORT, registered inMODEL_ROUTING_NAMESinkeystore.verify.ts.verify:i18npasses. Russian was not required by the coverage checks.