Add transparent and matte background modes#2
Open
JeffLtz wants to merge 1 commit into
Open
Conversation
Adds two output modes that expose the matte as real transparency, so a consumer can composite the isolated subject over their own surface (e.g. a WebGL scene) instead of one of the built-in backgrounds: - `transparent` — RGBA = (subject.rgb · a, a): the subject on a transparent background (premultiplied), matte as the canvas alpha channel. - `matte` — RGBA = (a, a, a, a): the raw alpha as a white silhouette; a debug view and a reusable mask. Both plug into the existing pipeline with no context changes — the WebGL context is already alpha:true/premultiplied and the WebGPU canvas is configured alphaMode:'premultiplied'; the stock compositors just hardcode alpha 1.0. They surface through the same `background` normalization and the `attachPreview` / `setPreview` machinery, so `main:'none'` + a `preview:'transparent'` renders the isolated subject straight to a page-owned canvas (the output MediaStream can't carry alpha). Implementation mirrors composite_solid across both backends: - new GLSL / WGSL shaders (+ f16 WGSL variants) and per-backend ops - CompositorTransparent / CompositorMatte effect wrappers - CompositeSpec `transparent` / `matte` modes wired through buildCompositor, sameSpec, and renderer.translateBackgroundFor - 'transparent' / 'matte' keywords in BackgroundInput / Background + normalize - backend.presenters interface + WebGL/WebGPU registration - tests: WebGL pixel-exact premultiplied output + WebGPU compile/submit smoke - README backgrounds section Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Author
|
Hi Sam! In case it's not obvious this whole PR was generated by one of my agents so critical feedback is very very welcome. You can check out what I made here: https://butterchurn-rust.vercel.app/ And if you check "Isolate subject" you'll see how I incorporated Longpipe. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds two output modes that expose the matte as real transparency, so a consumer can composite the isolated subject over their own surface (a WebGL scene, another canvas, the DOM) instead of one of the built-in backgrounds:
transparent—RGBA = (subject.rgb · a, a): the subject on a transparent background (premultiplied), matte as the canvas alpha channel.matte—RGBA = (a, a, a, a): the raw alpha as a white silhouette; a debug view and a reusable mask.Why
Today the SDK only emits an opaque composite (subject over blur/image/color), and the output
MediaStreamcan't carry alpha. There was no way to get just the subject out to composite it yourself. This came out of putting a webcam subject in front of a MilkDrop-style WebGL visualizer — the cleanest path was to have longpipe hand back the subject on transparency and stack it, rather than chroma-key a synthetic background.How
No context changes were needed — the WebGL context is already
alpha:true/premultiplied and the WebGPU canvas is configuredalphaMode:'premultiplied'; the stock compositors just hardcodealpha = 1.0. This mirrorscomposite_solidacross both backends:CompositorTransparent/CompositorMatteeffect wrappersCompositeSpectransparent/mattemodes wired throughbuildCompositor,sameSpec, andrenderer.translateBackgroundFor'transparent'/'matte'keywords inBackgroundInput/Background+normalizeBackgroundbackend.presentersinterface + WebGL/WebGPU registrationBecause the network + alpha are shared across
main/previewand a preview effect already forces inference,main:'none'+preview:'transparent'renders the isolated subject to a page canvas efficiently (theMediaStreamoutput remains opaque, as documented).Tests
tests/effects/composite_alpha.test.ts:transparent→(255,0,0,255)/(0,0,0,0),matte→(255,255,255,255)/(0,0,0,0).npm run typecheckandnpm run buildpass.🤖 Generated with Claude Code