Skip to content

Add transparent and matte background modes#2

Open
JeffLtz wants to merge 1 commit into
sb2702:mainfrom
JeffLtz:transparent-matte-output
Open

Add transparent and matte background modes#2
JeffLtz wants to merge 1 commit into
sb2702:mainfrom
JeffLtz:transparent-matte-output

Conversation

@JeffLtz

@JeffLtz JeffLtz commented Jul 2, 2026

Copy link
Copy Markdown

What

Adds two output modes that expose the matte as real transparency, so a consumer can composite the isolated subject over their own surface (a WebGL scene, another canvas, the DOM) instead of one of the built-in backgrounds:

  • transparentRGBA = (subject.rgb · a, a): the subject on a transparent background (premultiplied), matte as the canvas alpha channel.
  • matteRGBA = (a, a, a, a): the raw alpha as a white silhouette; a debug view and a reusable mask.
new EffectsPipeline(stream, { background: 'transparent' })
// or, to preview onto a page-owned canvas while `main` stays untouched:
pipeline.attachPreview(canvas)
await pipeline.setPreview({ background: 'transparent' })

Why

Today the SDK only emits an opaque composite (subject over blur/image/color), and the output MediaStream can't carry alpha. There was no way to get just the subject out to composite it yourself. This came out of putting a webcam subject in front of a MilkDrop-style WebGL visualizer — the cleanest path was to have longpipe hand back the subject on transparency and stack it, rather than chroma-key a synthetic background.

How

No context changes were needed — the WebGL context is already alpha:true/premultiplied and the WebGPU canvas is configured alphaMode:'premultiplied'; the stock compositors just hardcode alpha = 1.0. This mirrors composite_solid across both backends:

  • new GLSL / WGSL shaders (+ f16 WGSL variants) and per-backend ops
  • CompositorTransparent / CompositorMatte effect wrappers
  • CompositeSpec transparent / matte modes wired through buildCompositor, sameSpec, and renderer.translateBackgroundFor
  • 'transparent' / 'matte' keywords in BackgroundInput / Background + normalizeBackground
  • backend.presenters interface + WebGL/WebGPU registration
  • README backgrounds section

Because the network + alpha are shared across main/preview and a preview effect already forces inference, main:'none' + preview:'transparent' renders the isolated subject to a page canvas efficiently (the MediaStream output remains opaque, as documented).

Tests

tests/effects/composite_alpha.test.ts:

  • WebGL: pixel-exact premultiplied output — transparent(255,0,0,255) / (0,0,0,0), matte(255,255,255,255) / (0,0,0,0).
  • WebGPU: compile-and-submit smoke (guarded on adapter availability).

npm run typecheck and npm run build pass.

🤖 Generated with Claude Code

Adds two output modes that expose the matte as real transparency, so a
consumer can composite the isolated subject over their own surface (e.g. a
WebGL scene) instead of one of the built-in backgrounds:

- `transparent` — RGBA = (subject.rgb · a, a): the subject on a transparent
  background (premultiplied), matte as the canvas alpha channel.
- `matte` — RGBA = (a, a, a, a): the raw alpha as a white silhouette; a debug
  view and a reusable mask.

Both plug into the existing pipeline with no context changes — the WebGL
context is already alpha:true/premultiplied and the WebGPU canvas is
configured alphaMode:'premultiplied'; the stock compositors just hardcode
alpha 1.0. They surface through the same `background` normalization and the
`attachPreview` / `setPreview` machinery, so `main:'none'` + a
`preview:'transparent'` renders the isolated subject straight to a page-owned
canvas (the output MediaStream can't carry alpha).

Implementation mirrors composite_solid across both backends:
- new GLSL / WGSL shaders (+ f16 WGSL variants) and per-backend ops
- CompositorTransparent / CompositorMatte effect wrappers
- CompositeSpec `transparent` / `matte` modes wired through buildCompositor,
  sameSpec, and renderer.translateBackgroundFor
- 'transparent' / 'matte' keywords in BackgroundInput / Background + normalize
- backend.presenters interface + WebGL/WebGPU registration
- tests: WebGL pixel-exact premultiplied output + WebGPU compile/submit smoke
- README backgrounds section

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@JeffLtz

JeffLtz commented Jul 2, 2026

Copy link
Copy Markdown
Author

Hi Sam! In case it's not obvious this whole PR was generated by one of my agents so critical feedback is very very welcome.

You can check out what I made here:

https://butterchurn-rust.vercel.app/

And if you check "Isolate subject" you'll see how I incorporated Longpipe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant