dmihalcik-virtru · Oct 19, 2026 · Oct 19, 2026 · Oct 19, 2026 · Oct 20, 2026
@@ -0,0 +1,274 @@
+---
+ticket: DSPX-159
+title: Remove streamsaver.js from opentdf/web-sdk example code
+status: draft
+authors:
+    - dmihalcik@virtru.com
+branches:
+    - opentdf/web-sdk:DSPX-159
+prs:
+    - opentdf/web-sdk#1098
+created: 2026-10-19T00:00:00Z
+updated: 2026-10-20T00:00:00Z
+jira_priority: Major
+---
+
+
+# Remove streamsaver.js from opentdf/web-sdk example code
+
+## Summary
+
+Replace StreamSaver in the sample web-app with a **first-party streaming
+download service worker** served from the app's own origin. Encrypt/decrypt
+output streams straight into the browser's download manager, like a normal
+download link. Memory and disk use don't depend on file size, and there is no
+storage quota, so files over 10 GiB work. Chromium, Firefox and WebKit all use
+the same path. Only contexts without service workers fall back to an
+in-memory Blob.
+
+This removes the third-party MITM page from the download path. Investigating
+it also found the real cause of DSPX-5084: Playwright's Firefox drivers keep
+download bodies in memory. StreamSaver was not the cause, so DSPX-5084 does not
+depend on this work.
+
+## Problem / Motivation
+
+- `web-app/src/App.tsx` `toFile()` piped output into
+  `streamsaver.createWriteStream()`. StreamSaver 2.0.6 is unmaintained, and by
+  default it runs through `jimmywarting.github.io/StreamSaver.js/mitm.html` plus
+  a service worker on that third-party origin. Sample code for a data-security
+  SDK should not route plaintext through a page we don't control.
+- StreamSaver needs the MITM only because a library can't assume its host page
+  can register a service worker. We own the web-app, so we can register the
+  worker on our own origin.
+- The app **already registered its own service worker** (`web-app/public/sw.js`),
+  apparently adapted from `native-file-system-adapter`'s downloader example.
+  It could not have worked:
+  - **It rejected every message.** Its origin check compared against
+    `'http://localhost:65432/'`, and an origin never ends in `/`.
+  - **It had no backpressure.** It replied `PULL` as soon as it queued a chunk,
+    not when the stream pulled one, so nothing bounded its queue.
+  - **It had no keepalive.**
+- The "Save As" sink used `native-file-system-adapter`'s `showSaveFilePicker`.
+  On Firefox and Safari that falls back to the library's downloader, which
+  talked to the same broken `sw.js`. Its own keepalive used `setTimeout`, so it
+  fired only once.
+- Staging output in the origin private file system (OPFS) was considered and
+  rejected as the main path. It is capped by origin quota (about 10 GiB
+  best-effort on Firefox), doubles disk use, and delays the visible download
+  until processing finishes.
+
+## Design (as built)
+
+### Streaming download service worker
+
+`web-app/public/sw.js` holds the worker; the page side is
+`web-app/src/downloadSink.ts`. `registerDownloadWorker()` registers `./sw.js`
+from `main.tsx`. It does nothing, with a warning, where service workers are
+unavailable.
+
+**Flow for one download:**
+
+1. **Page:** `toFile()` waits up to 5 s for `navigator.serviceWorker.ready`. It
+   then creates a `MessageChannel` and posts
+   `{ type: 'register', id, filename }` to `registration.active`, transferring
+   one port.
+2. **Service worker:** it checks `event.origin === self.location.origin`, wraps
+   the port in a `ReadableStream` (high-water mark 4 MiB, measured in bytes),
+   stores it under `id`, and replies `{ type: 'ready' }`.
+3. **Page:** if `ready` doesn't arrive within 5 s (for example, an outdated
+   worker), it falls back to a Blob. The stream hasn't been touched yet, so
+   that is still possible.
+4. **Page:** it opens a hidden `<iframe>` at
+   `<scope>__download__/<id>/<encoded filename>`. The iframe is a navigation
+   inside the worker's scope, so the worker handles it even when the page
+   itself isn't controlled.
+5. **Service worker:** it answers once per `id` with the stream and these
+   headers: `Content-Disposition: attachment` (RFC 5987 `filename*` plus a
+   sanitised ASCII fallback), `Content-Type: application/octet-stream`,
+   `Cache-Control: no-store`, `X-Content-Type-Options: nosniff`. Unknown or
+   already-used ids get a 404.
+6. **Page:** `stream.pipeTo(new WritableStream(portSink, { highWaterMark: 1 }), { signal })`.
+
+**Backpressure, credit-based.** The worker posts `{ type: 'pull' }` from its
+source's `pull()` method. The page's sink `write()` waits until it holds a
+credit, then posts one chunk. The chain becomes: download manager disk writes →
+`Response` body reads → worker `pull()` → page `write()` → SDK output → SDK
+source. Every stage holds a bounded amount. Chunks are copied before transfer
+(`chunk.slice()`, then transferring its buffer), because a chunk may be a view
+into a buffer the SDK still holds.
+
+**Keepalive.** While a download streams, the page posts `{ type: 'ping' }` to
+the worker every 10 s. Chunks arrive on a `MessagePort`, which may not count
+as worker activity.
+
+**Errors and cancels.**
+- **Page abort or SDK error:** the sink posts `{ type: 'abort', reason }`, and
+  the worker errors the response body, so the browser marks the download
+  failed instead of leaving a truncated file.
+- **Abort while waiting for a credit:** `WritableStream` calls the sink's
+  `abort()` only after the in-flight `write()` settles. A write waiting for a
+  credit would deadlock, so the sink also watches `controller.signal`.
+- **Cancel in the browser's download UI:** the worker's `cancel()` posts
+  `{ type: 'cancel' }`, and the page sink errors, aborting the SDK pipeline.
+  Chromium doesn't report this cancel
+  ([crbug 638494](https://bugs.chromium.org/p/chromium/issues/detail?id=638494));
+  the app's Cancel button still works there.
+
+**Lifecycle.** The worker keeps `skipWaiting()` + `clients.claim()`. Fetches
+outside `__download__/` fall through to the network. The hidden iframe is
+removed 30 s after the stream ends.
+
+### Fallback
+
+If no worker is available or none answers, `toFile()` buffers the whole output
+into a Blob and clicks an anchor. Size is then limited by RAM, and it logs a
+warning. `toFile()` returns which strategy it used.
+
+### "Save As" sink: native only
+
+`native-file-system-adapter` is gone. The app calls `window.showSaveFilePicker`
+where it exists (Chromium). Elsewhere, `#fsapiSink` is disabled with a tooltip.
+Firefox has said it won't ship the File System Access API.
+
+### Other options considered
+
+- **OPFS staging:** deferred. It would only help contexts that have OPFS but no
+  service worker. If needed: write with `createWritable()`, then `getFile()` →
+  `URL.createObjectURL` → anchor click. Check `navigator.storage.estimate()`
+  first, keep at most one staged file, and sweep on load.
+- **Dedicated worker:** it can't start a download, so it can't replace the
+  service worker. It could run the SDK off the main thread and hand chunks
+  straight to the service worker (possible follow-up).
+- **Encrypt/decrypt inside the service worker:** rejected. It would move OIDC
+  tokens and DPoP keys into the worker and tie the whole job to the worker's
+  lifetime.
+
+## Inputs / Outputs / Contracts
+
+```ts
+// web-app/src/downloadSink.ts
+export type SinkStrategy = 'service-worker' | 'blob';
+
+export function registerDownloadWorker(): void;
+export function canSaveAs(): boolean; // 'showSaveFilePicker' in window
+export async function toFile(
+  stream: ReadableStream<Uint8Array>,
+  filename?: string, // default 'download.tdf'
+  options?: StreamPipeOptions
+): Promise<SinkStrategy>;
+```
+
+Messages between page and service worker:
+
+```ts
+// page → SW, via registration.active.postMessage
+{ type: 'register'; id: string; filename: string } // one port transferred
+{ type: 'ping' }
+// page → SW, via the port
+{ type: 'chunk'; chunk: ArrayBuffer } | { type: 'close' } | { type: 'abort'; reason: string }
+// SW → page, via the port
+{ type: 'ready' } | { type: 'pull' } | { type: 'cancel'; reason?: string }
+```
+
+`toFile()` keeps its old call shape, so the `#fileSink` radio and the Playwright
+specs (`#fileSink` + `page.waitForEvent('download')`) are unchanged.
+
+## Edge Cases & Constraints
+
+- **Secure context.** Service workers need HTTPS or `localhost`. A deployment on
+  plain HTTP falls back to the Blob.
+- **Updating the worker.** `skipWaiting()` replaces a running worker right away,
+  which kills any download it is streaming. That is acceptable for a sample
+  app.
+- **Concurrent downloads.** Each download has its own id, port and credits.
+- **Firefox's aborted download.** Stock Firefox leaves a 0-byte placeholder and
+  a `.part` file in the download folder after a failed download.
+- **Security.** Only same-origin pages can register streams, and ids are random,
+  single-use UUIDs. Plaintext goes only to the user's download location.
+
+## Testing
+
+- **`web-app/tests/tests/sink.spec.ts`** needs no platform. The page imports
+  `downloadSink.ts` from the dev server and streams a byte pattern through
+  `toFile()`. It checks:
+  - byte-exact output;
+  - that an abort fails the download (skipped on Playwright's Firefox, which
+    never settles `download.failure()`);
+  - with `SINK_TEST_BYTES`, a large download with browser memory sampled.
+    `SINK_TEST_MIBPS` paces the source.
+- **`web-app/tests/playwright.moz-firefox.config.ts`** runs the suite on stock
+  Firefox over WebDriver BiDi. Stock Firefox ignores Playwright's download
+  folder, so specs locate downloads with `downloadPath()` from `acts.ts`.
+- **`huge.spec.ts`** now runs on Chromium and WebKit, and skips Playwright's
+  bundled Firefox (see Findings).
+
+## Findings (2026-10-19)
+
+Measured on macOS with Playwright 1.56 (chromium, firefox-1495, webkit-2215)
+and stock Firefox 158.
+
+- **Works everywhere, byte-exact.** The service-worker path works in Chromium,
+  Firefox and WebKit.
+- **Chromium memory is flat.** 4 GiB moved browser RSS from 0.34 to 0.35 GiB.
+  Against a slow reader, the producer stays a bounded 5–6 MiB ahead.
+- **Stock Firefox memory is flat.** Launched by hand with no automation, the
+  parent stayed at about 640 MiB on an unthrottled 8 GiB (14 s), and at
+  640–825 MiB on 4 GiB paced at 64 MiB/s.
+- **The worker survives long downloads.** 2 GiB paced at 6 MiB/s (342 s, past
+  Firefox's 5-minute limit) completed on Chromium and Firefox with the
+  keepalive.
+- **Playwright's bundled Firefox (Juggler) keeps every response body in
+  memory.** `NetworkObserver.js` `onDataAvailable` pushes each body into
+  `_responseBodyChunks`. A plain 4 GiB HTTP download moved the parent from 0.6
+  to 5.7 GiB, and 8 GiB through the sink peaked at about 10.5 GiB. This is what
+  killed the DSPX-5084 runner. Its pages also never become service-worker
+  controlled, although in-scope navigations are still intercepted.
+- **Playwright's BiDi channel (`moz-firefox`) also buffers.** It registers
+  `network.addDataCollector` for response bodies, and the parent grew from 0.9
+  to 5.2 GiB on 4 GiB at 64 MiB/s. So `playwright.moz-firefox.config.ts` can
+  run `huge` only where RAM is plentiful.
+- **WebKit buffers when the source far outruns the disk.** It is flat when
+  paced (about 0.52 GiB on 4 GiB at 64 MiB/s). Unthrottled, at about 4 GB/s,
+  memory grows with size: 3.0 GiB peak on 8 GiB, 7.9 GiB on 16 GiB. It is not
+  yet known whether real Safari does this; tracked in DSPX-5246.
+
+## Follow-ups
+
+- **DSPX-5084:** restore Firefox in `huge.spec.ts` without downloads. Encrypt to
+  the Dump sink, and decrypt CLI- or Chromium-made ciphertext to the Dump sink.
+  Report the buffering to Playwright.
+- **DSPX-5246:** WebKit backpressure; check real Safari.
+- **Content-Length on decrypt.** The plaintext size is known from the manifest,
+  and setting it would give the download UI a real progress bar.
+- **CI memory logging** in `large-tests.yaml` (a `free -m` loop), so a killed
+  runner leaves evidence.
+- **Optional:** OPFS fallback; running the SDK in a dedicated worker; testing
+  long downloads without the keepalive.
+
+## Out of Scope
+
+- SDK-side decrypt memory growth (DSPX-5075, DSPX-4654).
+- The Node `file-browser` source >4 GiB slice issue (DSPX-5239).
+
+## Acceptance Criteria
+
+- [x] `streamsaver`, `@types/streamsaver`, and `native-file-system-adapter` are
+      removed. No requests go to third-party origins during a download.
+- [x] Chromium and Firefox stream Download-sink output through the first-party
+      service worker with flat memory (sink-level: Chromium 4 GiB, stock
+      Firefox 8 GiB).
+- [ ] A >10 GiB encrypt/decrypt roundtrip through the app succeeds on Chromium
+      and Firefox, verified locally at least once.
+- [x] Backpressure works: a slow reader stalls the producer instead of filling
+      memory (Chromium, stock Firefox; WebKit exception tracked in DSPX-5246).
+- [x] Abort in the app fails the browser download rather than truncating it
+      (Chromium, WebKit, stock Firefox).
+- [ ] Cancelling in Firefox's download UI aborts the pipeline.
+- [x] `sink.spec.ts` passes on Chromium, Firefox, and WebKit.
+- [ ] `roundtrip.spec.ts` passes on Chromium, Firefox, and WebKit, and
+      `huge.spec.ts` passes on Chromium and WebKit.
+- [ ] Save As works on Chromium, and is disabled with an explanation elsewhere
+      (implemented; not yet checked by hand).
+- [x] The Playwright finding is recorded on DSPX-5084, with a re-scope that no
+      longer depends on this ticket.