feat: native character encoding with Effect streams - #8133
Conversation
🦋 Changeset detectedLatest commit: 726906a The changes in this PR will be included in the next version bump. This PR includes changesets to release 30 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
|
Implemented in 4ad6253: explicit codec entry points, isolated registries, and an opt-in All registry. How it worksimport * as CharacterEncoding from "effect/CharacterEncoding"
import * as Utf8 from "effect/encoding/Utf8"
import * as Windows1251 from "effect/encoding/Windows1251"
const encoded = CharacterEncoding.encode("Привет", Windows1251.encoding)
const converted = source.pipe(
CharacterEncoding.transcodeStream(Windows1251.encoding, Utf8.encoding)
)
const registry = CharacterEncoding.makeRegistry([
Utf8.encoding,
Windows1251.encoding
])
registry.resolveUnsafe("cp1251") // Windows1251.encoding
registry.encodingExists("cp932") // false
// registry.resolve(label) provides a typed Effect lookup instead.The core operators import no mapping tables. Each codec module loads only its own mapping data. Typed lookup arrays and decoding tries are still initialized lazily on first use and cached; each conversion has independent incremental state. For applications needing all runtime labels, explicitly import Verified bundle boundariesMinified browser ESM; sizes are bytes, with gzip in parentheses. These standalone decoder entry points include retained Effect infrastructure.
The restricted registry retains Effect infrastructure for its typed lookup method; it still imports only one mapping table. These cases expose different APIs, so sizes are not interchangeable workload comparisons. The checked-in bundle harness asserts the imported mapping files and executes output bundles from both bundlers. Throughput benchmarks were also rerun; Effect rates remain roughly similar, with baseline variability called out in the report. Verification: 110 tests pass on Node/Bun/Deno, type tests pass on TS 5.9.3/6.0.3, 7,556 differential checks pass, and regeneration is reproducible. Source check and lint pass. Full local checks still report the previously documented unchanged Schedule/Logger issues. |
Multibyte codec optimization: what, why, and benchmark resultsFollowing the runtime experiments, we retained only the runtime-independent multibyte optimization. Status: committed and pushed in What changed
There are no runtime checks, new dependencies, or public API changes. Why retain this optimization?The simpler encoder gives substantial gains on Node, Bun, and Deno. It reduces work and over-allocation within the codec, without relying on runtime-specific APIs or changing Effect Streams. The other two experiments are not included in the committed implementation: the Buffer UTF-16LE path regressed on Deno, and direct CP1251-to-UTF8 transcoding regressed on Bun. Neither is part of these measurements. Fresh benchmark resultsMedian throughput in MiB/s, baseline → candidate. Percentages are medians of paired changes, so they need not equal the ratio of the displayed, rounded medians.
Method: macOS ARM64; Node 24.20.0, Bun 1.4.0, Deno 2.9.6. Baseline and candidate run in the same process, with five alternating 150 ms measurement rounds after 250 ms warmups per implementation. Runtime processes run sequentially. Inputs are approximately 64 KiB UTF-8-equivalent text; streaming uses 4,093-byte chunks. String-input throughput uses UTF-8 byte length; byte-input throughput uses actual input length. Output equality is checked before timing; timed one-shot cases consume output lengths, and streams consume chunks without collecting them. No I/O or cold-start costs are included. The clearest reproducible gain is CP932 encoding: approximately 42–47% across all three runtimes. Decoder and streaming benefits vary. Small changes may be noise; these microbenchmarks are not application-wide speedup claims. Correctness and validation
The benchmark report, reproduction commands, and links to raw samples are now committed alongside the benchmark harness and differential checks. |
|
I think this will need some redesign before being considered. It feels too complex right now. |
UTF-16LE Buffer encoding enabled — rerun and Deno fixCommitted and pushed in What changedUTF-16LE encoding now uses the available global UTF-16 decoding and UTF-16BE encoding are unchanged. No new dependency or runtime/version guard was added. Deno issue and solutionThe original Deno regression led to denoland/deno#36803 and the proposed fix denoland/deno#36804. The fix replaces the per-code-unit JavaScript conversion plus temporary-array copy with V8's native UTF-16 write into aligned destination buffers, preserving raw code units and safely handling unaligned destinations. That Deno PR is still open and unmerged. Released Deno 2.9.6 still regresses with this Effect change enabled. The fixed-build results below require the proposed Deno patch; they do not describe an already released fix. Fresh resultsEffect baseline: Median MiB/s, baseline → candidate. Percentages are medians of paired changes.
The matched local Deno builds make the difference clear: on the fixed runtime, the Buffer path improves raw encoding throughput about 22× over the JavaScript loop, the full-byte-checksum workload by 47%, and streaming throughput by 96%. On the unpatched local runtime, the same Effect change still regresses. Earlier results, for comparisonThe earlier confirmation run used Effect baseline
The rerun remains broadly consistent with the previous Node/Bun gains and the old Deno regression. The patched Deno results demonstrate that the bottleneck was in Deno's Buffer conversion path, not a fundamental Effect Streams limitation. Method and validation
|
I think it got more complicated because of the registry. But it's there because when you want to limit the bundle size, you can decide to register the encodings you want to bundle yourself. Otherwise it would inflate the bundle to 500kb. This means for example that if you know which encodings you need to support you can supply functions your own registry, which they can use to check if the encoding is supported, and use it, otherwise fail. But the registry makes it more complicated indeed. |
2 participants
Summary
Draft a native character-set conversion module with synchronous incremental codecs, typed Effect failures, and backpressured Effect Stream encode/decode/transcode operators.
2472166ea5a4825ca091b9550713c403852a566b(local 1.0.0-alpha.2 checkout), with full MIT attribution and a regeneration script.Draft decisions / scope
The requested source location is
packages/effect/src, so this currently exportseffect/CharacterEncoding, not a separately published@effect/encodingpackage. Core versus standalone packaging needs review. Existingeffect/Encodingremains unchanged.This is not a compatibility-complete replacement. UTF-7, CESU-8, automatic UTF-16/32 endianness, pseudo-codecs, transliteration and iconv's wider stateful codec set are outside this initial draft. UTF-16 decoding follows TextDecoder / iconv-lite alpha replacement semantics rather than 0.7.x raw Buffer semantics. Codecs now have explicit entry points, with an optional complete registry. Further mapping compression, broader malformed-input corpora and multibyte performance need work. No MSSQL dependency changes are included.
Explicit codecs and selective bundles
Conversion takes codec values, not globally resolved strings:
makeRegistryaccepts only explicit codecs, supports normalized aliases, and offers typedresolveplus synchronousresolveUnsafe. Unknown labels do not trigger implicit loading.effect/encoding/Allis an explicit opt-in to every codec and alias.esbuild and Rolldown bundle checks verify zero mapping modules for UTF-8, exactly one for CP1251 (also for a restricted UTF-8/CP1251 registry), and 89 for All. They execute the generated bundles too. The report includes exact byte/gzip sizes and reproduction.
Benchmarks
Node 24.20.0, macOS ARM64; approximately 64 KiB source text, 4,093-byte stream chunks, 100 ms warmup, five rotating-order rounds of 150 ms per provider. Full output equality is checked before timing. Synchronous rows use Unsafe APIs; streaming rows include each framework's pipeline overhead.
Throughput was rerun after the explicit-codec refactor, with label resolution outside timing. Effect throughput remains roughly in line with the first draft. Results are still mixed: single-byte conversion is promising; multibyte conversion and most streaming workloads trail iconv-lite. Published 0.7.3 has a substantial UTF-16 speed advantage through raw Buffer paths. The alpha CP1251 streaming baseline was unusually slow in this rerun; the report flags this variability rather than attributing an apparent win to the refactor.
Complete stable/alpha tables, all raw samples, limitations and reproduction commands:
Verification
pnpm lint-fix,pnpm --filter effect check, andgit diff --checkpass.pnpm checkreports TS6133 in unchangedpackages/effect/typetest/Schedule.tst.ts:80(metadata).pnpm jsdocs --checkreports two issues in unchangedLogger.ts(tag ordering and unresolvedconsolePrettylink); none in the new module.Keeping this draft for API/placement review, broader compatibility testing and performance work.