Skip to content

Schema compiler - #7908

Draft
gcanti wants to merge 55 commits into
mainfrom
schema-compiler
Draft

Schema compiler#7908
gcanti wants to merge 55 commits into
mainfrom
schema-compiler

Conversation

@gcanti

@gcanti gcanti commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Runtime compilation

Schema offers experimental, opt-in JIT and AOT compilation. Both work through
the normal SchemaParser APIs for decoding, encoding, type guards, and construction; schemas
remain composable and do not acquire a separate compiled type.

To enable JIT globally, import its side-effect entrypoint during startup:

import "effect/unstable/schema/SchemaJITCompiler/enable"
import { Schema, SchemaParser } from "effect"

const User = Schema.Struct({
  id: Schema.Number,
  name: Schema.String
})

const decodeUser = SchemaParser.decodeUnknownSync(User)

decodeUser({ id: 1, name: "Ada" })

Compilation is lazy: the import enables it, but parsers initialize their
operations only when first used. If the environment forbids new Function or
JIT compilation fails, Schema falls back to interpreted parsing without retrying
the failed compilation. This also applies to lazy checkpoints, without repeating
earlier transformations or middleware. Exceptions during parsing keep their
normal behavior; they do not trigger a restart in the interpreter.

One cache, interchangeable parsers

SchemaCompiler uses one WeakMap<AST, Entry> for interpreted, JIT, AOT, and
manually installed decoders. Installed decoders are CompiledDecoder objects,
not just decoding functions. Each entry provides lazy decodeEffect and makeEffect
operations, either installed or interpreted. Decoders can also supply validate
and is, as described below. The cache never stores parsing results.

The registry adds a parseEffect function to coordinate validate and decodeEffect,
and lazily prepares interpreted construction when the installed bundle omits makeEffect.
Both constructor and decoder functions are cached on that same entry. It also adds
an origin flag, either "interpreted" or "installed". This flag controls
replacement during installation, not validation: selective JIT preserves already
installed descendants but can replace interpreted ones. Callers of set supply
only the decoder operations, not these internal fields.

On first use, a parser reuses the cached entry or creates and caches a compiled
or interpreted decoder. Children use the same cache, so an interpreted parent
can have compiled children. Internal decoding resolves the entry's raw parser;
the public parseEffect boundary materializes its successful output. Construction
resolves makeEffect instead. Selective compilation remains active for children
even when their parent uses an interpreted constructor. All operations are lazy,
so recursive children resolve after their parent entry has been installed. No
separate constructor cache or recursive placeholder cache is needed.

On first use of a Declaration, its declared type parameters are registered through
the entry lookup carried by the same resolver before its callback runs. Their
operations remain lazy. This lets callbacks using public parsers find selectively
compiled children; new ASTs created inside a callback follow the normal registry
policy.

Choose how to populate it:

API Behavior
Import SchemaJITCompiler/enable Enables lazy JIT globally without replacing existing entries.
SchemaJITCompiler.enable(ast) Installs one root immediately and compiles its parsing/construction dependencies as needed. Operations remain lazy; already installed descendants are preserved.
SchemaCompiler.set(ast, decoder) Installs a trusted decoder, replacing any entry for that AST. AOT uses the same registry.

These modules live under effect/unstable/schema. Selective installation takes
an AST: use schema.ast for decoding, SchemaAST.flip(schema.ast) for encoding,
and SchemaAST.toType(schema.ast) for type guards and construction. The exact returned AST is
the cache key. Different AST objects have separate entries, even if structurally
equal; operations using the same AST object share an entry.

Install before the first execution of parsers you want to accelerate.
Creating a parser earlier is fine. Late installation is safe, but a parser that
already captured an entry keeps it, even when later calls change parse options.
This includes make, makeOption, and makeEffect: constructing a value can
populate the entry before its first decode. A later global JIT import does not
upgrade it. Explicit set still replaces the whole entry for new consumers;
omitting makeEffect from a replacement restores interpreted construction for them.

Parsing behavior and constraints

Every entry provides the complete decodeEffect operation. Two optional fast paths
avoid work that is unnecessary for successful decoding or boolean validation.
All operations initialize independently when first needed:

Operation Result Purpose
is, optional boolean Validates without constructing output, when checks do not require reconstructed values.
validate, optional Decoded value or SchemaCompiler.invalid Validates and constructs output without generating diagnostic issues.
decodeEffect, required Effect<value, SchemaIssue, R> Returns the actual output or detailed failure.
makeEffect, optional Effect<value, SchemaIssue, R> Constructs the node without replay; omission selects interpreted construction.

decodeEffect is required so every entry can produce output and explain failures,
even without any fast paths. It also handles transformations, middleware, and
asynchronous work when the AST requires them. It can be compiled or interpreted;
calling decodeEffect does not necessarily mean returning to the interpreter.

validate is optional because a synchronous, diagnostic-free first pass is not
always supported or safe to repeat. In particular, ASTs containing encodings
omit it so a later failure cannot repeat transformations or middleware.

is is optional because preserving validation semantics can require constructing
output. For example, a check on a Struct must see the reconstructed object with
extra properties removed. Such a schema uses validate, or decodeEffect if
validate is unavailable, instead of an output-free is. Omitting either fast
path removes an optimization, not parsing capability.

For decoding, entry.parseEffect tries validate when available. Success already
contains the output, so no detailed pass is needed. invalid contains no error
location or explanation, so failure requires one detailed decodeEffect pass. This
favors valid inputs at the cost of traversing invalid inputs again, only where
repetition is safe. Without validate, or for the missing sentinel, it calls
decodeEffect directly. Interpreter, JIT, and AOT implementations supply the operations
without implementing this dispatch. The synchronous decode and encode adapters
share a direct version of it, returning successful validate output without an
intermediate Effect. Encoding uses the flipped AST; when it equals the original,
both adapters use the same entry and execution path. Detailed traversal does not
restart fast validation at every child.

Construction calls entry.makeEffect directly, without is or validate.
It has different semantics from decoding: defaults are applied to Struct fields,
tuple/array elements, and Record values before the child is constructed. They do
not become defaults for that AST used as a root, a Union member, or a Record key.
Union selection stays conservative for missing discriminants. Class construction
preserves recognized instances; otherwise it constructs the source and creates
the instance. Ordinary Declaration callbacks retain their own parsing choices.

Using only construction does not initialize the decoder or validators. JIT
preparation failures select the interpreted implementation for the affected
operation, independently of decoding or construction that already works. If a
lazy child's compilation fails after a default has run, only that child falls
back. Neither default effects nor Class constructors are replayed. Normal Union
branch attempts remain unchanged, so more than one branch's defaults can run.

A composed Struct decoder can compile its fields, including transformations,
then apply the Struct's checks to the decoded output. Both stages read the same
AST, and only the complete decoder is installed. Checks run after stripping and
successful field decoding, without replaying transformations.

SchemaParser.is(schema) and Schema.is(schema) check toType(schema.ast)
with default parse options: excess properties are ignored and checks are enabled.
The guard uses is, then validate if is is unavailable, converting invalid
to false without a diagnostic pass. If neither exists, it uses ordinary
decoding and converts the outcome to a boolean; non-schema failures still throw.
For configurable type-side validation, use a decoding API with Schema.toType(schema).

The following constraints apply:

  • Runtime ParseOptions apply throughout parsing, without recompilation.
    Annotations cannot override them. Children parse sequentially; use Effect
    concurrency combinators for independent operations or inside transformations.
  • A failed validation can read properties and run checks twice. Checks must
    have no observable side effects; property getters must be deterministic and
    safe to repeat. Declaration parsers have the same constraint and must be
    synchronous.
  • ASTs with encodings enter decodeEffect directly. Transformations and middleware
    run once, with their validation checkpoints resolved through the shared cache.
    Local checkpoints preserve the original AST for checks and issues.
  • Unsupported nodes and code-size limits select composed or interpreted parsing;
    supported descendants can still compile. Parsing exceptions follow normal
    Effect defect behavior.

For custom set implementations, every operation must honor the active
ParseOptions. Return invalid only for invalid input, never to decline an
optimization; validate must not call decodeEffect and discard its issues.
User checks may themselves allocate issues. An absent optional input reaches
decodeEffect or makeEffect as SchemaCompiler.missing, distinct from a present
undefined. When a field produces no value, propagate missing as an Effect
success: the parent omits optional fields or reports a missing required key.
Public root adapters reject a final missing rather than exposing it.

Installation trusts the decoder to implement its AST and does not mutate the
supplied object. Operation accessors are evaluated once, on demand, with that
object as their receiver; missing optional operations are cached too. Installed
decodeEffect and makeEffect functions return ordinary Effects.

Ahead-of-time compilation

SchemaAOTCompiler.compile(asts) accepts a readonly array of ASTs and returns
a JavaScript ES module exporting install(asts). Share the root array between
the build script and the application:

export const roots = [Person.ast, Order.ast] as const

Generate the module at build time:

import * as SchemaAOTCompiler from "effect/unstable/schema/SchemaAOTCompiler"
import { writeFile } from "node:fs/promises"
import { roots } from "./schemas.js"

await writeFile("./schemas.decoders.js", SchemaAOTCompiler.compile(roots))

Install it before first parser execution:

import * as SchemaParser from "effect/SchemaParser"
import { install } from "./schemas.decoders.js"
import { Person, roots } from "./schemas.js"

install(roots)
const decodePerson = SchemaParser.decodeUnknownSync(Person)

Neither generation nor importing the generated module installs decoders.
install registers the roots and supported statically reachable dependencies.
Use [ast] for one schema; an empty array is a no-op. Repeated roots and shared
dependencies are emitted and installed once by identity.

JIT and AOT share generation rules and runtime support. Generated modules import
effect/unstable/schema/SchemaCompiler/runtime, not the code generator, and
work without dynamic function construction. Validators and composed Struct
decoders and constructors are static. Detailed diagnostics, transformation
orchestration, and Array, Record, Union, leaf, and Class constructors specialize
lazily in shared runtime support. AOT does not precompute every operation.
Constructor default links and Class source schemas are included among reachable
dependencies and read from the runtime ASTs without executing defaults during generation.

Keep these installation requirements in mind:

  • Regenerate after schema or Effect version changes. Installation trusts the
    array length, root order, AST structure, and sharing to match the build-time
    roots; it does not check compatibility.
  • Checks, symbols, transformations, and middleware come from the runtime ASTs,
    not serialization, preserving their identity.
  • Suspend thunks are not forced during generation. Their contents and other
    unsupported nodes use the interpreter, with no late JIT required.
  • Include type-side ASTs for construction and flipped ASTs for encoding separately
    when they differ from the supplied roots. As with set,
    late installation does not update parser closures that captured older entries.

Performance snapshot

Construction

Measured on 2026-09-08, Node 24.12.0, V8 13.6, Apple M3, macOS arm64.
Median ns/op through SchemaParser.make; nine isolated rounds, 500 ms measurement
and 150 ms warmup. Interpreted/JIT use calibrated batches; AOT uses batch 256
with dynamic code generation disabled and a separate fixture call site.
Schema setup is outside measurement except in the last row, which also includes
installation for AOT.

Case Interpreted JIT AOT
Struct, two fields 59.7 33.7 30.9
Struct, constructor default 101.2 65.8 63.4
Array of 32 Structs 792.1 811.8 801.4
Union, missing discriminant default 119.4 95.6 92.4
Class, plain input 156.3 155.6 151.8
Schema creation + first construction 3092.5 6857.0 4286.7

Retained parser heap, KiB/schema, for
Struct({ a: String, b: Number.withConstructorDefault(succeed(1)) }).
Five isolated processes per cell retain 1,000 distinct schemas and their public
parsers after first use with { a: "a", b: 1 }. Imports, static AOT modules,
64 warmup schemas, and schema construction precede the parser heap reading.
Two forced GCs run at each reading. Costs include public parser closures and AOT
installation, but exclude schema construction, about 3 KiB/schema.
These are retained V8 heap values, not peak or all native executable-code memory.

Operations used Interpreted JIT AOT
make 2.78 5.24 8.69
decode 1.83 2.35 5.01
make + decode 4.33 5.60 10.55
Decoding and type guards

Measured on 2026-09-08 on the same source revision as the construction snapshot.
All measurements use public SchemaParser APIs on Node 24.12.0, V8 13.6,
Apple M3, macOS arm64. The current snapshot includes every scenario in the
schema-compiler suite, not just the Moltar objects.

Moltar

Median ns/op, batch 256. Interpreted/JIT: five processes per case, 300 ms
measurement and 100 ms warmup. AOT: nine processes, 500 ms measurement and
150 ms warmup, with dynamic code generation disabled. AOT uses the same data and
worker with a different fixture call site. Sub-10 ns differences between JIT and
AOT are not a general ranking.

Case Interpreted JIT AOT
parseSafe, valid 265.5 5.8 5.9
parseSafe, extra property 265.3 5.9 5.9
parseSafe, invalid 2770.0 2840.0 3082.8
assertLoose, valid 266.6 3.3 3.3
assertLoose, extra property 266.7 3.3 3.3
assertLoose, invalid 118.7 3.0 1.4

Schema creation plus first use, median µs/op, five processes per case. AOT was
not measured in this fixture. Creation and first use are measured together.

Case Interpreted JIT
parseSafe 7.52 11.31
assertLoose 10.02 11.54
Other schema shapes

All 31 scenarios in schema-compiler, median ns/op from five processes per case,
300 ms measurement, 100 ms warmup, and shared calibrated batches.

Scenario Interpreted JIT
declaration-set-valid 2757.8 1393.7
checked-transformed-struct-valid 1712.1 1034.3
checked-transformed-struct-invalid 6067.8 5252.9
sync-decode-valid 79.1 6.8
sync-encode-valid 78.1 6.7
strict-record-1024-valid 203883.2 206146.6
strict-record-4096-valid 680157.5 758954.0
strict-record-4096-invalid 683783.0 1421426.9
array-100-valid 704.8 126.5
array-100-invalid-last 6181.0 4453.2
tuple-rest-valid 478.3 40.0
optional-struct-valid 1154.6 19.5
record-valid 3817.9 665.9
template-record-valid 9185.1 4052.9
struct-with-record-valid 1384.0 724.8
number-record-valid 4907.4 4547.7
transformed-key-record-valid 3575.2 3528.8
encoding-checked-struct-valid 779.2 30.9
literal-100-valid-last 33.2 10.0
literal-100-invalid 3247.7 3402.2
tagged-union-100-valid-last 113.0 27.1
tagged-union-100-invalid 3236.4 3456.3
checked-string-valid 20.0 8.9
template-literal-valid 161.8 48.2
transformation-struct-valid 1856.8 1271.6
transformation-root-valid 38.8 38.5
transformation-root-invalid 3531.6 3618.9
transformation-uppercase-valid 47.3 47.4
transformation-output-invalid 3542.5 3590.8
middleware-struct-valid 1961.3 381.6
recursive-node-valid 13543.7 8117.6
Memory and first-use CPU

Seven isolated processes per mode, each retaining 1,000 distinct schemas for
Struct({ name: String, count: Number, active: Boolean, tags: Array(String) }),
their inputs, and public decodeUnknownSync functions. The inputs contain
{ name: "a", count: 1, active: true, tags: ["a", "b"] }.
Two forced GCs run before construction, after construction, and after first use.

Retained heap, B/schema. Construction includes schemas, inputs and public parser
closures. First use includes AOT installation and lazy parser initialization.
Imports and the static AOT module precede the first reading and are excluded.
These are retained V8 heap values, not peak memory, RSS, or all native code memory.

Mode Construction First use Total
Interpreted 3208 3018 6226
JIT 3208 2425 5634
AOT 3203 5991 9194

First use, median µs/schema within the 1,000-schema batch, including AOT
installation. Schema construction, imports and GC are outside the timed region.
Process CPU includes background work and can exceed wall time. Warm CPU is not
measured separately.

Mode Wall time Process CPU
Interpreted 4.66 9.92
JIT 8.56 15.15
AOT 48.86 51.54

Coverage of the accompanying revision comparison: all 191 Effect cases in the
runtimeperf registry, including the 54 interpreter diagnostics and 31 Arbitrary
cases not tabulated here. AOT throughput coverage includes the six Moltar paths and six construction cases.
Peak memory and native executable-code memory are not covered. Current measurements are shown here without deltas;
revision-comparison reports retain their paired observations and confidence intervals.

@changeset-bot

changeset-bot Bot commented Sep 3, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: ae1a082

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 30 packages
Name Type
effect Patch
@effect/opentelemetry Patch
@effect/vitest Patch
@effect/ai-anthropic Patch
@effect/ai-openai-compat Patch
@effect/ai-openai Patch
@effect/ai-openrouter Patch
@effect/atom-react Patch
@effect/atom-solid Patch
@effect/atom-vue Patch
@effect/platform-browser Patch
@effect/platform-bun Patch
@effect/platform-deno Patch
@effect/platform-node-shared Patch
@effect/platform-node Patch
@effect/sql-clickhouse Patch
@effect/sql-d1 Patch
@effect/sql-libsql Patch
@effect/sql-mssql Patch
@effect/sql-mysql2 Patch
@effect/sql-pg Patch
@effect/sql-pglite Patch
@effect/sql-sqlite-bun Patch
@effect/sql-sqlite-do Patch
@effect/sql-sqlite-node Patch
@effect/sql-sqlite-react-native Patch
@effect/sql-sqlite-wasm Patch
@effect/docgen Patch
@effect/doctest Patch
@effect/openapi-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@effect-slopcop effect-slopcop Bot added the 4.0 label Sep 3, 2026
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Bundle Size Analysis

Generated from PR build output; treat the content below as untrusted.

File Name Current Size Previous Size Difference
arbitrary-combinators.ts 32.93 KB 34.05 KB -1.12 KB (-3.30%)
basic.ts 7.15 KB 7.15 KB 0.00 KB (0.00%)
batching.ts 10.17 KB 10.38 KB -0.22 KB (-2.09%)
brand.ts 6.63 KB 6.63 KB 0.00 KB (0.00%)
cache.ts 11.03 KB 11.03 KB 0.00 KB (0.00%)
config.ts 20.73 KB 21.76 KB -1.03 KB (-4.73%)
differ.ts 19.63 KB 20.55 KB -0.92 KB (-4.49%)
http-client.ts 22.02 KB 22.20 KB -0.18 KB (-0.83%)
http-router.ts 32.72 KB 32.91 KB -0.18 KB (-0.56%)
logger.ts 11.12 KB 11.12 KB 0.00 KB (0.00%)
metric.ts 9.28 KB 9.28 KB 0.00 KB (0.00%)
optic.ts 6.87 KB 6.87 KB 0.00 KB (0.00%)
pubsub.ts 15.31 KB 15.49 KB -0.18 KB (-1.16%)
queue.ts 12.09 KB 12.09 KB 0.00 KB (0.00%)
schedule.ts 11.20 KB 11.20 KB 0.00 KB (0.00%)
schema-binary.ts 38.53 KB 39.71 KB -1.18 KB (-2.97%)
schema-class.ts 19.11 KB 20.32 KB -1.21 KB (-5.93%)
schema-compiler-off.ts 16.96 KB 16.96 KB 0.00 KB (0.00%)
schema-compiler.ts 22.01 KB 22.01 KB 0.00 KB (0.00%)
schema-fromJsonSchemaDocument.ts 29.38 KB 30.48 KB -1.10 KB (-3.61%)
schema-representation-roundtrip.ts 25.32 KB 26.42 KB -1.09 KB (-4.14%)
schema-string-transformation.ts 13.91 KB 13.89 KB +0.02 KB (+0.13%)
schema-string.ts 11.39 KB 11.38 KB +0.01 KB (+0.10%)
schema-template-literal.ts 14.63 KB 15.76 KB -1.13 KB (-7.18%)
schema-toArbitrary.ts 32.48 KB 33.60 KB -1.11 KB (-3.31%)
schema-toCodeDocument.ts 23.53 KB 24.76 KB -1.23 KB (-4.98%)
schema-toCodecJson.ts 18.30 KB 19.52 KB -1.22 KB (-6.25%)
schema-toEquivalence.ts 18.58 KB 19.65 KB -1.08 KB (-5.48%)
schema-toFormatter.ts 18.69 KB 19.77 KB -1.08 KB (-5.46%)
schema-toJsonSchemaDocument.ts 22.60 KB 23.85 KB -1.24 KB (-5.21%)
schema-toRepresentation.ts 18.58 KB 19.79 KB -1.22 KB (-6.14%)
schema.ts 18.33 KB 19.53 KB -1.20 KB (-6.13%)
stm.ts 13.03 KB 13.03 KB 0.00 KB (0.00%)
stream.ts 10.06 KB 10.06 KB 0.00 KB (0.00%)

@gcanti
gcanti force-pushed the schema-compiler branch 2 times, most recently from 035d2b3 to 3ab7d8a Compare September 7, 2026 06:51
@effect-janitor effect-janitor Bot added the enhancement New feature or request label Sep 7, 2026
Remove schema annotation overrides and parsing concurrency, share interpreter and JIT helpers, and reuse option-independent generated functions. Update regression coverage, migration annotations, and runtime and bundle snapshots.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

4.0 enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant