Skip to content

Store a function's signature once, and expose it to primitives - #986

Merged
oflatt merged 9 commits into
egraphs-good:mainfrom
oflatt-claude:subst-egraph-introspection
Aug 13, 2026
Merged

Store a function's signature once, and expose it to primitives#986
oflatt merged 9 commits into
egraphs-good:mainfrom
oflatt-claude:subst-egraph-introspection

Conversation

@oflatt-claude

@oflatt-claude oflatt-claude commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Three additions to what a primitive body can see. None are specific to any one
extension: together they are what an out-of-tree primitive needs to walk the
term structure under an e-class and build a modified copy of it.

The motivating consumer is an unstable-subst substitution primitive in
egraphs-good/egglog-experimental#60, which lives there rather
than here because substitution is a language experiment, not core e-graph
machinery. This PR is only the introspection; it is worth reviewing on its own
terms.

Read::enodes_for_eclass(name, eclass, f)

Walks a constructor's rows by output e-class through the backend's lazy column
index, instead of scanning the table and filtering. Cherry-picked from #934,
along with the core-relations ExecutionState::for_each_matching_col and
egglog-bridge TableAction::for_each_output_value it rests on — credit to
that PR; this takes only the traversal hunks and none of the extraction work.

Read::constructor_schema(name), Read::function_schema(name), Read::table_subtype(name)

A table's declared signature, and whether it is a constructor or a function.
EGraph::functions_iter already exposes this from &EGraph, but a primitive
body only ever sees a state wrapper, and those carried no sort information at
all — so a primitive could read rows without being able to tell an e-class
column from a base value, or a constructor's eclass column from a function's
output.

The schema accessor is split by subtype and errors with WrongSubtype on a
mismatch, matching how the rest of Read and Write already work
(lookup/eclass_of, constructor_enodes/function_entries, set/add).
That split is not just cosmetic: a constructor's last column is an e-class and
a function's is an output. table_subtype is the error-free predicate for code
that accepts either.

Breaking: a signature is now stored once

The first cut of this branch added a FunctionSchemas registry to back those
accessors. That was a third copy of data egglog already kept twice —
TypeInfo::func_types holds a FuncType {name, subtype, input, output}, and
Function held the same content again as a ResolvedSchema plus
decl.subtype (and a third time, unresolved, in decl.schema). What was
missing was never the data, only a way to reach it from a state wrapper.

So TypeInfo::func_types became the single store and the shared cell, and the
new registry is gone:

  • TypeInfo::get_func_type returns Option<Arc<FuncType>>, not
    Option<&FuncType>.
  • Function points at the same Arc<FuncType>;
    Function::schema() -> &ResolvedSchema becomes
    Function::func_type() -> &FuncType.
  • ResolvedSchema is removed; its get_by_pos moves to FuncType.
  • declare_function reuses the signature typechecking already resolved instead
    of resolving the sorts a second time. It still resolves and records the
    functions desugaring generates (global bindings, proof tables), which never
    go through typechecking — so func_types now covers every declared table.
  • GenericFunctionDecl::resolved_schema is gone, along with the placeholder
    String::new() an unresolved decl carried in it and its
    ResolvedCall::view_types reader. Its only consumer was proof encoding, which
    runs over decls desugaring generates after typechecking (the functions
    global bindings lower to) and so cannot look those up by name — but it never
    needed a FuncType, only the column sorts, and those resolve by name from
    TypeInfo like any other sort. (Recording the generated signatures into
    TypeInfo instead does not work: the desugared program is typechecked again
    afterwards, so recording early trips the already-bound check.)
  • ResolvedCall::Func holds an Arc<FuncType>, so a resolved call site shares
    the signature rather than cloning it — and there is one of those per call in
    every rule. Patterns that matched
    ResolvedCall::Func(FuncType { subtype: Custom, .. }) cannot look through an
    Arc, so they match the variant and test the subtype in a guard via a new
    ResolvedCall::is_custom_func.

Two subtleties worth a reviewer's eye. TypeInfo::clone deep-copies the map: a
clone is an independent e-graph — a pushed copy, or the parallel typechecking
the proof checker keeps in original_typechecking — and declaring a function in
one must not make it resolve in the other. And pop restores the pushed
contents into the live cell the registered primitives already hold, rather than
swapping in a cell they have no handle to.

No lock: the signature reaches a primitive as a borrow

The obvious way to back those accessors is to hand the primitive an
Arc<RwLock<_>>, and the first cut of this branch did. That lock would never
have been guarding a race — declaring anything needs &mut EGraph, run_rules
already holds that borrow for the whole execution, and nothing on
Core/Read/Write can declare. It was there only because external functions
are stored as Box<dyn ExternalFunction + 'static>, so a wrapper cannot hold a
borrow of the e-graph.

So this changes the route instead of the synchronization. ExecutionState
carries an ExternalContext — a borrowed value the caller of an operation
makes visible to the external functions it reaches — and egglog passes
&TypeInfo into run_rules and with_execution_state*. The state wrappers
read it back, so constructor_schema hands out a &FuncType borrowed from the
e-graph itself.

The invariant is then enforced by the borrow checker rather than stood in for
by a lock: that &TypeInfo is live for exactly the operation that supplied it,
so the e-graph cannot be mutably borrowed to declare a function while a rule is
running. TypeInfo::func_types stays a plain owned map, get_func_type keeps
its Option<&FuncType> signature and costs a map lookup, TypeInfo::clone
stays derived, and push/pop restore it like every other field.

Rebuild paths pass None: they run generated rebuild rules whose actions
evaluate in the Write context, which has no schema accessors.

This does not touch the pre-existing ActionRegistry read lock in
RegistryPrimWrapper::invoke, which every registry primitive still pays per
invocation — the same seam would remove it, as a follow-up.

table_subtype also retires an existing workaround in egglog-experimental,
which determines a table's subtype by starting a constructor scan and reading
the error off the subtype check — removed in the companion PR.

Core::rebuild_container(type_id, value, remap)

Remaps a container value's contents and interns the result, over the existing
ContainerValues::rebuild_val_with. Out-of-tree code cannot reach this through
Core::register_container, which requires naming the container's Rust type —
impossible for an arbitrary container sort.

Notes

  • No behaviour change to existing programs; this is additive.
  • FunctionSchemas costs one HashMap entry per declared table.
  • Adds one rule to the repo's tidy-diff-docs skill: prefer an import over an
    inline full path for a type.
  • Verified: full workspace test suite passes (including the 799-program files
    harness), cargo clippy --tests --workspace clean, cargo fmt --check clean,
    and cargo doc adds no new warnings.

🤖 Generated with Claude Code

Three additions to what a primitive body can see, none of them specific to any
one extension. Together they are what an out-of-tree primitive needs to walk
the term structure under an e-class and build a modified copy of it.

Read::enodes_for_eclass(name, eclass, f) walks a constructor's rows by output
e-class through the backend's lazy column index, instead of scanning the table
and filtering. Cherry-picked from egraphs-good#934 along with the core-relations
ExecutionState::for_each_matching_col and egglog-bridge
TableAction::for_each_output_value it rests on.

Read::table_schema(name) and Read::table_subtype(name) report a table's
declared column sorts and its subtype. EGraph::functions_iter already exposes
this from &EGraph, but a primitive body only ever sees a state wrapper, and
those carried no sort information at all - so a primitive could read rows
without being able to tell an e-class column from a base value. Backed by a
FunctionSchemas map the e-graph shares with the wrappers exactly as it already
shares ActionRegistry, and snapshot/restored across push/pop so a popped table
stops resolving. table_subtype also replaces probing a subtype by starting a
scan and reading the error, which egglog-experimental does today.

Core::rebuild_container(type_id, value, remap) remaps a container value's
contents and interns the result. Out-of-tree code cannot go through
Core::register_container, which requires naming the container's Rust type.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
table_schema returned a schema for either subtype, which reads past the fact
that a constructor's last column is an e-class and a function's is an output.
Splitting it matches how the rest of Read and Write already work - lookup /
eclass_of, constructor_enodes / function_entries, set / add - so it is now
constructor_schema and function_schema, each erroring with WrongSubtype on a
mismatch. table_subtype stays as the error-free predicate to dispatch on when
either subtype is acceptable, which is what retires the subtype probe in
egglog-experimental.

Also applies the tidy-diff-docs skill to the comments this branch adds, and
records a new rule in that skill: import a type rather than naming it by an
inline full path, which is what FunctionSchemas was doing with
crate::util::HashMap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oflatt
oflatt requested review from yihozhang and removed request for saulshanabrook August 6, 2026 21:18
The first cut of this branch added a FunctionSchemas registry so a primitive
body could see column sorts. That was a third copy of data egglog already kept
twice: TypeInfo::func_types holds a FuncType {name, subtype, input, output},
and Function holds the same content again as a ResolvedSchema plus decl.subtype
(and a third time, unresolved, in decl.schema). What was actually missing was
not the data but a way to reach it from a state wrapper.

So there is now one store. TypeInfo::func_types becomes the shared cell -
Arc<RwLock<HashMap<String, Arc<FuncType>>>> - and the state wrappers hold a
handle to it, which is what backs constructor_schema / function_schema /
table_subtype. FunctionSchemas is gone. Function points at the same Arc<FuncType>
rather than storing its own copy, so Function::schema() -> &ResolvedSchema
becomes Function::func_type() -> &FuncType and ResolvedSchema is removed, its
get_by_pos moving to FuncType. declare_function reuses the signature
typechecking already resolved instead of resolving the sorts a second time; it
still resolves and records the functions desugaring generates (global bindings,
proof tables), which never go through typechecking.

Two things this had to get right. TypeInfo::clone deep-copies the map: a clone
is an independent e-graph - a pushed copy, or the parallel typechecking the
proof checker keeps - and declaring a function in one must not make it resolve
in the other. And pop restores the pushed contents into the live cell the
registered primitives already hold, rather than swapping in a cell they have no
handle to.

Also drops the second RwLock acquisition per primitive invocation the first cut
introduced: the wrapper holds the unlocked handle and the schema accessors lock
only when called, so a primitive that never asks pays nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oflatt-claude oflatt-claude changed the title Expose enough e-graph introspection to walk and rebuild a sub-e-graph Store a function's signature once, and expose it to primitives Aug 6, 2026
The additions belong with the name-indexed e-graph access entry they extend,
not as a standalone block longer than anything else on the list. The signature
consolidation keeps a one-line breaking bullet next to the other breaking ones.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codspeed-hq

codspeed-hq Bot commented Aug 6, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 5.88%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 36 untouched benchmarks
⏩ 227 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation tests[stresstest_large_expr] 1.6 s 1.5 s +5.88%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing oflatt-claude:subst-egraph-introspection (53f4fe5) with main (95f539c)

Open in CodSpeed

Footnotes

  1. 227 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Comment thread src/typechecking.rs Outdated
/// [`Read::constructor_schema`] / [`Read::function_schema`]. `EGraph`'s
/// `push`/`pop` snapshot and restore its contents so a popped declaration
/// stops resolving.
pub(crate) type SharedFuncTypes = Arc<RwLock<HashMap<String, Arc<FuncType>>>>;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a bit awkward, having a global like this

oflatt and others added 3 commits August 7, 2026 21:06
… handle

The schema accessors read TypeInfo through an Arc<RwLock<_>> the primitive
captured when it was registered. That lock was never guarding a race: declaring
anything needs &mut EGraph, run_rules already holds that borrow for the whole
execution, and nothing on Core/Read/Write can declare. It was there only
because external functions are stored as Box<dyn ExternalFunction + 'static>,
so a wrapper cannot hold a borrow of the e-graph.

So change the route rather than the synchronization. ExecutionState now carries
an ExternalContext - a borrowed value the caller of an operation makes visible
to the external functions it reaches - and egglog passes &TypeInfo into
run_rules and with_execution_state*. The state wrappers read it back with
downcast_ref, so `constructor_schema` hands out a &FuncType borrowed from the
e-graph itself.

With that, the invariant is enforced by the borrow checker instead of stood in
for by a lock: the &TypeInfo is live for exactly the operation that supplied
it, so the e-graph cannot be mutably borrowed to declare a function while a
rule is running. TypeInfo::func_types goes back to a plain owned map, so
get_func_type keeps its original Option<&FuncType> signature and costs a map
lookup again; TypeInfo::clone goes back to derive; and push/pop restore the map
like every other field, with no cell to reattach.

Rebuild paths pass None: they run generated rebuild rules whose actions
evaluate in the Write context, which has no schema accessors.

The one lock left on the invocation path is the pre-existing ActionRegistry
read in RegistryPrimWrapper::invoke, which this does not touch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GenericFunctionDecl held both an untyped `schema: Schema` and a
`resolved_schema: Head`, which for a resolved decl was a whole owned FuncType
inside a ResolvedCall - a fourth copy of the signature - and for an unresolved
one was a placeholder `String::new()` standing in for "not resolved yet".

It is now `Option<Arc<FuncType>>`: no placeholder, and a typechecked decl
points at the same FuncType TypeInfo holds rather than cloning it.

Worth recording why the field survives at all, since it looks redundant next to
TypeInfo. Proof encoding runs over decls that desugaring generates *after*
typechecking - the functions global bindings lower to - which no TypeInfo knows
about until they are executed. Deleting the field and looking signatures up by
name works for every ordinary declaration and then panics on `$I`. The doc
comment now says so, so the next reader does not have to rediscover it.

Also drops ResolvedCall::view_types, whose only caller was one of the two
readers this touches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two copies were left after the last pass.

GenericFunctionDecl::resolved_schema is gone. I had concluded it was load
bearing, because deleting it and looking signatures up by name panics on `$I`:
proof encoding runs over decls desugaring generates after typechecking, which
no TypeInfo knows about yet. That was the wrong conclusion. The encoder never
needed a FuncType - it needed the column *sorts*, and sorts resolve by name
from TypeInfo like any other. Resolving them from the decl's own untyped schema
drops the field, its ResolvedCall::view_types reader, and the placeholder
`String::new()` an unresolved decl carried in it.

(Populating TypeInfo with the generated signatures instead does not work: the
desugared program is typechecked again afterwards, so recording early trips the
already-bound check.)

ResolvedCall::Func now holds an Arc<FuncType> rather than an owned one, so a
resolved call site shares the signature instead of cloning it - and there are as
many of those as there are calls in every rule. The patterns that matched
`ResolvedCall::Func(FuncType { subtype: Custom, .. })` cannot look through an
Arc, so they match the variant and test the subtype in a guard via a new
ResolvedCall::is_custom_func.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.83820% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.76%. Comparing base (716c320) to head (53f4fe5).
⚠️ Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
src/lib.rs 85.93% 9 Missing ⚠️
egglog-bridge/src/lib.rs 88.13% 7 Missing ⚠️
core-relations/src/action/mod.rs 85.71% 5 Missing ⚠️
src/exec_state.rs 94.80% 4 Missing ⚠️
core-relations/src/hash_index/bench_support.rs 0.00% 1 Missing ⚠️
src/proofs/proof_normal_form.rs 85.71% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #986      +/-   ##
==========================================
+ Coverage   86.59%   86.76%   +0.17%     
==========================================
  Files          95       95              
  Lines       29676    29871     +195     
==========================================
+ Hits        25698    25918     +220     
+ Misses       3978     3953      -25     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Two reviews, one of them independent, converged on the same two defects:

- Deleting `ResolvedCall::view_types` left its `///` block behind, so the
  public `from_resolution_func_types` was documented as a different function.
- `func_type_of` reported `ApiError::MissingTable` for a table that exists
  whenever the execution carried no context, conflating "no declarations here"
  with "no such table". Now `ApiError::SchemasUnavailable`.

Coverage: the codecov report on the PR put `src/exec_state.rs` at 0% for this
patch - none of the new accessors were exercised in-tree, only from
egglog-experimental. `tests/api_introspection.rs` covers them, pinning
`enodes_for_eclass` against a filtered `constructor_enodes` scan (with a
row-count assertion so the comparison cannot pass vacuously), the subtype
split of the schema accessors, and `rebuild_container`'s three outcomes.
`core-relations` covers the context mechanism itself: an external function
reads back what the caller supplied, `None` when nothing was, and `None`
rather than a misparse when the type does not match.

Also from review: `record_signature` now debug-asserts a cached signature
agrees with the declaration it is reused for, `enodes_for_eclass` documents
that it matches the eclass column as stored, `is_custom_func` moved into the
existing inherent impl, the `context` parameters are documented on the entry
points that gained them, and `FuncType::get_by_pos` - carried over from the
deleted `ResolvedSchema` with no callers in the workspace - is dropped.

Doc tidy per the repo's tidy-diff-docs skill: five comments stated mechanism
instead of contract (the lazy-column-index narration on `for_each_matching_col`
and `for_each_output_value`, the rationale paragraph on `ExternalContext`, the
borrow-checker explanation on `type_info`, and "shares rather than copies" on
two `Arc` fields, which the type already says).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@yihozhang yihozhang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! Mostly just low-level nits.

Comment thread src/typechecking.rs Outdated
Comment thread src/lib.rs Outdated
Comment thread core-relations/src/action/mod.rs Outdated
Comment thread core-relations/src/action/mod.rs
Comment thread src/lib.rs Outdated
Comment thread src/lib.rs Outdated
Comment thread src/lib.rs Outdated
Comment thread src/exec_state.rs Outdated
Comment thread src/exec_state.rs Outdated
Comment thread src/exec_state.rs Outdated
From @yihozhang's pass:

- `func_type_arc` is gone; `get_func_type` returns `Option<&Arc<FuncType>>`, so
  the callers that keep a signature clone the `Arc` and the rest are unchanged.
- `record_signature` is inlined into `declare_function`, its only caller.
- `non_stale()` after a scan was compensating for a bug fixed in this release
  (`9369a702 Filter stale rows in constrained bounded scans`). The invariant is
  now written down on `Table::scan_generic_bounded` and the `scan_bounded` /
  `scan_project` wrappers, and the five vestigial filters over scan output are
  gone. The three in `table/mod.rs` stay: those read pending-insert queues,
  where rows really can be stale.
- `for_each_matching_col` now prefers a sort the table already has, then a
  cacheable column's index, then the constraint during the scan - the order
  `Database::process_constraints` uses. It previously built and refreshed an
  index even when `split_fast_slow` had already answered by binary search.
- `EGraph::with_execution_state{,_tracked}` always pass `&TypeInfo`, so no call
  site has to reason about whether a primitive it cannot see needs it.
- `FunctionSubtype::label` and `TableKind::label` replace a free function and
  two inline matches, and `check_subtype` derives both labels instead of taking
  one as a string. `Display for FunctionSubtype` delegates to `label` rather
  than repeating it.
- `rebuild_container` is `map_container`: it maps a container's contents and
  interns the result without touching the original, and "rebuild" collided with
  egglog's canonicalize-after-union sense. It returns `Option<Value>` now, so a
  `TypeId` that does not match the value is reported rather than looking like
  "nothing changed" - `ContainerEnv` already drew that distinction and only the
  outer wrapper discarded it.

New test: a `ReadPrim` that resolves a signature from inside a `:naive` rule,
covering the path that reaches its execution state through `run_rules` rather
than `with_execution_state`. Verified it fails if that call site passes `None`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oflatt
oflatt requested a review from yihozhang August 12, 2026 23:21

@yihozhang yihozhang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! I see it also fixes some other places that use non_stale() while an iter suffices, which is nice.

@oflatt
oflatt merged commit c2c0f15 into egraphs-good:main Aug 13, 2026
46 of 66 checks passed
oflatt-claude pushed a commit to oflatt-claude/egglog-experimental that referenced this pull request Aug 13, 2026
egraphs-good/egglog#986 merged, so the egglog dependencies point back at an
egraphs-good rev rather than the branch they were reviewed on.

The staging limit was a bullet among others, phrased as a mechanism ("a term
built in the same action is not visible"). It is the one thing a caller can get
wrong silently, so it is now a warning up front, phrased as the rule that
follows from it: pass a root the query bound, or one from an earlier command.
A root the action just built has no rows yet, so the walk finds nothing under
it and hands it back unchanged, with no error.

Also states the boundary, since it is not obvious: replacements are exempt. A
map's values are spliced into the copy without being walked, so those can be
built in the same action. `a_replacement_built_in_the_same_action_is_fine`
pins that, so the doc is checked rather than asserted.

Docs tidy over the diff: the alias `Constructor` and `constructors` said the
same thing about per-call resolution, the "globals lower to function tables"
fact appeared twice a few lines apart, and a memo field restated the method it
memoizes. Also drops a `let _ = &mut eg;` left over in a test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants