remap_expr_references doubles as both an index remapper and a validation pass — it rejects unsupported filter operators (WindowFunction, Subquery, and now Lambda/LambdaInvocation):
https://github.com/lance-format/lance/blob/main/rust/lance-datafusion/src/substrait.rs#L156
But it is only invoked when remove_extension_types actually drops one or more schema fields (the length-changed guard):
https://github.com/lance-format/lance/blob/main/rust/lance-datafusion/src/substrait.rs#L304-L316
So a Substrait filter whose base schema needs no field pruning skips validation entirely, and an unsupported expression (window function, subquery, lambda) reaches the DataFusion consumer instead of being rejected up front with our descriptive error.
Expected: run the rejection checks unconditionally, independent of whether remapping is needed. Suggested approach: split validation from index remapping — a validate_expr traversal that always runs, plus the existing remap that runs only when the mapping is non-trivial.
Add a regression test with an unpruned base schema containing an unsupported operator, asserting it errors.
Note: pre-existing behavior (the WindowFunction/Subquery arms have always had this gap); the Lambda arm was added in #7793 for the substrait 0.63 bump, which is what surfaced it via CodeRabbit review.
remap_expr_referencesdoubles as both an index remapper and a validation pass — it rejects unsupported filter operators (WindowFunction,Subquery, and nowLambda/LambdaInvocation):https://github.com/lance-format/lance/blob/main/rust/lance-datafusion/src/substrait.rs#L156
But it is only invoked when
remove_extension_typesactually drops one or more schema fields (the length-changed guard):https://github.com/lance-format/lance/blob/main/rust/lance-datafusion/src/substrait.rs#L304-L316
So a Substrait filter whose base schema needs no field pruning skips validation entirely, and an unsupported expression (window function, subquery, lambda) reaches the DataFusion consumer instead of being rejected up front with our descriptive error.
Expected: run the rejection checks unconditionally, independent of whether remapping is needed. Suggested approach: split validation from index remapping — a
validate_exprtraversal that always runs, plus the existing remap that runs only when the mapping is non-trivial.Add a regression test with an unpruned base schema containing an unsupported operator, asserting it errors.
Note: pre-existing behavior (the
WindowFunction/Subqueryarms have always had this gap); theLambdaarm was added in #7793 for the substrait 0.63 bump, which is what surfaced it via CodeRabbit review.