Skip to content

Expression simplification of col ~ '.*' to col IS NOT NULL drops NULL semantics #24379

Description

@viirya

Describe the bug

Expression simplification rewrites col ~ '.*' to col IS NOT NULL, which loses SQL NULL semantics. For a NULL input, NULL ~ '.*' is NULL (three-valued logic), but NULL IS NOT NULL is false. So in a projection context the rewrite returns false where the correct answer is NULL.

The !~ (RegexNotMatch) branch of the same rule is already NULL-aware (col IS NULL AND NULL); only the ~ (RegexMatch) branch drops the NULL. Same class as #24246, but a different rule (regex .*).

To Reproduce

SELECT s, s ~ '.*' AS m
FROM (VALUES (CAST(NULL AS VARCHAR)), ('x')) t(s);

Actual:

+---+-------+
| s | m     |
+---+-------+
|   | false |   <- wrong, should be NULL
| x | true  |
+---+-------+

Expected (what NULL semantics require):

+---+------+
| s | m    |
+---+------+
|   | NULL |
| x | true |
+---+------+

Expected behavior

col ~ '.*' is true for a non-NULL string and NULL for a NULL input, matching NestedLoopJoin-free evaluation.

Additional context

Root cause: simplify_regex_expr in datafusion/optimizer/src/simplify_expressions/regex.rs rewrites the ~ '.*' case to left.is_not_null(). In a WHERE filter this is fine (both FALSE and NULL reject the row, which is why existing SLT filter tests didn't catch it), but it is wrong in a general/projection context. Fix + regression test coming.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions