fix: nested-JSON "Add to Filters" crashes search page with Lucene SyntaxError

[MikeShi42]

Resolves HDX-4427

Summary

"Add to Filters" on an attribute reached through a parsed JSON string column builds a ClickHouse SQL expression as the filter field — e.g. JSONExtractString(LogAttributes['weird.key.payload'], 'abc.def.jqk/abcd'). filtersToQuery then emitted this expression as the field side of a Lucene term (<expr>:"asdf-14"). The Lucene PEG parser hits the [ after LogAttributes, opens a range expression, and throws when it sees ] where it expects a range body:

SyntaxError: Expected ".", "TO", "\\", [^ \t\r\n\x0C{}()"/\^~[\]], or whitespace but "]" found.

Because SearchQueryBuilder.build() parses the filter condition at render time for every tile/chart, the whole search page crashes.

Why this fix (option c from the issue): A field that is a raw SQL function call cannot be represented as a Lucene field name, and escaping a full SQL expression (parens, brackets, quotes, commas, slashes, spaces) to survive both lucene.parse and the encodeSpecialTokens/decodeSpecialTokens round-trip is fragile. The semantically correct representation is a SQL filter, which flows straight into the WHERE clause and never touches the Lucene parser (renderWhereExpressionStr only invokes SearchQueryBuilder/parse() when language === 'lucene').

filtersToQuery now detects raw SQL-expression keys — those containing parentheses/brackets/quotes/whitespace that cannot appear in a Lucene field name — and emits IN / NOT IN / BETWEEN SQL filters for them. Plain field paths (including bracket-form Map keys, which normalize to dot form) keep using Lucene exactly as before. The emitted SQL shape matches what the app's parseQuery/extractInClauses reads back, so these filters round-trip through the URL.

Changes

• packages/common-utils/src/filters.ts: detect SQL-expression field keys in filtersToQuery and emit sql filters (IN/NOT IN/BETWEEN) with SQL string escaping that inverts the app's getBooleanOrUnquotedString.
• packages/common-utils/src/__tests__/filters.test.ts: emission + escaping + "still Lucene for plain fields" cases.
• packages/common-utils/src/__tests__/nestedJsonAddToFilters.test.ts: regression — the old Lucene string throws via parse(); the new handler emits a SQL filter; any Lucene the handler still emits parses successfully.
• packages/app/src/__tests__/searchFilters.test.ts: full filtersToQuery → parseQuery round-trip for SQL-expression filters (included/excluded, single-quote escaping).

Screenshots or video

N/A — query-generation logic fix, no UI rendering change. The crash lived entirely in the filter-to-query logic, so it is best demonstrated by rendered SQL / parser output. End-to-end evidence (run against the built dist the app imports):

=== 1) Reproduce the ORIGINAL bug ===
  parse() throws -> SyntaxError: Expected ".", "TO", ... but "]" found.
  renderChartConfig THROWS (search-page crash) -> SyntaxError: ... but "]" found.

=== 2) What the FIXED handler now emits ===
  [{"type":"sql","condition":"JSONExtractString(LogAttributes['weird.key.payload'], 'abc.def.jqk/abcd') IN ('asdf-14')"}]

=== 3) renderChartConfig with the NEW sql filter ===
  SELECT * FROM logs WHERE ((JSONExtractString(LogAttributes['weird.key.payload'], 'abc.def.jqk/abcd') IN ('asdf-14')))

e2e_nested_json_filter_fix.log

How to test on Vercel preview

N/A — non-UI change.

References

• Linear Issue:
• Related PRs:

_{To show artifacts inline, enable in settings.}

  

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	Jun 2, 2026 5:28am
hyperdx-storybook	Ready	Preview, Comment	Jun 2, 2026 5:28am

[changeset-bot]

⚠️ No Changeset found

Latest commit: d30f971

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions]

E2E Test Results

✅ All tests passed • 139 passed • 3 skipped • 929s

Status	Count
✅ Passed	139
❌ Failed	0
⚠️ Flaky	6
⏭️ Skipped	3

Tests ran across 3 shards in parallel.

View full report →