perf: optimize struct_to_json (10% faster)

[andygrove]

Which issue does this PR close?

N/A

Rationale for this change

Optimize struct_to_json on the hot per-row path.

What changes are included in this PR?

• Pre-render "field_name": once per column so each field costs a single copy rather than one push per punctuation character.
• Classify each field up front (Quoted for Utf8/LargeUtf8/Utf8View, Raw when the rendering can never be NaN/Infinity, MaybeQuoted otherwise) so the per-value is_infinity/is_nan check is skipped on types like ints, decimals, and timestamps.
• Write directly into the StringBuilder (Write impl) via a small append helper instead of staging each row in a temporary String.
• Right-size the builder's value buffer from the rendered column offsets (not the underlying value_data buffer) so sliced inputs do not over-allocate.
• Fix: Utf8View fields were falling through to the unquoted branch (pre-existing bug); they are now quoted and escaped like Utf8/LargeUtf8.
• Tests: float NaN/Infinity, string escaping (", \n, \t, \\), non-ASCII passthrough, ignore_null_fields=false, and Utf8View field quoting.

Follow-up: adding a to_json criterion bench (there isn't one in-tree today is tracked separately.

How are these changes tested?

Existing tests plus five new unit tests listed above.
EOF
)

[mbutrovich]

The PR description reports to_json: 9.92% faster from a criterion to_json bench, but no such benchmark exists in the tree and the PR adds none. native/spark-expr/benches/ has 20 benches and not one covers to_json or struct_to_json, and gh pr view 4927 --json files lists only native/spark-expr/src/json_funcs/to_json.rs. There is nothing to reproduce the number and nothing to gate against a future regression.

• Evidence: PR file list is a single source file; grep -rln "to_json" native/spark-expr/benches/ returns nothing.
• Suggested change: add native/spark-expr/benches/to_json.rs (wired via a [[bench]] entry in spark-expr/Cargo.toml) that builds a StructArray with a mix of the shapes that exercise each FieldStyle (Utf8 needing escaping, Int/Bool/Decimal/Timestamp Raw, Float64 with a NaN/Infinity row, a nested struct, an all-null row, and ignore_null_fields both true and false), then run it before and after to confirm the delta and lock in the gate.

[mbutrovich]

The description says the PR replaces "the per-value byte scan" with "a single vectorizable column-wide escape scan." The diff does no such thing. String escaping is still per value inside the row loop via escape_string(string_value) in the FieldStyle::Quoted arm. What the PR actually removed is the per-value is_infinity/is_nan call for Raw types via never_renders_special_float, not the escape scan.

• Evidence: to_json.rs FieldStyle::Quoted arm calls escape_string(string_value) once per row; there is no precomputed per-column escaped buffer anywhere in the diff.
• Suggested change: reword the description to describe what actually changed (prefix precompute, direct-to-builder writes, right-sized buffer, and skipping the special-float check for types that cannot render NaN/Infinity). Do not claim a vectorized escape pass that is not present.

[mbutrovich]

FieldStyle maps only Utf8/LargeUtf8 to Quoted. Utf8View falls through to MaybeQuoted, so a string-view field is emitted as a bare token instead of a quoted, escaped JSON string, producing invalid JSON. This matches the old is_string predicate, so the PR does not regress, but the rewrite is where the type dispatch is being consolidated and it is worth closing.

• Evidence: to_json.rs style match arms only list DataType::Utf8 | DataType::LargeUtf8 => FieldStyle::Quoted; old is_string likewise omitted Utf8View.
• Suggested change: include DataType::Utf8View in the Quoted arm (arrow-schema already groups these three in DataType::is_string), and add a unit test with a Utf8View field containing a quote to confirm it is quoted and escaped. If deliberately out of scope, note it explicitly rather than carrying it silently through the consolidation.

[mbutrovich]

The Rust tests cover Bool/Int32/Utf8, nested struct, and null handling with ignore_null_fields=true. They do not cover the branches this PR restructures: MaybeQuoted floats hitting NaN/Infinity (the reason MaybeQuoted exists), Raw decimals and timestamps, escaping inside a Quoted value, unicode, an empty struct, or ignore_null_fields=false. The behavior is preserved, but the safety net for this specific refactor is thin.

• Evidence: to_json.rs mod test has test_primitives and test_nested_struct, both calling struct_to_json(..., true); no float/NaN, decimal, timestamp, escaping, unicode, empty-struct, or ignore_null_fields=false case.
• Suggested change: add unit tests for (a) a Float64 column with rows 1.5, NaN, Infinity, -Infinity asserting the special values are quoted and the finite value is raw, (b) a Utf8 value containing " and \n asserting escaping, (c) ignore_null_fields=false asserting null fields serialize as "name":null, and (d) an empty struct asserting {}.

[mbutrovich]

data_capacity uses values.value_data().len(), which is the full backing buffer of the cast StringArray, not the length of the offsets actually referenced. For a sliced or offset array this over-allocates the output buffer. It is only a hint so there is no correctness impact, and it never under-allocates, but it can waste memory on sliced inputs.

• Evidence: to_json.rs capacity sum uses values.value_data().len(); arrow-rs value_data() returns the entire value_data slice (byte_array.rs:283-285), independent of the array's offset window.
• Suggested change: if precision matters, compute the used span from the offsets (last_offset - first_offset) instead of value_data().len(). Given it is a hint, an inline comment noting the deliberate over-approximation is the lighter-weight alternative.

[andygrove]

Thanks for the careful read @mbutrovich. Addressed in 2a3c5b5:

• Rewrote the PR description so it matches the actual diff: no criterion bench in-tree (tracked as a follow-up), no column-wide escape scan, and I called out what the change actually does (pre-rendered prefixes, style classification, direct StringBuilder writes, right-sized buffer).
• Fixed the Utf8View pre-existing hole: string-view fields now go through the Quoted branch and are emitted as quoted, escaped JSON strings.
• Sized data_capacity from value_offsets first/last instead of value_data().len() so sliced inputs no longer over-allocate.
• Added five unit tests covering the branches you called out: float NaN/Infinity, string escaping, non-ASCII passthrough, ignore_null_fields=false, and Utf8View fields.