feat: Add Spark-compatible encode function to datafusion-spark

[JeelRajodiya]

Rationale

The datafusion-spark crate is missing the encode function. Spark's encode(expr, charset) converts a string or binary value into binary using a specified character encoding — commonly used in Spark SQL workloads and needed by engines built on DataFusion that target Spark compatibility.

What changes are included in this PR?

Adds SparkEncode to datafusion-spark's string functions, emulating Spark 3.5 semantics. It supports US-ASCII, ISO-8859-1, UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, and UTF-32LE, including common aliases (UTF8, LATIN1, etc.) and case-insensitive matching. The charset can be a constant or a per-row column. Binary input is decoded as lossy UTF-8 (invalid bytes → U+FFFD) before re-encoding, and unmappable characters are silently replaced with ?, matching Spark.

Are these changes tested?

Yes. Coverage lives in encode.slt (sqllogictest) and exercises all charsets and aliases, case-insensitive matching, null value/charset handling, per-row charsets, binary input (Binary/LargeBinary/BinaryView) with lossy UTF-8, Utf8View input, and the unsupported-charset error. A Rust unit test covers return-field nullability.

Are there any user-facing changes?

New encode scalar function available when using datafusion-spark.

[Zeel-e6x]

run benchmarks

[adriangbot]

Hi @Zeel-e6x, thanks for the request (#21331 (comment)). Only whitelisted users can trigger benchmarks. Allowed users: Dandandan, Fokko, Jefffrey, Omega359, adriangb, alamb, asubiotto, brunal, buraksenn, cetra3, codephage2020, comphead, erenavsarogullari, etseidl, friendlymatthew, gabotechs, geoffreyclaude, grtlr, haohuaijin, jonathanc-n, kevinjqliu, klion26, kosiew, kumarUjjawal, kunalsinghdadhwal, liamzwbao, mbutrovich, mzabaluev, neilconway, rluvaton, sdf-jkl, timsaucer, xudong963, zhuqi-lucas.

---

File an issue against this benchmark runner

[xanderbailey]

Looks good to me but you’ll need a committer to Approve also! Thanks for the PR!

[JeelRajodiya]

Hey @xanderbailey, Do I need to mention the maintainers for review? if yes please suggest whom incase you know.
I'm planning to open more PRs for implementing other functions but I'm waiting for this PR to get merged.

[xanderbailey]

They will normally pick it up within a week or so. If not we can ping them here.

[alamb]

Thanks @xanderbailey and @JeelRajodiya -- the PR load is pretty intense! I started the CI for this PR

[JeelRajodiya]

I pushed the fixes for clippy errors. @alamb Can you rerun the checks please?

[JeelRajodiya]

Please rerun the checks

[andygrove]

Spark also supports UTF-32. It would be worth adding a comment here explaining why this isn't or can't be supported.

  arguments = """
    Arguments:
      * str - a string expression
      * charset - one of the charsets 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16', 'UTF-32' to encode `str` into a BINARY. It is case insensitive.
  """,

[JeelRajodiya]

I missed adding support for UTF-32, I've added it now with respective tests.

[andygrove]

Thanks for iterating on this @JeelRajodiya. One issue I noticed:

spark-sql> SELECT hex(encode('A', 'UTF-32'));                                                                                                                                                                        
00000041                                                                 

This PR returns 0000FEFF00000041, with a BOM. Both Spark 3.5 and Spark 4.1 return the four-byte form.

Once challenge for this PR is that there is different behavior across Spark versions for the encode expression. Which Spark version is this PR targeting? It would be good to document that.

[JeelRajodiya]

Hey @andygrove, I realized that I shouldn't be using enable_ansi_mode flag inside encode function. In the spark definition they are not binding the ansi mode to encode function.

Moreover we should target Spark 3.5 which is more permissive and doesn't return errors when null inputs are passed. it simply replaces it with ?. But I've added a TODO in the doc comment pointing at the two real Spark 4.1 configs so a follow-up PR can wire them properly.

Below are the references to the spark definitions
Spark 3.5's Encode.scala:

protected override def nullSafeEval(input1: Any, input2: Any): Any = {
  input1.asInstanceOf[UTF8String].toString.getBytes(toCharset)
}

Just calls Java's String.getBytes, which replaces unmappable chars with the charset's default byte (?). No legacyErrorAction, no config, no exception.

Spark 4.1's Encode.scala added two new configs for the strict behavior:

case class Encode(str, charset, legacyCharsets: Boolean, legacyErrorAction: Boolean)
  def this(value, charset) =
    this(value, charset, SQLConf.get.legacyJavaCharsets, SQLConf.get.legacyCodingErrorAction)

Setting legacyErrorAction=true restores the Spark 3.5 ? behavior.

These spark.sql.legacy.javaCharsets and spark.sql.legacy.codingErrorAction are supported in 4.1 version. which can be left for future PR. Currently the PR targets Spark 3.5. I've added mentioned in the doc comment as well.

Let me know if we need to iterate on this further.

P.S I've fixed the BOM issue

[JeelRajodiya]

@andygrove Can you reivew the PR?