Skip to content

Fix: Keep the bytes of non-ASCII binary constants in the Java generator - #725

Open
0z5a wants to merge 1 commit into
meta:mainfrom
0z5a:fix/java-binary-const-non-ascii
Open

0z5a wants to merge 1 commit into
meta:mainfrom
0z5a:fix/java-binary-const-non-ascii

Conversation

@0z5a

@0z5a 0z5a commented Nov 13, 2026

Copy link
Copy Markdown

Summary

mstch_java writes a binary constant or field default as "...".getBytes(). It quotes the bytes with quote_java_string, which is meant for UTF-8 text: it copies bytes 0x80-0xf7 unchanged and writes 0xf8-0xff as \u00XX. For binary data that breaks in two ways:

  • const binary B = "\x80\xff"; puts a lone 0x80 byte into the .java file. javac -encoding UTF-8, the encoding the fbthrift Maven builds use, fails with:
    error: unmappable character (0x80) for encoding UTF-8
    
  • const binary B = "\xff\xfe"; compiles to "ÿþ".getBytes(). getBytes() encodes with the default charset, which is UTF-8 since JDK 18, so B holds c3 bf c3 be instead of ff fe.

This PR:

  • adds java::quote_java_binary, which escapes every byte from 0x80 up as \u00XX;
  • adds a quotedBinary const-value property;
  • renders binary constants as {{value.quotedBinary}}.getBytes(java.nio.charset.StandardCharsets.ISO_8859_1). ISO-8859-1 maps U+0000-U+00FF back to the same bytes on any JVM.

String constants are unchanged. b0c1341 made the same kind of change for Rust binary constants.

Before and after, for const binary HIGH_BYTES = "\xff\xfe"; (JDK 27):

before: "ÿþ".getBytes()                                              -> c3 bf c3 be
after:  "ÿþ".getBytes(java.nio.charset.StandardCharsets.ISO_8859_1) -> ff fe

Fixes #723

Test plan

  • New TEST_F(JavaUtilTest, test_quote_java_binary) in thrift/compiler/test/java_util_test.cc.
  • New non_ascii_bytes constant in the lazy_constants fixture (Java only).
  • Fixtures regenerated with the fixed compiler. Only three Java files change: complex-struct DefaultStruct.java, constants ModuleConstants.java and lazy_constants Constants.java.
  • What was run:
    • The compiler was built with CMake (--target thrift, -DTHRIFT_RPC=OFF) with GCC 13.3 on x86_64 Linux.
    • fixture_test.py for lazy_constants, complex-struct and constants, run through a small shim for the Buck-only imports:
      • with main's compiler, lazy_constants/java, complex-struct/java and constants/java fail;
      • with this change they pass.
      • constants/py3 and constants/py3_inplace fail the same way on both builds outside Buck.
    • java_util_test.cc, compiled by hand against GoogleTest 1.17.0 with generate/java/util.cc: 5/5 pass.
    • The generated byte[] constants for "\x80\xff" and "\xff\xfe", compiled with javac -encoding UTF-8 (JDK 27, macOS arm64) and run: before and after as shown above.
  • Not run: Buck targets; the Java runtime or Maven builds; javac on the full generated fixture classes.
  • The runs above used main @ 429c34f. The branch is rebased onto main @ c6bb72f; none of the touched files changed upstream in between, so they were not re-run.

Not changed here: java_deprecated/android have the same .getBytes() pattern and also write \xNN, which javac rejects ("illegal escape character", e.g. constants/out/android/.../Constants.java:52). That could be a separate change.

This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.

…ator

`mstch_java` writes a `binary` constant or field default as
`"...".getBytes()`, quoting the bytes with `quote_java_string`. That
function is meant for UTF-8 text: it copies bytes 0x80-0xf7 into the
source unchanged and writes bytes 0xf8-0xff as `\u00XX`. For binary data
this goes wrong in two ways:

- `const binary B = "\x80\xff";` puts a lone 0x80 byte into the `.java`
  file, and javac with `-encoding UTF-8` (the encoding the fbthrift Maven
  builds use) stops with "unmappable character (0x80) for encoding UTF-8".
- `const binary B = "\xff\xfe";` compiles, but `getBytes()` encodes the
  string with the default charset, which is UTF-8 since JDK 18, so the
  constant holds `c3 bf c3 be` instead of `ff fe`.

Add `quote_java_binary`, which escapes every byte from 0x80 up as
`\u00XX`, and encode binary literals with
`StandardCharsets.ISO_8859_1`, which maps U+0000-U+00FF back to the
same bytes regardless of the platform charset. String constants are
unchanged.
@meta-cla meta-cla Bot added the CLA Signed label Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mstch_java: binary constants with bytes >= 0x80 do not compile or hold the wrong bytes

1 participant