Repository navigation
Conversation
…ator `mstch_java` writes a `binary` constant or field default as `"...".getBytes()`, quoting the bytes with `quote_java_string`. That function is meant for UTF-8 text: it copies bytes 0x80-0xf7 into the source unchanged and writes bytes 0xf8-0xff as `\u00XX`. For binary data this goes wrong in two ways: - `const binary B = "\x80\xff";` puts a lone 0x80 byte into the `.java` file, and javac with `-encoding UTF-8` (the encoding the fbthrift Maven builds use) stops with "unmappable character (0x80) for encoding UTF-8". - `const binary B = "\xff\xfe";` compiles, but `getBytes()` encodes the string with the default charset, which is UTF-8 since JDK 18, so the constant holds `c3 bf c3 be` instead of `ff fe`. Add `quote_java_binary`, which escapes every byte from 0x80 up as `\u00XX`, and encode binary literals with `StandardCharsets.ISO_8859_1`, which maps U+0000-U+00FF back to the same bytes regardless of the platform charset. String constants are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mstch_javawrites abinaryconstant or field default as"...".getBytes(). It quotes the bytes withquote_java_string, which is meant for UTF-8 text: it copies bytes 0x80-0xf7 unchanged and writes 0xf8-0xff as\u00XX. For binary data that breaks in two ways:const binary B = "\x80\xff";puts a lone 0x80 byte into the.javafile.javac -encoding UTF-8, the encoding the fbthrift Maven builds use, fails with:const binary B = "\xff\xfe";compiles to"ÿþ".getBytes().getBytes()encodes with the default charset, which is UTF-8 since JDK 18, soBholdsc3 bf c3 beinstead offf fe.This PR:
java::quote_java_binary, which escapes every byte from 0x80 up as\u00XX;quotedBinaryconst-value property;{{value.quotedBinary}}.getBytes(java.nio.charset.StandardCharsets.ISO_8859_1). ISO-8859-1 maps U+0000-U+00FF back to the same bytes on any JVM.String constants are unchanged. b0c1341 made the same kind of change for Rust binary constants.
Before and after, for
const binary HIGH_BYTES = "\xff\xfe";(JDK 27):Fixes #723
Test plan
TEST_F(JavaUtilTest, test_quote_java_binary)inthrift/compiler/test/java_util_test.cc.non_ascii_bytesconstant in thelazy_constantsfixture (Java only).complex-structDefaultStruct.java,constantsModuleConstants.javaandlazy_constantsConstants.java.--target thrift,-DTHRIFT_RPC=OFF) with GCC 13.3 on x86_64 Linux.fixture_test.pyforlazy_constants,complex-structandconstants, run through a small shim for the Buck-only imports:main's compiler,lazy_constants/java,complex-struct/javaandconstants/javafail;constants/py3andconstants/py3_inplacefail the same way on both builds outside Buck.java_util_test.cc, compiled by hand against GoogleTest 1.17.0 withgenerate/java/util.cc: 5/5 pass.byte[]constants for"\x80\xff"and"\xff\xfe", compiled withjavac -encoding UTF-8(JDK 27, macOS arm64) and run: before and after as shown above.main@ 429c34f. The branch is rebased ontomain@ c6bb72f; none of the touched files changed upstream in between, so they were not re-run.Not changed here:
java_deprecated/androidhave the same.getBytes()pattern and also write\xNN, which javac rejects ("illegal escape character", e.g.constants/out/android/.../Constants.java:52). That could be a separate change.This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.