Repository navigation
[IcebergIO] Build read rows with attachValues - #40998
ahmedabu98 wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
📋 Review Summary
This PR optimizes Beam Row construction in IcebergUtils.icebergRecordToBeamRow and CdcOutputUtils.outputRow by replacing addValues(...) with attachValues(...) to avoid redundant validation and copying. However, attachValues(...) skips RowUtils.verify type conversion and nullability checks, which introduces critical runtime bugs in IcebergUtils.icebergRecordToBeamRow for non-nullable fields with null values, ITERABLE fields, and DATETIME fields.
🔍 General Feedback
Row.Builder.attachValuesvs.addValuesinvariants:RowUtils.verify(called byaddValues/build, but bypassed byattachValues) does more than just validation: it also normalizes values to the internal Java representations expected byRowgetters andRowCoder(e.g., convertingReadableDateTime/DateTimetoorg.joda.time.Instant, and allowingIterableinstead of requiringListforITERABLEschema fields).- Nullability enforcement:
icebergRecordValueToBeamFieldValueimmediately returnsnullwhen an Iceberg value isnullwithout checking!field.getType().getNullable(). Previously,Row.Builder.build()caught this and threw anIllegalArgumentException; withattachValues,nullis silently stored in non-nullable fields. - The change in
CdcOutputUtils.outputRowis safe because all values originate from an already-normalizedRowor match the exact non-nullableSTRING/INT64metadata types.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
Assigning reviewers: R: @chamikaramj for label iceberg. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40998 +/- ##
===========================================
- Coverage 79.83% 0 -79.84%
===========================================
Files 494 0 -494
Lines 83575 0 -83575
===========================================
- Hits 66722 0 -66722
+ Misses 16853 0 -16853
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Iceberg reads build every Row with
Row.Builder.build(). Its per-field checks are rather expensive, costing more than the rest of the conversion put together.This switches to
attachValues, which is howRowCoderbuilds rows when it decodes. Two values thatbuild()used to fix up are now converted directly: DATETIME to a jodaInstant, andbinaryinside lists and maps tobyte[].