Skip to content

fix(datagen): use an aligned buffer when generating random bytes arrays#7919

Open
SAY-5 wants to merge 1 commit into
lance-format:mainfrom
SAY-5:fix-datagen-bytes-alignment
Open

fix(datagen): use an aligned buffer when generating random bytes arrays#7919
SAY-5 wants to merge 1 commit into
lance-format:mainfrom
SAY-5:fix-datagen-bytes-alignment

Conversation

@SAY-5

@SAY-5 SAY-5 commented Jul 22, 2026

Copy link
Copy Markdown

RandomBytesGenerator fills a Vec<u8> and reinterprets it as T::Native through ScalarBuffer::new, but a Vec<u8> is only byte-aligned, so generating zero Float16/Decimal128/Decimal256 rows panics on the dangling pointer (#7911). Switching to MutableBuffer::from_len_zeroed gives an allocation that is aligned for any scalar type, for empty and non-empty arrays alike.

Closes #7911

Signed-off-by: Sai Asish Y <say.apm35@gmail.com>
@github-actions github-actions Bot added the bug Something isn't working label Jul 22, 2026
@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

RandomBytesGenerator now allocates aligned memory through MutableBuffer, fills it with random bytes, and wraps it for primitive arrays. A regression test covers zero-row and three-row Float16, Decimal128, and Decimal256 generation.

Changes

Random byte generation

Layer / File(s) Summary
Aligned generation and regression coverage
rust/lance-datagen/src/generator.rs
RandomBytesGenerator uses aligned zeroed storage and adds length assertions for empty and non-empty Float16 and decimal arrays.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: xuanwo

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the main change: switching random byte generation to an aligned buffer for datagen arrays.
Description check ✅ Passed The description directly explains the alignment fix and the zero-row panic it addresses.
Linked Issues check ✅ Passed The PR fixes #7911 by using aligned allocation and adds coverage for empty Float16 and decimal arrays.
Out of Scope Changes check ✅ Passed The changes stay focused on the alignment bug fix and its regression test, with no clear unrelated additions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
rust/lance-datagen/src/generator.rs (1)

3532-3548: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Exercise typed value access in the regression test.

The test currently verifies only Array::len(). Add assertions for the expected data_type() and concrete primitive array downcasts, then access the non-empty values so the test proves alignment remains valid beyond construction.

As per coding guidelines, every bugfix and feature must have corresponding tests.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-datagen/src/generator.rs` around lines 3532 - 3548, Extend
test_rand_bytes_zero_rows to validate each generated array’s expected data_type
and downcast it to the corresponding concrete primitive array type, then access
the values from the three-row result. Retain the existing zero- and
non-empty-length assertions so the test covers both construction and typed value
access for Float16, Decimal128, and Decimal256.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@rust/lance-datagen/src/generator.rs`:
- Around line 3532-3548: Extend test_rand_bytes_zero_rows to validate each
generated array’s expected data_type and downcast it to the corresponding
concrete primitive array type, then access the values from the three-row result.
Retain the existing zero- and non-empty-length assertions so the test covers
both construction and typed value access for Float16, Decimal128, and
Decimal256.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 0782fe02-1d46-4d35-9166-fb880eb1264a

📥 Commits

Reviewing files that changed from the base of the PR and between be778df and 04f44e3.

📒 Files selected for processing (1)
  • rust/lance-datagen/src/generator.rs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lance-datagen panics when generating empty Float16 or decimal arrays

1 participant