Skip to content

ByteArrayDecoderPlain::read in the parquet crate can silently fail #11221

Description

@etseidl

Describe the bug

ByteArrayDecoderPlain::read can read fewer values than requested if its input buffer is exhausted early on an exact record boundary. The relevant (edited for brevity) code is

        let mut read = 0;
        let buf = self.buf.as_ref();
        while self.offset < self.buf.len() && read != to_read {
            if self.offset + 4 > buf.len() {
                return Err(ParquetError::EOF("eof decoding byte array".into()));
            }
            ...
            read += 1;
        }
        self.max_remaining_values -= to_read;
        Ok(to_read)

Within the loop a check is made for a premature EOF, but if the buffer ends on an exact boundary, the loop will terminate, but we still decrement by and return to_read rather than read.

To Reproduce

A unit test which fails on main:

    #[test]
    fn test_plain_decoder_reports_values_actually_read() {
        // The page claims to contain two values, but its buffer contains only
        // one complete PLAIN-encoded BYTE_ARRAY value.
        let buffer = Bytes::from_static(&[3, 0, 0, 0, b'f', b'o', b'o']);
        let mut decoder = ByteArrayDecoderPlain::new(buffer, 2, Some(2), false);
        let mut output = OffsetBuffer::<i32>::with_capacity(2);

        assert_eq!(decoder.read(&mut output, 2).unwrap(), 1);
        assert_eq!(output.values.as_slice(), b"foo");
        assert_eq!(output.offsets.as_slice(), &[0, 3]);
        assert_eq!(decoder.max_remaining_values, 1);
    }

Expected behavior

read should either report the actual number of values read, or return the same EOF error in this scenario.

Additional context

This was discovered by Codex during a review of #10420.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions