[Parquet][Python][Dataset] dataset(...).to_table() sometimes returns a truncated table instead of raising when one listed Parquet fragment cannot be opened (freshly written files)

[PepijnBrasseur]

Describe the bug, including details regarding any error messages, version, and platform.

Summary

A pyarrow.dataset built from an explicit list of Parquet files, one of which is not a valid Parquet file, is expected to raise on to_table(). It usually does. Occasionally it instead returns a table containing only the rows of the file(s) that sort before the invalid fragment, with no exception. It happens only when the files were written immediately before the scan (a fresh directory per iteration); reusing the same files never reproduced it. Seen on Windows and on Linux, threaded and unthreaded.

Environment

	Windows	Linux
pyarrow	23.0.1	23.0.1
Python	3.11.1	3.11.15
OS	Windows 11, build 26200 (platform.platform() → Windows-10-10.0.26200-SP0)	Ubuntu 24.04.4 LTS (x86_64; a cloud workspace, the same class on every run)
filesystem	NTFS (local disk, the system temp directory)	ext4 on a virtual disk, the system temp directory

What was checked

• The file list is complete when the scan is short. On a truncated read, the listed paths included the invalid file and the dataset held all 5 fragments (fragment count = number of listed files) — the scan stopped early and returned without error. A fragment-count check before the scan therefore cannot detect it.
• A 0-byte invalid file in the same position raises too (in the same test shape) — the truncation is not an empty file being skipped.
• Threading is not the variable: it reproduces with use_threads=False on both platforms.
• Freshness matters: 0 truncations in 3,600 reads of reused files, against the fresh-file rates above. Why fresh files matter is not understood.
• The shape of the failing fragment matters: the 25-byte non-Parquet file truncates; a half-written Parquet file raised every time in 6,000 tries. Not understood.

What remains as candidates: the dataset scanner's handling of a fragment-open error, or something in the filesystem beneath it for files closed moments earlier — the two cannot be told apart from here.

Workaround in use

Opening every listed file's footer independently before the scan (pyarrow.parquet.ParquetFile(path).metadata) turns the silent truncation into a raise (0 truncations in 2,000 tries on the Windows reproduction); comparing the rows read with the rows the footers declare backstops the case where a footer opens but its data does not.

Component(s)

Python, Parquet

[kita-renji]

I can reproduce this on Linux with pyarrow 23.0.1 and 25.0.1, and I think I've found the cause. It's a race in MergedGenerator, not the filesystem.

On a plain LocalFileSystem I couldn't hit it in ~15k iterations. Routing the same files through a pyarrow.fs.PyFileSystem wrapper (which adds thread handoffs) makes it show up at roughly 1-3% with 8 files and one 25-byte bad file. That's with both use_threads=False and True, and even with fragment_readahead=1.

The root cause is in cpp/src/arrow/util/async_generator.h. When an inner subscription (here: the fragment whose GetReaderAsync fails) delivers an error while no caller is waiting, InnerCallback sets state->broken = true under the mutex, releases it, and only then stores the error in MarkFinalError() (final_error = err). If the consumer calls operator() in that window, it sees broken with final_error still OK and gets a plain end-of-stream. The source node finishes normally, the plan completes OK, and ToTable() returns whatever batches were already delivered. OuterCallback has the same pattern.

This also fits the "shape" observation. The unlocked path is only taken when nobody is waiting, i.e. while the consumer is still processing the previous batch. A 25-byte file fails its footer check almost immediately after that batch is handed off, so the two collide. A larger half-written file fails later, when the consumer is usually already waiting, and that path delivers the error safely.

A standalone C++ loop hits it naturally about 1 in 200k runs. With a 200µs sleep injected before the final_error store it goes to 1990/2000. For pyarrow, I rebuilt libarrow_dataset.so from apache-arrow-25.0.1 and swapped it into the 25.0.1 wheel: 57/4500 truncated with the unmodified source, 0/11.6k with the fix.

The fix is small: store final_error while still holding the lock (and only defer the all_finished callback for the case where a caller is already waiting). I'm happy to open a PR with it and a regression test if that sounds right.

Until then, the workaround from the issue (opening each footer with pq.ParquetFile(path).metadata before the scan) is a reasonable guard.

[PepijnBrasseur]

Thank you — this matches what we see. Our Linux rate on a plain local filesystem was ~1 in 16,000 across three runs, and higher on Windows, which fits the extra hand-offs explanation; the "small file fails early, half-written file fails late" shape is what our runs recorded too. A PR with a regression test would be very welcome from our side. We are keeping an application-side row-count check against the footer in the meantime.

[kita-renji]

@PepijnBrasseur thanks for confirming. The PR is up in #51498. It stores the error while the lock is still held and adds two C++ regression tests, one for inner (fragment) errors and one for outer errors. The tests use a small test-only hook to pull at the exact point where the error used to get lost, so they fail every time without the fix. Until it's released, your row-count check against the footer is a good safeguard.