[Python] CSV reader returns different values in 23.0.0

[rhshadrach]

Describe the bug, including details regarding any error messages, version, and platform.

This is currently causing failures on the pandas CI.

import pyarrow.csv

table = pyarrow.csv.read_csv(io.BytesIO(f"data\n10E-617".encode()))
print(table)
# pyarrow.Table  <-- pyarrow 22.0.0
# data: double
# ----
# data: [[0]]

# pyarrow.Table  <-- pyarrow 23.0.0
# data: string
# ----
# data: [["10E-617"]]

It's not clear to me if this was an intentional change. The only thing that looked possibly related from the release notes was

#47925

but I haven't bisected to find if this was the cause.

Component(s)

Python

[AlenkaF]

@pitrou do you think this issue could be related to the fuzzing work in #47925?

[pitrou]

I doubt it, but I'm not sure. Perhaps someone can bisect and find the commit that introduced the change?

[pitrou]

It's much more likely to be because of #47882 .

[AlenkaF]

Let me bisect and confirm the commit that introduced this.

[AlenkaF]

You are correct @pitrou, it is because of #47882.

git checkout c72572e^

gives

>>> pa.__version__
'22.0.0.dev236+g35bf4884a.d20260128'

>>> table = pyarrow.csv.read_csv(io.BytesIO(f"data\n10E-617".encode()))
>>> print(table)
pyarrow.Table
data: double
----
data: [[0]]

but

git checkout c72572e

results in

>>> pa.__version__
'22.0.0.dev237+gc72572ed5.d20260128'

>>> table = pyarrow.csv.read_csv(io.BytesIO(f"data\n10E-617".encode()))
>>> print(table)
pyarrow.Table
data: string
----
data: [["10E-617"]]

[pitrou]

Ok, so the question is: what should our behavior be here and can we do something about it? I see that Pandas reads such very small number as double:

>>> pd.read_csv(io.BytesIO(f"data\n10E-600\n".encode()))
   data
0   0.0

@jorisvandenbossche @thisisnic

[pitrou]

From a quick experiment, the fast_float library can be massaged to return the desired value:
fastfloat/fast_float#363