BUG: pd.to_datetime behaves strange (unexpected change to ns precision)

[behrenhoff]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

#!/usr/bin/env -S uv run --script
#
# /// script
# requires-python = ">=3.14"
# dependencies = [
#     # "pandas<3",
#     "pandas==3.0.0rc0",
# ]
# ///
import datetime

import pandas as pd

print("Pandas version:", pd.__version__)

df = pd.DataFrame(
    {
        "A": pd.date_range("20230101", periods=3),
        "days_to_add": [1, 2, 3],
    }
)
print(df)
print(df.dtypes)

print("adding days via datetime.timedelta")
df_via_datetime = df.copy()
df_via_datetime["A"] = df["A"] + df["days_to_add"].apply(
    lambda d: datetime.timedelta(days=d)
)


print(df_via_datetime)
print(df_via_datetime.dtypes)

print("adding days via pd.Timedelta")
df_via_timedelta = df.copy()
df_via_timedelta["A"] = df["A"] + pd.to_timedelta(df["days_to_add"], unit="d")

print(df_via_timedelta)
print(df_via_timedelta.dtypes)

pd.testing.assert_frame_equal(df_via_datetime, df_via_timedelta)
print("OK, both variants work the same!")

Issue Description

Trying out Pandas 3.0rc0, I get a zillion of failed tests in my test suite. Some of them expected (str / object) but most of the unexpected ones lead to the datetime change from ns to us.

Consider the script above. Why does pd.to_timedelta use ns precision? When you run the script with pandas 3rc0, the assert fails as the dtypes are different. When run it with pandas<3 instead, it passes.

With this change, suddenly pd.to_timedelta and datetime.timedelta behave differently. As a user, I find this very confusing. Why does it change to nanoseconds when I just have day granularity?

This said, in my code I am only working with daily data, so I don't really care about sub-second precision. However I would like to be able to use pd.testing.assert_frame_equal and the like and get predictable results. I also don't want to blindly ignore dtypes as float vs int can be very relevant for me.

Furthermore I have some code that works with pure integers (dates). Previously dtseries.to_numpy() did the job as all was in ns. Now I somehow need to normalize the values. How do you do that? Solutions I came up is

factor = 1000 if dtseries.dtype == '<M8[us]' else 1
call_external_func(dtseries.astype("int64").to_numpy() * factor)

But this assumes only the two us and ns exist.

So basically I am asking for ways to work with these kinds of problems. How do I set/change the underlying representation?

This is certainly related to #63270 (just from a few hours ago)
I thought I put it here anyway as this causes many date tests to fail and I fear it is not only me.

Expected Behavior

Clearer reasons when/why precision changes.

Installed Versions

doesnt matter

[rhshadrach]

https://pandas.pydata.org/pandas-docs/dev/whatsnew/v3.0.0.html#datetime-resolution-inference

One solution would be in your code to convert to nanosecond resolution after using such methods. That would mean no updates to your test suite.

[behrenhoff]

Sorry, I don't get it. I've read the paragraph multiples times, yet still don't know how I could fix the script above. Why should I even need to convert to nanoseconds when only working with daily data? It should be perfectly fine to work with us only (us lets us go farther in the future, so that sounds good to me). And I don't see anything in the code that should cause a conversion to ns IMHO. The page also doesn't mention anything about pd.to_timedelta doing the change to ns.

Contrary: it reads as if it would now default to us. Which many functions do. But pandas.timedeltas doesn't do it but datetime.timedelta does?

Maybe forget about the second comment about wanting to convert to int64. I can fix that, not a problem. The main problem is unexpected changes to ns.

In particular, your linked page claims:

Similarly, the Timedelta constructor and to_timedelta() with a string input now defaults to a microsecond unit, using nanosecond unit only in cases that actually have nanosecond precision.

I am using pd.to_timedelta([1,2,3], unit="d") and get NANOsecond precision. Why?

>>> pd.to_timedelta([1,2,3], unit="D")
TimedeltaIndex(['1 days', '2 days', '3 days'], dtype='timedelta64[ns]', freq=None)

Granted, it doesn't say anything about int values and days. But why would that result in ns?

Let's boil down the question to this simple one-liner that works in Pandas 2 but fails in Pandas 3:

pd.testing.assert_frame_equal(
    pd.DataFrame({"d": pd.to_datetime(["2020-01-01"])}) + pd.Timedelta(1, unit="D"),
    pd.DataFrame({"d": pd.to_datetime(["2020-01-02"])})
)

In my optinion, this should always work. Please tell me how I would fix this.

When using pd.Timedelta("1D") instead, the problem is gone. But in reality a have a series of days to add. It seems nonsensical to do pd.to_timedelta(day_series.astype(str) + "D") instead of pd.to_timedelta(day_series, unit="D").

[jbrockmendel]

I am using pd.to_timedelta([1,2,3], unit="d") and get NANOsecond precision. Why?

There's been some discussion about this, and I think participants generally agree it would be nice to give a lower unit here (i.e. output unit match input unit, or closest-to-supported). This works for integer data, but can screw up with floats (xref #53445).

Will need to give this some more thought.