Add pandas 3 compatibility for string dtype handling

[guillaume-vignal]

Fixes #714

Add pandas 3 compatibility for string/categorical dtype handling, while keeping pandas 2 support.

Main updates:

• Replace fragile "str" dtype checks with pandas-aware logic.
• Add shared text-like helpers (is_text_like, text_like_columns) and reuse them across core modules.
• Improve object/string/category handling in transform, clustering, consistency, webapp example, and correlations.
• Keep category-encoder inverse dtype metadata with a safe fallback.
• Add/adjust unit tests for pandas string metadata and text-like behavior.
• Align tutorials/notebooks with the new shared helper.
• Add a CI pandas compatibility leg (2.2.2 / 2.3.* / 3.*, Python 3.12).

[Copilot]

Pull request overview

This PR aims to make Shapash compatible with pandas 3’s default string dtype inference by updating string/categorical dtype checks across core utilities, examples, and tests, and by bumping the minimum supported pandas version.

Changes:

• Update string/categorical column detection logic in utilities and example code to account for pandas 3 string inference.
• Adjust unit tests’ expected preprocessing metadata (data_type) to reflect string dtype handling changes.
• Bump pandas minimum version to >=2.3.0 in pyproject.toml.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tutorial/production_and_ops/tuto-prod03-batch-scoring-parquet.ipynb	Updates dtype-based selection before Parquet write to cast text-like columns.
tests/unit_tests/utils/test_transform.py	Updates expected data_type values in preprocessing-mapping related tests.
tests/unit_tests/utils/test_columntransformer_backend.py	Updates expected data_type values in CT inverse/transform tests.
tests/unit_tests/utils/test_check.py	Updates expected data_type values in preprocessing consistency tests.
tests/unit_tests/utils/test_category_encoders_backend.py	Updates expected data_type values in category-encoder backend tests.
tests/unit_tests/explainer/test_smart_predictor.py	Updates expected data_type values used in preprocessing checks.
tests/unit_tests/decomposition/test_inverse_contribution.py	Updates expected data_type values used in inverse contribution tests.
shapash/webapp/webapp_launch_DVF.py	Updates example categorical feature detection for encoding.
shapash/utils/transform.py	Updates categorical missing-value handling selection logic.
shapash/utils/clustering.py	Updates categorical detection for color encoding in clustering/plots.
pyproject.toml	Raises minimum pandas version to >=2.3.0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 40 out of 40 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 40 out of 40 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 40 out of 40 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

shapash/utils/check.py:421

• In the regex branch, the DataFrame-type error message incorrectly mentions the upper/lower method, which can mislead users when diagnosing invalid postprocessing configuration.

                    if not pd.api.types.is_string_dtype(x[key]):
                        raise ValueError(f"Expected string object to modify with upper/lower method in {key} dict")

[Copilot]

Pull request overview

Copilot reviewed 40 out of 40 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 40 out of 40 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 4 comments.

[milton-minervino]

Consolidated review — replaces all my earlier inline comments

I re-reviewed against current head (237899e) rather than my June comments, and you've already
addressed most of what I raised. Please disregard my earlier inline comments — the
_is_string_dtype_metadata helper in check.py, the tgt_enc.get("data_type") or "object"
fix, the infer_dtype narrowing in report/common.py, and the df.copy() in
plot_correlations.py all resolve them.

I verified the current code on pandas 2.2.2, 2.3.3 and 3.0.3 (Python 3.12):
select_dtypes(include=["object","string","category"]) works on all three with no warning, and
your is_string_dtype(s) or s.dtype.name in ("object","category") predicate gives correct
results in all 15 dtype cases I tried. The approach is sound.

Five things left.

1. Two behaviour changes that should be explicit

Neither is wrong, but both change output for existing users and aren't called out anywhere:

(a) report/common.py — object columns holding non-text now return TYPE_UNSUPPORTED
instead of TYPE_CAT:

column contents	infer_dtype	before	after
["a", None, "b"]	string	TYPE_CAT	TYPE_CAT
[["a"], {"b": 1}]	mixed	TYPE_CAT	TYPE_UNSUPPORTED
datetimes as object	datetime	TYPE_CAT	TYPE_UNSUPPORTED
all-NaN / empty object	empty	TYPE_CAT	TYPE_CAT

I think this is a genuine improvement, but a report that previously rendered will now drop
those columns. Worth a changelog line.

(b) plot_correlations.py — keep_mask = df[col].isna() | df[col].isin(top_categories)
means missing values now stay NaN, whereas previously .where(isin(top), other="Other")
folded them into "Other". That changes the computed correlation matrix for any dataset with
missing categoricals. Also an improvement in my view, but same request: make it deliberate and
note it.

2. One notion of "text column", not three

The PR currently uses three different predicates, which disagree with each other:

# transform.py
df.select_dtypes(include=["object", "string", "category"])
# clustering.py, consistency.py, webapp_launch_DVF.py
is_string_dtype(s) or s.dtype.name in ("object", "category")
# report/common.py, plot_correlations.py
infer_dtype(s, skipna=True) in ("string", "unicode", "empty")

For an object column holding lists or datetimes, the first two say "categorical" and the third
says "not". That's the inconsistency I was clumsily gesturing at in June. Could we land one
shared helper and route all six sites through it, so the strict/permissive choice is made once
and visibly?

If you keep the permissive semantics, this is exactly equivalent to your current expression —
I checked all 15 dtype cases on 2.2.2 / 2.3.3 / 3.0.3, zero differences — and it's shorter and
avoids the O(n) value inference is_string_dtype(series) does on object columns:

def is_text_like(series: pd.Series) -> bool:
    dtype = series.dtype
    return is_string_dtype(dtype) or isinstance(dtype, CategoricalDtype)

(Worth knowing, since it's what caused our disagreement: is_string_dtype(series) infers from
values and returns False for pd.Series(["a", np.nan], dtype=object), while
is_string_dtype(series.dtype) reads the dtype and returns True. You were right that my
original suggestion was broken; it was one .dtype away from working.)

Minor: "unicode" in the infer_dtype tuples is a pandas-1-era return value and can't occur
on pandas 2/3 — harmless, but removable.

3. Nice to have: a pandas 2 / pandas 3 CI leg

.github/workflows/main.yml matrixes Python 3.11–3.14 against a single
(latest) pandas, so nothing in CI actually exercises the compatibility this PR is about, on
either side. With pandas>=2.2.2 declared in pyproject.toml, I'd like the floor tested:

  pandas_compat:
    name: pandas ${{ matrix.pandas }}
    needs: code_quality
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        pandas: ["2.2.2", "2.3.*", "3.*"]
    steps:
      - uses: actions/checkout@v6
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"   # pinned: pandas 2.2.2 has no cp313/cp314 wheels
      - run: |
          python -m pip install --upgrade pip
          python -m pip install '.[all]'
          python -m pip install 'pandas==${{ matrix.pandas }}'
          python -c "import pandas; print(pandas.__version__)"
      - run: make coverage

This would be a great consolidation test and, with green light, it would remove any doubt, but I'll leave it optional, up to you if you want to add it or not. If it turns out 2.2.2 can't be supported, raising the floor in
pyproject.toml is a fine outcome too.

4. Notebook output churn

37 files, but only 8 are shapash/ source (+61/−14).
tuto-debug02-recourse-what-if-simulation.ipynb alone is +698/−693, which looks like a
re-execution rather than a content change. It could be worth to revert the only-outputs changes.

5. One unrelated change

tests/integration_tests/test_contributions_multiclass.py changes
model.fit(self.x_train, self.y_train) to .y_train.values.ravel(). That looks like a sklearn
shape/warning fix rather than anything dtype-related — is it actually required by pandas 3? If
not, I'd rather it went separately. Not blocking.

---

Nothing else from me. (3) is the only hard gate; (1) just needs acknowledgement, (2) is a
consolidation I think is worth doing, (4) is mechanical, (5) is a question.