[python] Align DATE partition paths with Java

[zhengguangzhuo]

Fixes #10034

Summary

• Align PyPaimon DATE partition data-file paths with Java when partition.legacy-name=true.
• Convert only DATE partition values to epoch days, so 1970-01-02 is written under day=1.
• Apply the same compatibility path to local data files, external data files, bucket indexes, and commit recovery.

Compatibility

• Existing Python-written ISO directories such as day=1970-01-02 remain readable.
• partition.legacy-name=false keeps the existing ISO naming behavior.
• Non-DATE partition components keep their existing PyPaimon formatting.
• This is a compatibility-focused rework of the DATE path issue after [python] Align DATE partition data paths with table naming options #10045 was closed because its broader implementation introduced breaking changes.

Verification

• pypaimon/tests/date_partition_path_test.py: 5 passed.
• Focused writer/commit regression tests: 3 passed.
• pycodestyle, pyflakes, compileall, and git diff --check passed.
• The Java Maven test was attempted, but dependency resolution was blocked by a timeout from the configured Maven mirror in the local environment.

[wangzhigang1999]

Thanks for working on this issue. I reported it and also attempted a fix in #10045, which I later closed because of compatibility concerns. I think this implementation still misses some of the difficulties I encountered.

There is a read-after-write regression with composite partitions. With partition.legacy-name=true, day=1970-01-02, and region=a/b, this version writes to day=1/region=a/b/..., but the Python reader cannot resolve that mixed-format path and fails with FileNotFoundError. The added composite-partition test checks the path without verifying readback.

The historical-directory test also writes with the new code and then renames the entire directory. That doesn’t cover historical and new paths coexisting in the same logical partition/bucket, or compatibility between client versions.

We need a compatibility matrix covering Java and Python writers/readers, old and new versions, and both partition naming modes, with the full Cartesian product of supported combinations. This should include different writers appending to the same partition/bucket, verifying that readers return all expected rows, and checking reads after rollback.

I previously tried adding compatibility handling for different path formats, but the additional lookups caused performance regressions. We haven’t settled on how to handle that tradeoff, so this change needs performance evaluation alongside the compatibility tests.

Could you address these gaps before we proceed with the write-path change? Preserving reads of historical ISO directories alone isn’t enough to establish compatibility.

Please also use English for the added comments and docstrings, consistent with the surrounding code.

[JingsongLi]

Confirmed the composite-partition read-after-write issue at 7557c7e144 with an actual table write, commit and read. The DATE compatibility problem in #10034 is worth fixing, but this version should be blocked from production.

[P1] Make the writer and reader agree on the new composite partition layout

pypaimon/utils/file_store_path_factory.py:247-273 converts only DATE components, creating a third layout that neither reader path resolves. With default partition.legacy-name=true, day=1970-01-02 and region=a/b:

New writer:             day=1/region=a/b/bucket-0/data-....parquet
Historical reader path: day=1970-01-02/region=a/b/bucket-0/data-....parquet
Canonical reader path:  day=1/region=a%2Fb/bucket-0/data-....parquet

The commit succeeds, then readback raises FileNotFoundError. An in-process control retaining the previous writer path reads the same row successfully; partition.legacy-name=false also reads back successfully. This confirms a newly introduced failure rather than an existing unreadable table.

I ran the added DATE tests plus file-store-commit and Mosaic writer regressions: 39 tests and 2 subtests passed; changed files passed Flake8. The current composite test asserts only the path and misses the failing readback.

Please make planning resolve the actual new layout, or use a fully canonical layout with historical compatibility, and add write → commit → read assertions for composite DATE partitions with values requiring escaping. Preserve historical/new-file coexistence while doing so.

[Akash3121]

+1