Search before asking
Paimon version
master (1.5-SNAPSHOT)
Compute Engine
Flink (postpone-bucket write path).
Minimal reproduce step
- Create a postpone-bucket table and set
data-file.prefix to a value that contains the -s- separator (for example a-s-b).
- Write data so postpone-bucket data files are produced. Each file name is
{data-file.prefix}-u-{commitUser}-s-{writeId}-w-{uuid}-{count}.
- Read or compact the table, which routes splits by
getWriteId(fileName) % parallelism.
What doesn't meet your expectations?
PostponeBucketFileStoreWrite.getWriteId recovered the write id by splitting on the first -s- in the file name. When data-file.prefix (or a custom commit user) already contains -s-, the first -s- is not the writer's separator, so the parse either throws Data file name ... does not match the pattern or returns a wrong write id. A wrong id then routes the split to the wrong reader under getWriteId(...) % parallelism.
Anything else?
The correct anchor is the -w- writer marker that always terminates the interpolated prefix, so the write id is the segment between the last -s- before -w- and -w- itself.
Are you willing to submit a PR?
Search before asking
Paimon version
master (1.5-SNAPSHOT)
Compute Engine
Flink (postpone-bucket write path).
Minimal reproduce step
data-file.prefixto a value that contains the-s-separator (for examplea-s-b).{data-file.prefix}-u-{commitUser}-s-{writeId}-w-{uuid}-{count}.getWriteId(fileName) % parallelism.What doesn't meet your expectations?
PostponeBucketFileStoreWrite.getWriteIdrecovered the write id by splitting on the first-s-in the file name. Whendata-file.prefix(or a custom commit user) already contains-s-, the first-s-is not the writer's separator, so the parse either throwsData file name ... does not match the patternor returns a wrong write id. A wrong id then routes the split to the wrong reader undergetWriteId(...) % parallelism.Anything else?
The correct anchor is the
-w-writer marker that always terminates the interpolated prefix, so the write id is the segment between the last-s-before-w-and-w-itself.Are you willing to submit a PR?