Skip to content

Postpone-bucket getWriteId misparses the write id when data-file.prefix or the commit user contains the -s- separator #10265

Description

@LuciferYang

Search before asking

  • I searched in the issues and found no similar issues.

Paimon version

master (1.5-SNAPSHOT)

Compute Engine

Flink (postpone-bucket write path).

Minimal reproduce step

  1. Create a postpone-bucket table and set data-file.prefix to a value that contains the -s- separator (for example a-s-b).
  2. Write data so postpone-bucket data files are produced. Each file name is {data-file.prefix}-u-{commitUser}-s-{writeId}-w-{uuid}-{count}.
  3. Read or compact the table, which routes splits by getWriteId(fileName) % parallelism.

What doesn't meet your expectations?

PostponeBucketFileStoreWrite.getWriteId recovered the write id by splitting on the first -s- in the file name. When data-file.prefix (or a custom commit user) already contains -s-, the first -s- is not the writer's separator, so the parse either throws Data file name ... does not match the pattern or returns a wrong write id. A wrong id then routes the split to the wrong reader under getWriteId(...) % parallelism.

Anything else?

The correct anchor is the -w- writer marker that always terminates the interpolated prefix, so the write id is the segment between the last -s- before -w- and -w- itself.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions