Skip to content

Track benchmark performance in CI with CodSpeed #2587

Description

@LucaCappelletti94

sqlparser_bench never runs in CI, only clippy checks it. The benchmarks once measured error paths without anyone noticing (#1559), and the six parse_*_chain groups that guard the exponential parse-time fixes (#2344, #2350, #2352, #2372, #2392) would not catch a regression either.

The proposal is to run them with CodSpeed in simulation mode, which counts instructions under Valgrind on ubuntu-latest, so results stay stable on shared runners.

Two ASF prerequisites.

apache/arrow-rs#9975 and apache/datafusion#21165 plan the same setup. OpenDAL dropped CodSpeed in apache/opendal#5626 over false positives from I/O-bound benchmarks, while parsing is CPU-bound.

  • Get the action approved allowlist: re-add CodSpeedHQ/action v5.2.1 infrastructure-actions#1327
  • Swap criterion for codspeed-criterion-compat in sqlparser_bench/Cargo.toml. Bench sources and local cargo bench stay unchanged.
  • One workflow on push to main plus a nightly schedule, and PR runs once the numbers prove stable.
  • About 30 benchmarks, so one job.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions