Skip to content

Update DuckDB to 1.5.0+, Run each benchmark query in its own job for OOM resilience#93

Merged
jiayuasu merged 2 commits into
mainfrom
update/duckdb-150-benchmark-defaults
Mar 24, 2026
Merged

Update DuckDB to 1.5.0+, Run each benchmark query in its own job for OOM resilience#93
jiayuasu merged 2 commits into
mainfrom
update/duckdb-150-benchmark-defaults

Conversation

@jiayuasu
Copy link
Copy Markdown
Member

@jiayuasu jiayuasu commented Mar 23, 2026

Summary

  • Remove DuckDB <1.5.0 version cap so stable resolves to 1.5.x+ (fixes spatial extension install issue)
  • Default DuckDB and SedonaDB to stable releases instead of nightly (nightly remains opt-in via workflow_dispatch)
  • Default scale factors to SF1 and SF10 with 600s per-query timeout
  • Run each benchmark query (q1-q12) as a separate GitHub Actions job so that if one query OOMs and kills the runner, remaining queries still execute on fresh runners
  • Cache pip dependencies across per-query jobs to reduce setup overhead
  • Summarize script merges per-query result files and fills in missing engines/queries as not_started (OOM) so the report always shows all engines

Closes #92

@jiayuasu jiayuasu force-pushed the update/duckdb-150-benchmark-defaults branch 5 times, most recently from b8f4ce7 to 7111b96 Compare March 23, 2026 08:08
Copy link
Copy Markdown
Member

@paleolimbot paleolimbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

10 seconds might be a bit aggressive for timeout (many of the result display QX: Query qX timed out after 10 seconds (process killed)

- Remove DuckDB <1.5.0 version cap so nightlies resolve to 1.5.x+
- Default DuckDB and SedonaDB to stable releases (nightly opt-in)
- Default scale factors to SF1,SF10 with 600s per-query timeout
- Run each query as a separate GitHub Actions job so OOM on one
  query does not block remaining queries
- Cache pip dependencies across per-query jobs for faster setup
- Summarize script merges per-query result files, fills missing
  engines/queries as not_started (OOM), and shows partial results
@jiayuasu jiayuasu force-pushed the update/duckdb-150-benchmark-defaults branch from 4cec883 to c22b277 Compare March 24, 2026 05:57
@jiayuasu jiayuasu changed the title Update DuckDB to 1.5.0+, default to SF10 with 300s timeout Run each benchmark query in its own job for OOM resilience Mar 24, 2026
@jiayuasu jiayuasu changed the title Run each benchmark query in its own job for OOM resilience Update DuckDB to 1.5.0+, Run each benchmark query in its own job for OOM resilience Mar 24, 2026
@jiayuasu jiayuasu merged commit f788c0f into main Mar 24, 2026
88 of 111 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] DuckDB 1.5.0 not included

2 participants