external-repo benchmark framework + 9 bench integrations - #66
Merged
Conversation
…MMMU, MMMU-Pro, MathVista, BFCL, GAIA, tau-bench) with Docker sandbox and per-sample HTML reports
- Fix HumanEval+/MBPP+ id-range handling for max_samples (handle gaps in dataset ids) - Add per-sample detail JSONL output for LiveCodeBench with enriched fields - Clear stale LiveCodeBench output to prevent caching issues - Update leaderboard reference scores with more benchmarks
…orts - Update render_report.py to support image_paths field in samples - Fix image path handling (convert to relative paths with forward slashes) - Move images to top of detail view and increase max size to 500px - Add question text to MMMU/MMMU-Pro/MathVista prompts for better readability - Update bridge scripts to save images and include image_paths in sample records Affected benchmarks: MMMU, MMMU-Pro, MathVista
- Add mbppplus benchmark with 7 models from EvalPlus leaderboard - Restrict Python to 3.10-3.11 for Windows compatibility (whylogs-sketching has no 3.12 wheel) - Add uv.lock to ensure consistent dependency versions across environments
- SKILL.md: update external_repo description to reflect ExternalRepoRunner automation - bench_gallery.json: fix mbppplus sample count (399→378) in download_config.reason - run_mathvista_oneeval.py: remove --test_split param, hardcode testmini (test split has no public labels) - leaderboard_scores.json: add mmmu reference scores (6 entries, val split)
- Fix MMMU option format from "A. option" to "(A) option" to align with official MMMU parse logic (was causing ~random accuracy due to parser fallback) - Fix MMMU log accuracy format from :.2f% to :.4f (value is 0-1, not percent) - Add smoke test support for external_repo benches in run_eval.py - Remove unimplemented benches from bench_gallery.json and bench_gallery.md
MMMU:
- Image-text interleaving: split by <image N> placeholders and insert
images at correct positions (aligned with VLMEvalKit split_MMMU)
- Option format: A. -> (A) to match official parser expectations
MMMU-Pro:
- parse_answer: use rfind("Answer:") + full fallback chain from
official evaluate.py parse_multi_choice_response
- Standard mode: extract image order from full text (question+options+prompt),
replace placeholders with <image>, append images in occurrence order
(aligned with official replace_images_tokens + make_interleave_content)
- Vision mode: text-first then image (aligned with official)
- Remove unused get_images_for_standard function
- Increase Docker mode timeout from 2h to 4h - Increase non-Docker mode timeout from 1h to 2h - Set LiveCodeBench max_workers to auto-detect CPU count (default -1) - Update LiveCodeBench supported versions to v1-v6
- Add --num_process_evaluate parameter (default: 1) - Root cause: evaluation stage was using LCB default of 12 processes - 12 eval processes × subprocesses caused memory spike in 16GB container - Single-process evaluation is slower but memory-safe for full v6 dataset
scuuy
approved these changes
Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.