Infrastructure: Tier I/II/III aggregated scoring and weighting

[duncanwp]

Description

Design and implement the overall scoring scheme that combines results across tiers.

Open questions

• How to weight individual metrics within each tier
• How to combine Tier I pass/fail with Tier II continuous scores
• Treatment of extended vs core metrics in overall score
• Presentation format (leaderboard, radar plots, etc.)
• Whether Tier I failures should gate Tier II/III scoring entirely

Notes

• This is a protocol-level design question that needs community input
• Start with a simple additive scheme and iterate

[duncanwp]

Proposed solution: adapt ICONEval's portrait plot and HTML summary

ICONEval provides two components for aggregated results:

1. Portrait plot — recipe_portrait_plot computes normalized RMSE across 26 variable/level combinations with CMIP6 comparison. This is directly usable as the Tier II scorecard. Adapt the normalization scheme to match ClimateBench2 scoring weights.

2. HTML summary — ICONEval's _summarize.py generates Bootstrap 5 card layouts with thumbnails, search, and per-realm breakdown (atmosphere, ocean, land, sanity checks). This maps to ClimateBench2's tier structure. The pattern of auto-generating summaries from recipe outputs with embedded metadata and provenance tracking is worth adopting.

Implementation:

• Port recipe_portrait_plot as the cross-variable scoring view
• Build a summary generator (inspired by ICONEval's _summarize.py) that produces HTML from results CSVs, linkable from the existing ClimateBench web app
• Add provenance tracking: record which recipe version + data version produced each score

Phase: 4 (Scorecard and reporting) in the integration plan
ICONEval reference: recipe_portrait_plot.yml, iconeval/output_handling/_summarize.py