feat: Add retrieval metrics, hierarchical assertion scoring, and data-linked questions

[ha2trinh]

Summary

This PR adds several major features to the benchmark-qed evaluation framework: a retrieval metrics evaluation pipeline, hierarchical assertion scoring with multi-RAG comparison support, data-linked cross-document question generation, and improvements to the existing question generation pipeline.

Major Features

Retrieval Metrics Evaluation Pipeline

Retrieval metrics measure how well a RAG system's retrieval step finds the right source documents. This feature adds an end-to-end pipeline to evaluate retrieval quality by comparing retrieved text units against curated reference sets, computing standard IR metrics (precision, recall, fidelity).

• New retrieval-scores CLI command with configurable settings
• Support for multiple input formats: parquet, csv, json, jsonl
• Flexible column name mapping via TextUnitFieldsConfig
• Relevance assessor types: rationale-based and bing-style raters
• Built-in caching to avoid redundant LLM assessments
• Significance testing utilities for comparing retrieval methods

Hierarchical Assertion Scoring with Multi-RAG Comparison

Assertion-based evaluation scores RAG answers against factual claims (assertions) derived from source documents. This feature extends the existing assertion scoring to support hierarchical assertions — global claims that are backed by supporting sub-claims — and enables side-by-side comparison of multiple RAG methods in a single evaluation run with statistical significance testing.

• Hierarchical evaluation pipeline: global assertions with supporting sub-assertions, evaluated in staged mode (global first, then supporting for passed globals) or joint mode (single LLM call)
• Multi-RAG comparison: evaluate multiple RAG methods against the same assertions with automatic statistical analysis
• Significance testing: repeated measures tests (Friedman/Wilcoxon), clustered permutation tests (accounting for within-question correlation), and combined p-value summaries
• Shared metric aggregation functions extracted into a reusable aggregation.py module

Data-Linked Question Generation

Data-linked questions test a RAG system's ability to synthesize information across multiple source documents, rather than answering from a single document. This feature adds generation of cross-document questions with four relationship types.

• Bridge questions: connecting facts across two documents
• Comparison questions: comparing entities/events across documents
• Intersection questions: finding common themes across documents
• Temporal questions: reasoning about time-ordered events across documents
• Batch validation with entity relevance checking
• Dedicated validators for linked and global question types

Question Generation Improvements

• MMR (Maximal Marginal Relevance) sampler for better question deduplication
• Improved entity extraction prompts for higher quality context
• Weighted sampling for data_global categories (log-weighted by question count)
• New min_questions_in_context config option to filter low-question categories

Other Changes

Code Quality

• Fix all pre-existing ruff lint and formatting errors across the codebase (0 remaining)
• Add type annotations for full pyright typecheck and verifytypes compliance (100% type completeness score)

Configuration & CLI

• Add cluster_match_by config option for retrieval metrics (text/id/short_id)
• Add configurable reference filename and flexible reference format parsing

Datasets

• Rename AP_News generated_questions to generated_questions_v1
• Add generated_questions_v2 for both AP_News and podcast with assertion files for data_global, data_local, and data_linked question types

Documentation

• Update example notebooks: autoe.ipynb, autoq.ipynb, retrieval_metrics.ipynb
• Update CLI docs for autoe and autoq