feat: measure the shortlist funnel loss, then size the shortlist per patient by majdabd · Pull Request #30 · cbib/TrialMatchAI

majdabd · 2026-08-06T09:25:28Z

Two commits: measure the pipeline's largest loss, then act on its dominant cause.

1. `feat(eval)`: expose the shortlist funnel loss

recall@k measures the first-level candidate list. nDCG@k measures the ranked list. Neither can see the stage between them, which is where the pipeline loses the most: the shortlist handed to the eligibility reasoner is far shorter than the candidate list, and a relevant trial dropped there can never be ranked, however good the reasoning is.

Run	Track	First level	Shortlist	Size	Loss
qwen36_medcpt	2021	0.866	0.577	196	−33.4%
h100_baichuan	2021	0.866	0.581	197	−32.9%
qwen36_medcpt	2022	0.813	0.517	155	−36.4%
l40_trec23	2023	0.644	0.254	243	−60.6%

The 2023 row shows why this stayed hidden. That run has the best nDCG@10 of any run (0.881) and the worst shortlist recall (0.254). Condensed nDCG only orders the judged trials that survive the funnel, so it cannot report the three quarters that never arrived.

evaluate() now reports shortlist_recall, shortlist_size, funnel_depth_loss and shortlist_selection_delta whenever top_trials.txt is present.

Where the loss comes from (TREC 2021):

Stage	Recall	Cause
Full first level (2000)	0.866	—
First-level top-200	0.595	Depth: −0.272 (94% of the loss)
Actual shortlist (196)	0.577	Ordering: −0.017 (6% of the loss)

shortlist_selection_delta is negative on all four runs (−0.032, −0.027, −0.022, −0.009): at the same depth, the criterion reranker plus RRF fusion currently select worse than taking the first-level top-N and doing nothing. Small beside the depth loss, but free to recover — and it means the second level should be re-tested rather than assumed helpful. Not addressed here.

2. `feat(search)`: per-patient shortlist depth

Depth is 94% of the loss, and no single number fits. The depth a patient needs to reach 90% of its own first-level recall ranges from 50 to 1550 trials, spread evenly — 17 of 75 patients need ≤200, 21 need >700. Sizing for the worst case wastes about 65% of the reasoner's compute; sizing for the median drops the hard patients.

New search.shortlist.policy:

fixed — existing divisor sizing, unchanged. Still the default.
relative_to_max — keep trials scoring ≥ alpha × this patient's top score.

A peaked first-level score curve means retrieval was confident and few trials are plausible; a flat curve means many are. The cut is relative to the patient's own maximum because RRF scores are comparable only within one patient.

Tuning

Done offline by replaying first_level_scores.json from the completed runs — no GPU time. Compared against fixed depth at equal mean depth, which is the only fair test: a policy must spend compute better, not merely spend more.

Track	Gain at equal compute
TREC 2021	+0.017 to +0.029 recall (peak at mean depth ~305)
TREC 2022	+0.007 to +0.025 recall
TREC 2023	−0.010 to +0.006 — questionnaire topics, no gain

Two other policies were tried and rejected: cumulative score mass tracked fixed depth to within ±0.003 everywhere, and half-max curve width won only below ~350 mean depth and lost above it.

Two effects, reported separately

At alpha=0.25 the policy also chooses to go deeper (196 → 305 trials on 2021), lifting shortlist recall 0.609 → 0.710. That part is bought with compute. The equal-cost table above is the free part. Both are real and they should not be conflated.

Safety

Default fixed, so enabling this is an explicit A/B, never a silent change. Verified: existing configs parse to policy: fixed.
Degrades to the fixed depth when first-level scores are absent (resumed runs).
Never exceeds what the reasoner can actually consume (rag.max_trials_rag), since trials past that cap get no eligibility output and vanish from the ranking.
Writes shortlist_depth.json recording the decision and the depth the old sizing would have chosen.

Verification

433 tests pass; ruff check src/ tests/ clean.
12 new tests for the depth policy, 2 for the funnel metrics.
The production choose_shortlist_depth() was replayed over the real runs and reproduces the offline simulator's depths and recalls exactly.
No behaviour change under default config; runs predating top_trials.txt still evaluate, with funnel keys None.

Not in this PR

The negative shortlist_selection_delta, constraint/calculator tools for numeric criteria, and any iterative retrieval loop. This PR deliberately contains no agent loop — the measurements say depth allocation is the dominant lever, so it comes first and gets evaluated on its own.

🤖 Generated with Claude Code

recall@k measures the first-level candidate list and nDCG@k measures the ranked list, so neither can see the pipeline's largest loss: the shortlist handed to the eligibility reasoner is far shorter than the candidate list, and a relevant trial dropped there can never be ranked. Measured on the completed runs, the shortlist discards 33% (TREC 2021), 36% (2022) and 61% (2023) of the relevant trials the first level had already found. The 2023 run shows why this stayed hidden: it has the best nDCG@10 of any run (0.881) and the worst shortlist recall (0.254), because the condensed metric only orders the judged trials that survive the funnel. evaluate() now reports, whenever top_trials.txt is present: - shortlist_recall .............. recall of the list the reasoner read - shortlist_size - funnel_depth_loss ............. recall given up by shortening alone - shortlist_selection_delta ..... second-level ordering vs a plain first-level top-N cut at the same depth The split matters for what we fix first. On TREC 2021 depth accounts for 94% of the loss (0.866 -> 0.595 at top-200) and ordering for 6% (-> 0.577). The selection delta is negative on all four runs (-0.032, -0.027, -0.022, -0.009): the criterion reranker plus RRF shortlist fusion currently select worse than taking the first-level top-N and doing nothing. Runs without top_trials.txt still evaluate; the funnel keys report None. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copilot

Pull request overview

This PR extends the TREC qrels evaluation to expose “funnel” metrics that quantify recall loss between the first-level candidate list and the second-level shortlist (when top_trials.txt exists), and adds tests to validate the new outputs.

Changes:

Add parsing of top_trials.txt and compute shortlist/funnel metrics in evaluate().
Include new funnel metrics in the returned per-query rows and mean aggregate.
Add unit tests covering funnel metric reporting and behavior when the shortlist file is absent.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
tests/test_qrels_eval.py	Adds tests asserting new funnel metrics and their absence (`None`) when `top_trials.txt` is missing.
src/trialmatchai/trec/qrels.py	Implements shortlist parsing and aggregates new funnel metrics into `evaluate()` results.

Suppressed comments (1)

src/trialmatchai/trec/qrels.py:245

Funnel metrics are currently computed only when shortlist is truthy and retrieved is truthy. This skips reporting when top_trials.txt exists but is empty (or when the first-level list is empty), despite the docstring/PR intent to report whenever the shortlist file is present. Using an explicit None sentinel (see _shortlist_for_patient) and checking is not None avoids conflating “absent” with “empty” and still keeps metrics None when the file truly doesn’t exist.

        shortlist = _shortlist_for_patient(patient_dir)
        if shortlist and retrieved:
            depth = len(shortlist)
            short_r = recall_at_k(shortlist, rel_set, depth)
            first_r = recall_at_k(retrieved, rel_set, depth)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+def _shortlist_for_patient(patient_dir: Path) -> list[str]:
+    """The trials that actually reached the eligibility (CoT) stage.
+
+    This is the funnel's narrowest point: a relevant trial dropped here is
+    unrecoverable, however good the first-level search was.
+    """
+    shortlist = patient_dir / "top_trials.txt"
+    if not shortlist.exists():
+        return []
+    return [line.strip() for line in shortlist.read_text().splitlines() if line.strip()]


+    - ``shortlist_recall`` -- recall of the list the reasoner actually read.
+    - ``funnel_depth_loss`` -- recall given up by shortening the list alone.
+    - ``shortlist_selection_delta`` -- second-level ordering minus a plain
+      first-level cut at the same depth. Negative means the second level
+      selected worse than doing nothing.


…curve The shortlist divisor sizes every patient identically, but depth is 94% of the measured shortlist recall loss (PR #30) and no single number fits: the depth a patient needs to reach 90% of its own first-level recall ranges from 50 to 1550 trials, spread evenly across that range. Sizing for the worst case wastes ~65% of the reasoner's compute; sizing for the median drops the hard patients. Adds search.shortlist.policy: fixed existing divisor sizing, unchanged -- still the DEFAULT relative_to_max keep trials scoring >= alpha x this patient's top score A peaked first-level score curve means retrieval was confident and few trials are plausible; a flat curve means many are. The cut is relative to the patient's own maximum because RRF scores are only comparable within one patient. Tuned offline by replaying the completed runs' first_level_scores.json, so no GPU time. Compared against fixed depth AT EQUAL MEAN DEPTH, which is the only fair test -- a policy must spend compute better, not merely spend more: TREC 2021 +0.017 to +0.029 recall (peak at mean depth ~305) TREC 2022 +0.007 to +0.025 recall TREC 2023 -0.010 to +0.006 recall -- questionnaire topics, no gain Two other policies were tried and rejected: cumulative score mass tracked fixed depth to within +-0.003 everywhere, and half-max curve width won only below ~350 mean depth and lost above it. At alpha=0.25 the policy also chooses to go deeper (196->305 trials on 2021), which lifts shortlist recall 0.609->0.710. That part is bought with compute, not won for free; the equal-cost gain above is the free part. Both are real and they should be reported separately. Default stays "fixed" so enabling this is an explicit A/B. Degrades to the fixed depth when first-level scores are absent (resumed runs), never exceeds what the reasoner can consume, and writes shortlist_depth.json recording the decision and the depth the old sizing would have picked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ond-level scores The depth policy added in 91f724f is tuned only against replayed first-level scores. Whether the extra depth reaches nDCG@10 and P@10 -- rather than just shortlist recall -- needs a GPU run. This adds the reporting half of that experiment. compare_shortlist_ab.py reads the arms of a shortlist-policy A/B and always prints shortlist_size beside the quality metrics, because a depth policy produces two effects that must not be conflated: spending compute BETTER same mean shortlist size, more recall -- a free gain spending compute MORE bigger shortlist, bought with GPU time Reporting only the second reads as a much larger win than it is. The planned arms are fixed (~196 trials/patient), relative_to_max alpha 0.25 (~305, deeper), and relative_to_max alpha 0.33 (~196, equal cost). The equal-cost arm is the one that actually tests the policy. The job script itself stays untracked: *.slurm is gitignored repo-wide and none of the 26 existing job scripts are committed, since they encode site-specific paths, partitions and staging. Also persists second_level_scores.json for the whole second-level pool, not just the shortlist. With first_level_scores.json this makes shortlist fusion replayable offline, so the negative shortlist_selection_delta measured in 05c6f11 -- the fused shortlist selecting worse than a plain first-level cut at the same depth -- can be investigated without spending a GPU job on each candidate fix.

The llm_expansion search channel was dead. LLMQueryExpansionBackend (the Protocol), parse_llm_query_expansion (the parser), LLMQueryExpansion (the schema), the channel's 0.5 weight and search.first_level.llm_expansion_enabled all existed, but nothing ever built a backend and main.py never passed one to ClinicalTrialSearch. Setting the flag logged "no expander is enabled but no expander is configured" and contributed no terms. Adds FirstLevelQueryExpander, which subclasses QueryExpander to reuse its engine, chat template and structured-output machinery -- so it shares the one cached vLLM engine rather than loading a second copy -- and swaps in a retrieval-query prompt and schema. This is a genuinely different task from the existing expander, not a rename. QueryExpander enriches the patient SUMMARY (conditions plus narrative sentences) for the expand stage. This one writes RETRIEVAL QUERIES: short noun phrases that should match a trial's title, condition list or eligibility text, bucketed into the six fields the planner turns into weighted query channels. To support both, QueryExpander's prompt and schema became overridable class attributes; the base class's values are unchanged. The per-field maxItems caps are deliberately uneven. llm_max_terms is a SHARED budget across the five query fields, spent in field order, so a model that pads primary_queries starves biomarker_queries and treatment_queries entirely -- a test pins that behaviour. primary_queries is therefore capped at 3 (a patient has one main disease), leaving the budget for the later fields. Off by default: the builder returns None unless llm_expansion_enabled is set, and returns None rather than raising if the expander cannot be constructed. A failed expansion degrades to empty and is logged, because this is one channel of eight -- losing it should cost recall, not the run. Wired once per run in main_pipeline, not per patient, so the engine is resolved once.

ndcg@10 cannot answer a question about depth, and reading the A/B on it would have given exactly the wrong answer. It normalizes by the ideal over judged-AND-RANKED trials, which makes it recall-independent by construction: ranking fewer trials shrinks the ideal along with the DCG. Replaying the finished runs at reduced shortlist depth (re-ranking from cached CoT, no inference) shows it moving the WRONG way as depth grows, on all four runs. TREC 2021 / qwen36_medcpt: depth shortRec ndcg@10 ndcg_full@10 P@10(elig) 10 0.0787 0.8948 0.6028 0.3853 50 0.2565 0.8134 0.7916 0.6800 100 0.3924 0.8210 0.8113 0.7360 196 0.5774 0.8215 0.8197 0.7680 ndcg@10 reads BEST at depth 10, where the shortlist holds 8% of the relevant trials. Meanwhile ndcg_full@10 (+0.2169) and P@10(eligible) (+0.3827) both rise strongly with depth, because they are normalized against the full judged pool and so count a trial that never entered the shortlist as a miss. Same pattern on h100_baichuan/21, qwen36/22 and l40/23. So the funnel premise holds: depth does reach the top ten. It is simply invisible in the metric the comparison led with. ndcg_full@10 and P@10(eligible) move to QUALITY; ndcg@10 moves to DIAGNOSTIC with the reason recorded beside it. The replay also shows clear diminishing returns -- 25->50 buys +0.051 ndcg_full@10 while 100->196 buys +0.008 -- so the deeper arm should be expected to gain modestly, not doubled. TREC 2023 is the exception and is still climbing steeply at its full depth (shortlist recall only 0.254), which makes it the strongest case for more depth rather than 2021.

… claims First tool-grounded step of the agent: the reasoner's per-criterion verdicts are now checkable against the deterministic constraint engine instead of being taken on trust. The eligibility stage decides Met / Not Met / Violated by reading criterion text, which is where language models are weakest -- comparing an age or a lab value against a numeric threshold. GPT-4 scores ~51% on MedCalc-Bench with chain-of-thought and 81-85% when given the computation instead; AgentMD reports 87.7% against 40.9% for CoT on the same shape of task. TrialMatchAI already had a deterministic constraint engine, but it was consulted only during second-level retrieval scoring, blended in at weight 0.25 -- the reasoner never saw it. verify_trial_output() re-evaluates each criterion's constraints against the patient and compares the verdict with what the reasoner claimed, reporting every disagreement. The split is the whole design. Deterministic verdicts are authoritative ONLY for age, sex, lab and performance_status -- decidable comparisons against a structured patient value. Condition, phenotype, medication, procedure and biomarker identity stay with the reasoner, which beats a regex at them. That is the neuro-symbolic division alphaNeSy-CTM measured on TREC CT 2021-2023: adding a symbolic verifier moved specificity -- the ability to correctly REJECT a trial -- from 24.7% to 75.7%, while the agentic loop around it added 1.4 points. The verifier is where the value is, so it lands before any loop. Deliberately conservative. It abstains unless a constraint is decisive (status matched or violated, confidence >= 0.75); an unknown patient age decides nothing; a criterion whose text does not pair with a stored criterion is left alone rather than guessed at; malformed reasoner output cannot raise. Corrections record ReasonerClassification and VerifiedBy so a corrected run stays auditable, and apply_corrections=false gives a measure-only mode to size the effect before letting it act. Off by default (verification.enabled). Calls no model, so it is testable and tunable without a GPU -- being replayed over the completed runs now.

…on TREC Two findings from replaying the verifier over the completed runs, no inference. 1. The first version was UNSOUND. It asserted the positive label whenever every authoritative constraint matched. But a criterion is a conjunction and the verifier sees only its authoritative part: refuting the age clause of "age >= 18 with confirmed glioma" refutes the whole criterion, while matching it proves nothing about the glioma clause. That produced 484 "unclear -> met" and 322 "not met -> met" flips on TREC 2021 and cost -0.0098 ndcg_full@10 / -0.0133 P@10(eligible). It now abstains unless something is actually refuted; two regression tests pin the asymmetry. 2. Corrected, it is flat, and the reason is structural: TREC 2021 671 disagreements (4.6/100) ndcg_full@10 +0.0002 P@10(elig) -0.0040 TREC 2022 535 disagreements (6.9/100) ndcg_full@10 -0.0007 P@10(elig) -0.0020 TREC 2023 3 disagreements (0.0/100) no change Nearly every firing constraint is sex (473) or age (226) -- and hard_filters already enforces both at retrieval, so a trial whose age or sex excludes the patient never reaches the reasoner. The verifier is re-litigating a filter that already ran, against criterion prose rather than structured trial fields. Meanwhile lab and performance_status almost never fire, because TREC profiles are narrative summaries with no structured lab or ECOG values: 3 disagreements across 8,987 trials on 2023. The ceiling is set by the data, not the logic. Kept off by default. Still worth having for deployments with real EHR records, where labs and performance status exist and the filters may be looser -- but it is not a benchmark win and should not be sold as one. Also corrects an overreach in my earlier framing: alphaNeSy-CTM's specificity numbers come from a balanced binary eligible/ineligible task (100/100 per year), not corpus ranking, so they never predicted an ndcg/P@10 gain here. Its missing half -- LLM abduction of structured patient attributes from sparse notes -- is what would give a verifier material to work with on narrative data.

…sured and is worse The hard disqualification looks indefensible on inspection -- one Violated label out of a mean 9 (max 63) exclusion criteria irreversibly sinks a trial, and it fires on 19% of trials the TREC judges rated ELIGIBLE, 25% on 2023. That invites a future "fix". It was measured by replaying the completed runs from cached CoT, and every relaxation loses: rule 2021 ndcg_full@10 / P@10(elig) 2022 2023 >=1 violated (kept) 0.8197 / 0.7680 0.7481/0.6940 0.8806/0.8270 >=2 violated 0.7907 / 0.7280 0.7317/0.6760 0.8592/0.7676 >=3 violated 0.7839 / 0.7147 0.7293/0.6760 0.8515/0.7568 never disqualify 0.7837 / 0.7133 0.7295/0.6780 0.8502/0.7541 The rule pays because disqualification is ~2.3x more likely on an irrelevant trial than an eligible one (44% vs 19% on 2021): removing true violations buys more precision than the false ones cost. Reading the disagreements also shows many are not model errors but genuine clinical ambiguity -- the reasoner excluding a patient for prior bevacizumab under "no prior biologic therapy" is defensible even though the assessor graded that trial eligible. Comment only; no behaviour change.

…/No mode Two defects in the reranker prompt, both cheap to fix, both pointing at the same measured symptom: shortlist_selection_delta is negative on every completed run, i.e. the second level selects WORSE than a plain first-level cut at the same depth. 1. It asks the wrong question. The prompt asks whether the patient text contains "sufficient information to evaluate" the criterion -- answerability, not relevance and not eligibility. It correlates with relevance (a matching trial's criteria are more discussable) but is a different quantity, and the downstream 0.5 cut in aggregate_to_trials therefore discards trials whose criteria the patient text merely fails to mention, however eligible the patient is. 2. It answers in a binary that does not aggregate. Criterion scores are summed and averaged into a trial score, so magnitude matters as much as order, and binary Yes/No logprob is the formulation known to lack that discrimination. Three independent sources converge: Setwise (SIGIR 2024) measures pointwise Yes/No at 0.386 BEIR nDCG@10, BELOW BM25's 0.436; "Don't Overthink Passage Reranking" (2025) shows it collapsing the partial-relevance band to ~0% of scores in 0.1-0.9 against 11.4% for a well-behaved reranker; ERank and TFRank (2025) were both built specifically because it "lacks the necessary scoring discrimination". "Beyond Yes and No" (NAACL 2024) measured the fix at +2.2 BEIR average and +7.1 on SciFact, for one forward pass and zero generated tokens. Adds LLM_reranker.scoring: binary (default) P(Yes) over Yes/No -- historical behaviour, byte-identical prompt graded Expected Relevance Value sum(p_k * y_k) over Not/Somewhat/Highly, normalized to [0, 1], with a prompt that asks for relevance Deliberately kept as one switch: the graded label space is what makes the relevance question aggregable, so splitting them would produce a scale the 0.5 threshold cannot read. Same cost -- one forward pass, one token, no generation. Default is unchanged and the binary prompt is byte-identical, because the LoRA adapter was tuned against that exact wording; tests assert both. _yes_probability falls back to binary when label_token_ids is absent, so the pre-existing test_model_helpers contract test passes unmodified, which is the evidence the old path is untouched. Not yet validated on GPU. It needs only the 2B reranker, not the 35B CoT model, so a second-level-only ablation is cheap: measure how many judged-relevant trials survive to the shortlist under each mode.

…oning budget per trial Two hardcoded constants were the pipeline's binding bottleneck, and neither was reachable from config. SecondStageRetriever.size defaulted to 250 criteria PER QUERY and main.py never passed it. The TREC 2023 candidate pool is ~1,860 trials x ~17 criteria ~= 31,000 criteria, so the second level examined about a tenth of its own pool. Measured on 37 patients, retrieval only, no GPU: size unique trials % of pool recall of judged-relevant 250 803 43.1% 0.5849 <- old default 500 1,041 55.8% 0.7135 1000 1,200 64.3% 0.7870 2000 1,345 72.1% 0.8352 4000 1,477 79.2% 0.8507 That recall is a CEILING: the reranker and the 0.5 aggregation cut can only drop trials from what retrieval surfaced, never add them. Every knob tuned so far -- shortlist divisor, max_trials_rag, the depth policy -- sat behind it. Raising size to 1000 lifts the ceiling 0.585 -> 0.787, at the cost of retrieval plus 2B-reranker scoring; it does not touch the 35B eligibility model. aggregate_to_trials' 0.5 threshold was the second such constant: a trial whose every criterion scores below it never reaches the shortlist at all. Adds search.second_level.{per_query_size, aggregation_threshold, aggregation_method}. Also adds rag.adaptive_token_budget, sizing the eligibility output budget to each trial's criterion count rather than giving every trial the full vllm.max_new_tokens. m1 (arXiv:2504.00869) measures an optimal MEDICAL reasoning budget near 4K tokens "beyond which performance may degrade due to overthinking" -- but that is for a single question, while this stage judges every criterion of a trial in one call and trials range from a handful of criteria to 60+. A flat 4K cap would truncate the large ones into invalid JSON, so the budget scales with the work: a ~17-criterion trial lands near 4K, a 60-criterion trial keeps the full ceiling. It is clamped to max_new_tokens, so it can only ever LOWER the budget -- no trial is truncated more than it already would be. All defaults reproduce previous behaviour exactly, verified against both shipped configs, so every change here is an explicit A/B. aggregate_to_trials keeps working on an instance built without __init__ (the pre-existing test_audit_fixes contract), which is the evidence the old path is untouched.

…nfig _track_config assigned the TREC funnel unconditionally: search["max_trials_first_level"] = 1000 search["max_trials_second_level"] = 500 search["second_level_keep_divisor"] = 1 rag["max_trials_rag"] = 250 Whatever the config file said was discarded, with no warning. That made funnel experiments impossible to run and produced results that looked like findings: * A run configured for a 600-trial shortlist got 250. The "depth 600" TREC 2023 run therefore never changed depth at all -- its shortlist came out at 243, which I attributed to the second-level pool binding. It was this line. * A config asking for max_trials_second_level=1000 got 500. The width A/B was measuring a truncation constant: at per_query_size=1000 the second level aggregated 1,149 trials and at 250 it aggregated 573, and BOTH were cut to exactly 500 before the shortlist saw them. Across 120 completed patients, 56% land on exactly 500 and nothing ever exceeds it. So this cap sat upstream of every knob tuned so far -- per_query_size, the aggregation threshold, the shortlist divisor, max_trials_rag -- which is a large part of why each of them moved so little. The preset is now a DEFAULT rather than an override: each value is applied only where the base config left it at the schema default, i.e. the user expressed no opinion. setdefault alone is insufficient because a shipped config always carries these keys, so the preset would never apply at all. The resolved funnel is logged with which values came from the preset and which from the config, so this can never again be silent. Note this changes the effective funnel for the shipped configs, which set max_trials_second_level=1000: TREC runs will now honour that instead of 500. That is the intended fix, but it means new runs are not comparable to earlier ones without re-baselining.

…m_expansion, widen the funnel llm_expansion was a SECOND llm pass over a summary the FIRST pass had already written. It read keywords.json and handed the same aliases back. Sampled over TREC 2023 patients: patient terms searched already in keywords.json genuinely new trec-20231 15 12 Glaucoma, Open-Angle Glaucoma, Ocular Hypertension trec-202310 8 5 GAD, Mental Disorders, Neurobehavioral Manifestations trec-202311 17 15 Lung Diseases, Respiratory Tract Diseases trec-202312 15 13 Lung Disease, Respiratory Disease TOTAL 55 45 (82%) 10 (18%) 82% duplication, and every genuinely new term is a BROADER CATEGORY. Re-searching terms that already drive primary_condition (weight 1.0) and other_condition (0.25) adds nothing -- RRF just reinforces trials already found. So the categories are generated by the one expansion call instead, and the module is gone: * query_expansion gains "broader_conditions" (<=8) and "discarded_or_uncertain" (<=20). The latter preserves the one genuinely good behaviour of the deleted prompt: it put "No Prior Cataract Surgery" AND the un-negated "Cataract Surgery" in the discard list, because searching the bare term retrieves exactly the wrong trials. No other channel has that. * The planner routes broader_conditions to the existing broader_disease channel at weight 0.35, replacing hand-coded ontology rollups (kept as fallback when expansion is off). Weight matters: folding broad categories into main_conditions would search them at 1.0 alongside the specific diagnosis and flood the pool. * FirstLevelQueryExpander, build_first_level_expander, LLMQueryExpansion, LLMQueryExpansionBackend, parse_llm_query_expansion, the llm_expansion channel kind and weight, and the llm_expansion_enabled / llm_max_terms knobs are all removed, along with the dead keys in all 14 shipped configs. One LLM call per patient instead of two. Also widens the funnel, per the measured ceilings. First-level recall plateaus at 0.9110 with the pool exhausting at ~2,525 trials, but the stages behind it were far narrower: search.second_level.per_query_size 250 -> 1000 (surfaces 64% of the pool at 0.787 recall, vs 43% / 0.585 at 250) search.max_trials_second_level 1000 -> 2000 (must exceed what aggregation produces or it truncates -- this is the constant that silently capped every run at 500) rag.max_trials_rag 300 -> 500 (TREC 2023 patients carry ~604 judged relevant trials; 250 capped shortlist recall at ~41% regardless of model) max_trials_rag doubles chain-of-thought cost per patient. That is the deliberate trade: the shortlist was the binding constraint on every recall-aware metric.

…cripts benchmarks/ holds local embedder measurements rather than source, and should not have been committed. Untracked with --cached so all 11 files stay on disk; the directory is now gitignored, as are the per-run analysis outputs scripts/ writes to the repo root (candidate-id dumps run to megabytes) and the experiment result trees. Adds the two analysis scripts that produced this session's retrieval findings: llm_expansion_ab.py runs one arm at a time and PERSISTS each arm's per-patient candidate lists, so an arm is never re-run to answer a later question and a crash keeps completed patients. Both properties came from real failures: a depth sweep discarded its id lists and had to be regenerated, and an earlier A/B lost a finished arm when the job died. It also must run as a file rather than a heredoc -- once the embedder initialises CUDA, vLLM falls back to spawn multiprocessing, and spawn re-imports __main__, which fails for a script piped to `python -`. show_llm_expansions.py prints what the expander actually searched for. Inspecting four patients with it is what showed 82% of llm_expansion's terms were already in keywords.json, which is why that module was removed rather than tuned.

Copilot AI lite review requested due to automatic review settings August 6, 2026 09:25

Copilot started reviewing on behalf of majdabd August 6, 2026 09:26 View session

Copilot AI reviewed Aug 6, 2026

View reviewed changes

majdabd changed the title ~~feat(eval): expose the shortlist funnel loss in TREC metrics~~ feat: measure the shortlist funnel loss, then size the shortlist per patient Aug 6, 2026

majdabd added 11 commits August 6, 2026 14:31

Navigation Menu

Uh oh!

feat: measure the shortlist funnel loss, then size the shortlist per patient - #30

feat: measure the shortlist funnel loss, then size the shortlist per patient#30
majdabd wants to merge 13 commits into
mainfrom
feat/agentic-matching

majdabd commented Aug 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

majdabd commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. feat(eval): expose the shortlist funnel loss

2. feat(search): per-patient shortlist depth

Tuning

Two effects, reported separately

Safety

Verification

Not in this PR

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

majdabd commented Aug 6, 2026 •

edited

Loading

1. `feat(eval)`: expose the shortlist funnel loss

2. `feat(search)`: per-patient shortlist depth