ReCaRe — Retrieval Baselines

Retrieval baselines and full experimental code for the paper

ReCaRe: A Bilingual Legal Benchmark for Revision Candidate Retrieval. Takumi Ito, Yuma Kurokawa, Makoto P. Kato, Sumio Fujita.

The benchmark dataset itself is released at kasys/ReCaRe (DOI 10.57967/hf/8642). This repository provides the code, run scripts, prompts, results, and documentation for §4.2 Retrieval Experiments of the paper — BM25, dense retrievers, cross-encoder rerankers, RankGPT zero-shot listwise LLM reranking, query/document expansion, and dense-retriever domain adaptation.

The dataset construction pipeline (the scraping / alignment / validation code), and the characterization metrics (§4.1), are intentionally outside the scope of this public release. The construction methodology itself — amendment-rationale extraction, article-level alignment, qrel derivation, splits — is documented in docs/dataset_construction.md for reproducibility.

Contents

ReCaRe/
├── src/recare_baselines/   # Python package (CLI: recare-baselines)
├── scripts/                # End-to-end run scripts for each paper table
├── prompts/                # LLM prompt cards (expansion, RankGPT)
├── docs/                   # Dataset construction + reproduction guides
├── tests/                  # pytest unit + smoke tests
├── results/                # Per-cell metrics + result-summary markdown
└── data/expansion/         # LLM query expansions (test split, ~7 MB)

Document	What it covers
docs/dataset_construction.md	How ReCaRe was constructed from EUR-Lex + e-Gov: source selection, rationale extraction, article-level alignment (Dice ≥ 0.7 / Simpson ≥ 0.95), qrel derivation, event-level train/val/test split
docs/reproduction.md	Step-by-step paper-table reproduction (Tables 2–4)
docs/experiments.md	Per-method implementation details and runtime expectations
docs/smoke_test.md	CPU-only lightweight reproducibility check
docs/data_format.md	On-disk JSONL / metrics / runfile schemas

Quickstart

# 1. Install (Python 3.11; uv handles the venv)
git clone https://github.com/kasys-lab/ReCaRe.git
cd ReCaRe
uv sync

# 2. Smoke test: BM25 + me5-small dense on 5 queries per cell
#    (~10 min on a CPU laptop; no API keys required)
export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
bash scripts/smoke_test.sh

# 3. Run paper Table 2 BM25 row (all 4 cells, full test split)
bash scripts/run_bm25.sh

The HuggingFace dataset is fetched on demand; no manual download is needed. See docs/smoke_test.md for details.

Reproduction

Each script reproduces one block of the paper:

Paper	Script	What it does
Table 2 BM25 row	scripts/run_bm25.sh	Index EN/JA, run 4 cells
Table 2 short-context dense	scripts/run_short_dense.sh	mDPR / mContriever / mE5 × 4 cells
Table 2 long-context dense	scripts/run_long_dense.sh	BGE-M3 / jina-v3 × 4 cells (GPU)
Table 2 augmentation	scripts/run_expansion.sh	Q2E/Q2D/d2q/d2e × {BM25, jina-v3}
Table 3 cross-encoder rerankers	scripts/run_rerankers.sh	BGE / Jina / Qwen3-4B/8B × 4 cells
Table 3 RankGPT	scripts/run_rankgpt.sh	3 Azure LLMs × 4 cells (≈ $156)
Table 4 domain adaptation	scripts/run_domain_adaptation.sh	5 models × 4 cells, hard-neg → train → eval

See docs/reproduction.md for the full step-by-step guide, including expected runtimes, GPU requirements, and how to verify the reproduced numbers against the paper.

Results

All numerical results are committed in results/:

• results/baselines.md — Table 2 (BM25 + 5 dense retrievers)
• results/rerankers.md — Table 3 cross-encoder rerankers
• results/rankgpt.md — Table 3 RankGPT + Azure cost ledger
• results/augmentation.md — Table 2 augmentation block
• results/domain_adaptation.md — Table 4 with Holm-corrected paired t-tests

Per-cell metrics (188 JSON files) live under results/metrics/. The full results index is at results/README.md, which also maps every paper-table cell to its source JSON.

Beyond-the-paper experiments

The repository also retains supplementary experiments that were performed but not included in the paper for space reasons:

• The 0.6B Qwen3 reranker (smallest variant)
• max-P chunking strategies for dense retrievers (maxp-doc, maxp-q, maxp-both) as supplementary long-input ablations
• Full 7-metric set (R@{10,100,1000}, nDCG@{10,100,1000}, MAP) for every cell — the paper reports only Recall@100 (Table 2) and nDCG@10 (Tables 3-4)
• Both gpt-4.1-mini AND qwen3.5-9b query expansions for every family — paper reports the gpt-4.1-mini cells only
• Holm-corrected paired t-tests of every method vs BM25, and every augmentation vs jina-v3 (see results/ttest_holm/)

See results/README.md for the complete map.

Pre-generated artifacts on HuggingFace

Two large generated artifacts are hosted on HuggingFace rather than committed to this repo:

• d2e / d2q full-corpus expansions (~150 MB per language) → kasys/ReCaRe-expansions (dataset)
• Domain-adaptation checkpoints (5 base models × 4 cells = 20 adapters) → kasys-lab/recare-<base>-<task>-<lang> (models)

See docs/data_format.md for download commands.

Prerequisites

• Python 3.11
• uv for dependency management
• JDK 21 (for BM25 via Pyserini)
• CUDA-capable GPU strongly recommended for dense retrievers; required for Qwen3 rerankers and the 8192-token jina-v3 / BGE-M3 corpus encoding
• Azure OpenAI access only for RankGPT (Table 3 RankGPT rows) and LLM query/doc expansion generation (Table 2 augmentation block, Phase 1):
  • AZURE_OPENAI_API_KEY
  • AZURE_OPENAI_ENDPOINT
  • AZURE_OPENAI_API_VERSION

BM25, dense retrievers (Table 2), cross-encoder rerankers (Table 3 rows 1-4), and domain adaptation (Table 4) all run without any OpenAI / Azure credentials. The smoke test (scripts/smoke_test.sh) works on CPU and requires no API keys.

Citation

@misc{ito2026recare,
  title  = {ReCaRe: A Bilingual Legal Benchmark for Revision Candidate
            Retrieval},
  author = {Ito, Takumi and Kurokawa, Yuma and Kato, Makoto P. and
            Fujita, Sumio},
  year   = {2026},
  note   = {Manuscript under review}
}

The benchmark dataset has its own citation; see the ReCaRe HuggingFace dataset card.

License

• Code: MIT (see LICENSE)
• Results, prompts, documentation: CC BY 4.0
• Dataset (kasys/ReCaRe): CC BY 4.0 (EU statutory articles, e-Gov Japanese statutes) and CC0 1.0 (metadata); see the dataset card for per-component terms.