Add Polars-backed lazy parquet data structures with pluggable executor and materialization strategies

[aoustry]

Overview

Introduces a Polars-backed lazy loading path for time-series data stored as Parquet files, alongside a pluggable Executor for parallel subproblem execution and a MaterializationStrategy enum to control when and how much data is loaded into RAM.

New data structures (gems/study/data.py)

Three new AbstractDataStructure subclasses defer parquet I/O to get_value() call time:

Class	Parquet layout	Polars optimisation
LazyTimeSeriesData	1 column "0", T rows	row indexing deferred
LazyScenarioSeriesData	S columns "0"…"S-1", 1 effective row	.head(1) + column pruning
LazyTimeScenarioSeriesData	S columns, T rows	column pruning — only requested scenarios read

A fourth class handles the materialized case:

• MaterializedTimeScenarioSeriesData — holds a pre-loaded (T, n_cols) numpy array for a worker's scenario subset. Carries a pre-built dense lookup array so column remapping is a single vectorized numpy index rather than a Python loop.

All get_value implementations use np.unique(..., return_inverse=True) for duplicate-scenario deduplication, replacing the previous Python dict-loop pattern.

DataBase.materialize(mc_scenario_ids)

Returns a new DataBase with every lazy entry replaced by in-memory data. Only the data-series columns actually needed by the given MC scenario IDs are read from each parquet file (via ScenarioBuilder when present). Eager entries are carried over unchanged. The original DataBase is never mutated.

Parquet detection and backward compatibility

_build_data() checks for a .parquet file first; falls back transparently to the existing .txt / .tsv eager path when none is present. Both formats can coexist in the same directory — parquet always takes precedence.

MaterializationStrategy enum (gems/session/session.py)

Replaces the previous materialize_per_worker: bool field with a three-value enum:

Value	Behaviour
NONE (default)	Lazy I/O on every get_value call
UPFRONT	Full database loaded into RAM once before the run loop starts
PER_WORKER	Each scenario (sequential) or batch (parallel) materializes only its own columns

UPFRONT is the right choice when data fits comfortably in RAM and I/O is expensive relative to compute (e.g. network-attached storage). PER_WORKER is the right choice for distributed workers where minimal per-worker RAM footprint matters. NONE is appropriate when parquet files are local and small relative to the time horizon.

Pluggable executor (SimulationSession)

SimulationSession gains an optional executor: concurrent.futures.Executor field:

• Sequential mode — each scenario is submitted as an independent task; None runs them in the calling thread.
• Parallel mode — (block, scenario) work items are grouped into batches and submitted concurrently. _run_batch receives the (possibly pre-materialized) study as an explicit argument.

The lazy data structures are cheaply picklable (URI strings only), so remote workers (Ray, ProcessPoolExecutor) receive near-zero serialisation overhead.

Thread safety (Study.__post_init__)

Study.model_components and Study.models are cached_property fields. Pre-populating them in __post_init__ ensures concurrent workers only ever perform reads, eliminating the racy first-write through cached_property.__set_name__.

Tests

• tests/unittests/data/test_data.py — 13 new tests covering MaterializedTimeScenarioSeriesData (get_value, duplicates, missing keys, check_requirement) and DataBase.materialize (all lazy types, eager carry-over, immutability, selective columns, ScenarioBuilder mapping).
• tests/e2e/functional/test_parquet_dataseries.py — 6 tests: one per lazy type, parquet-takes-precedence, PER_WORKER, and UPFRONT.
• tests/e2e/functional/test_parquet_complex_study.py — 2 study-level comparison tests on the 7_4 study (12 components, 168 timesteps): parquet+tsv coexistence and parquet-only mode, both asserting numerical identity with the TSV reference run.

https://claude.ai/code/session_01GYyht7zSYAB5m4V9mL2cB5