Context
Per-simulation seeding from #1054 makes a single simulate() run reproducible, but continuing a run across calls with append=True is a separate piece that still has two gaps. This came up while reviewing #1054, and I am filing it on its own so that PR can stay scoped to the per-index seeding in #1053.
The two gaps
The root seed is not persisted. simulate() re-derives the seed root from random_seed on every call, so:
mc.simulate(100, random_seed=42)
mc.simulate(200, append=True) # random_seed defaults to None -> a fresh root
the second call draws a new root, and indices 100..199 belong to a different random draw than the first 100. Passing a different seed on the append is worse, since the output then mixes samples from two unrelated roots with no warning.
The next index comes from a row count. _initial_sim_idx is num_of_loaded_sims, which counts rows in the output file. If a parallel run is interrupted and leaves a gap (indices 0, 2, 3 written, 1 missing), the append starts at index 3: index 3 repeats and index 1 is never filled. Workers claim an index before running and finish out of order, so a gap after a failure is a real possibility rather than a corner case.
A possible direction
Persist the root state (plus a small schema version) in a sidecar next to the output. On append=True, load that root instead of deriving a new one, take the next index from the set of indices actually present rather than a row count, and reject an append whose random_seed disagrees with the stored root. Output files written before this exists would need an explicit migration or a re-baseline.
Happy to take this on as a follow-up to #1054.
Context
Per-simulation seeding from #1054 makes a single
simulate()run reproducible, but continuing a run across calls withappend=Trueis a separate piece that still has two gaps. This came up while reviewing #1054, and I am filing it on its own so that PR can stay scoped to the per-index seeding in #1053.The two gaps
The root seed is not persisted.
simulate()re-derives the seed root fromrandom_seedon every call, so:the second call draws a new root, and indices 100..199 belong to a different random draw than the first 100. Passing a different seed on the append is worse, since the output then mixes samples from two unrelated roots with no warning.
The next index comes from a row count.
_initial_sim_idxisnum_of_loaded_sims, which counts rows in the output file. If a parallel run is interrupted and leaves a gap (indices 0, 2, 3 written, 1 missing), the append starts at index 3: index 3 repeats and index 1 is never filled. Workers claim an index before running and finish out of order, so a gap after a failure is a real possibility rather than a corner case.A possible direction
Persist the root state (plus a small schema version) in a sidecar next to the output. On
append=True, load that root instead of deriving a new one, take the next index from the set of indices actually present rather than a row count, and reject an append whoserandom_seeddisagrees with the stored root. Output files written before this exists would need an explicit migration or a re-baseline.Happy to take this on as a follow-up to #1054.