Add per-job state isolation so one discipline server can support concurrent clients

[chrislupp]

Problem

A Philote server holds one discipline instance and shares it across every client that connects. The protocol requires each client to run a setup sequence before it computes, and several steps in that sequence mutate the shared instance. Two clients against one server therefore corrupt each other.

The interleaving that breaks:

1. Client A calls Setup. The server runs _clear_data(), setup(), setup_partials() and builds _var_meta (discipline_server.py:155).
2. Client A calls GetVariableDefinitions and caches the metadata locally.
3. Client B calls Setup. _clear_data() discards _var_meta and rebuilds it from B's options.
4. Client A calls ComputeFunction. The server preallocates input buffers from B's metadata (discipline_server.py:245) and reads A's arrays into them.

If A and B happen to declare identical variables, step 4 is harmless. Otherwise A either aborts with INTERNAL on a KeyError, or — worse — writes a short array into a longer buffer and returns a result padded with zeros. That path is silent.

philote_mdo/examples/rosenbrock.py is the sharpest instance of this, because its variable shape is derived from an option:

def initialize(self):
    self.add_option("dimension", "int")

def set_options(self, options):
    self.dimension = int(options["dimension"])   # writes to the shared instance

def setup(self):
    self.add_input("x", shape=(self.dimension,))

Client A requests dimension=2 and client B requests dimension=10. Whichever calls SetOptions last fixes the shapes that both clients get.

The same interleaving applies to SetStreamOptions (discipline_server.py:95) and to the in-place shape mutation in SetVariableShapes (discipline_server.py:172).

This blocks two deployments we want: one analysis server shared by several concurrent optimizations, and one server evaluating several design points in parallel for a gradient-free optimizer.

Proposed design

Introduce a job: a server-side session that owns one discipline instance and the scratch storage that goes with it. A client starts a job, receives a job id, and presents that id on every subsequent call. Jobs are independent, so concurrent clients no longer interfere.

Jobs are session-shaped rather than request-shaped. We expect few of them (order 2–8), each alive across thousands of compute calls, each holding real resources — a mesh, an initialised solver, restart files. That profile argues for one discipline instance per job rather than a stateless handler with an explicit context argument. It also means every discipline hook signature stays exactly as it is today, and domain state stays where authors already put it, on self.

The job carries identity and lifecycle only:

class Job:
    job_id:      str
    workdir:     Path                  # <root>/<job_id>
    discipline:  Discipline            # from the factory; server sets discipline.job = self
    stream_opts: StreamOptions         # moves off the server; it is per-client
    state:       NEW | SETUP | READY | CLOSED
    lock:        threading.Lock
    last_used:   float

Variable metadata, partials metadata and options need no new home. They already live on the discipline instance, and that instance is now per job.

Transport

The job id travels as a gRPC metadata header, philote-job-id, not as a message field.

Measured cost is 7.7 µs on a unary call and 13.9 µs on a stream. HPACK indexes the repeated value into the dynamic table, so the id costs a byte or two on the wire after its first appearance on a connection. The server can read the header before it consumes the request iterator, which was verified and is what makes this work for the four streaming RPCs.

A proto field was considered and rejected. All the evaluation RPCs are stream VariableMessage, so a field would be re-serialised on every chunk of every array while only the first occurrence was ever read. The unary RPCs all take google.protobuf.Empty today, so a field would require inventing a request message for each one.

Lifecycle

Two new RPCs on DisciplineService:

message JobHandle { string job_id = 1; }

rpc StartJob(google.protobuf.Empty) returns (philote.JobHandle) {}
rpc EndJob(google.protobuf.Empty) returns (google.protobuf.Empty) {}   // id in metadata

Setup does not return the job id. SetStreamOptions and SetOptions are per-job state and run before Setup, so the job has to exist ahead of them.

StartJob is deliberately cheap. Setup may become expensive once it loads a mesh, so the two stay distinct.

The resulting call order also fixes shapes server-side, which is what dynamic_shape was reaching for:

StartJob -> UploadFile(mesh) -> SetOptions -> Setup -> GetVariableDefinitions -> compute...

Setup now runs after the mesh is resident, so a discipline can size its outputs from the mesh directly.

Requirements

R1. DisciplineServer, ExplicitServer and ImplicitServer accept discipline_factory=<callable> and construct one discipline per job.

R2. The existing discipline=<instance> constructor continues to work and binds to a single default job. A request carrying no philote-job-id header resolves to that default job, so existing clients, examples and tests are unaffected.

R3. Every RPC except GetInfo and GetAvailableOptions resolves a job from the header. Those two report properties of the discipline class and are job-independent.

R4. An unknown or expired job id returns NOT_FOUND. The client raises and does not transparently start a replacement job: the uploaded mesh would be gone, and the optimizer would proceed against un-uploaded geometry and return plausible but wrong results.

R5. The server enforces max_jobs and returns RESOURCE_EXHAUSTED from StartJob when the cap is reached. A leaked job now holds a mesh, so the failure has to be an explicit refusal rather than an OOM.

R6. An idle-TTL sweep evicts jobs whose last_used has aged out. A KeepAlive RPC lets a client hold a job open while an optimizer sits between design iterations.

R7. Discipline gains an optional teardown_job() hook. EndJob and TTL eviction both call it before removing the workdir, so authors can close file handles and release solvers.

R8. Discipline gains a self.job handle, giving authors self.job.workdir and self.job.job_id. No existing hook signature changes.

R9. Locking is per job. Calls within one job serialise; separate jobs run concurrently with no global lock.

R10. The Job enforces its state machine. SetOptions after Setup on the same job returns FAILED_PRECONDITION rather than being silently accepted against metadata already built from the previous value.

R11. philote_mdo/general/job.py provides Job and JobStore (mapping, lock, TTL sweep, cap).

R12. Tests cover: two concurrent clients with different rosenbrock dimensions; TTL eviction; max_jobs refusal; unknown job id; the SetOptions-after-Setup refusal; and the legacy no-header path.

R13. CHANGELOG.md gains an entry under [Unreleased].

Out of scope

Filed or deferred separately so this change stays reviewable and bisectable:

• File transfer. UploadFile / DownloadFile scoped by the same header, writing into job.workdir. This issue only has to establish the workdir and its cleanup. Chunk file transfers around 1 MiB, per Batch multiple variables per VariableMessage to cut per-message gRPC overhead #75.
• Option validation and defaults. Nothing currently checks the incoming Struct against options_list; SetOptions forwards it unvalidated (discipline_server.py:141), so the declared types serve GetAvailableOptions and nothing else. Adding type validation and add_option(default=) is worth doing and does not depend on jobs.
• Discrete-variable arity. The servers call compute() with two arguments or four depending on whether discrete variables exist (explicit_server.py:73, :126; implicit_server.py:179, :250). Collapsing that into always-passed keyword arguments is a separate change with its own test surface.
• Message batching. Batch multiple variables per VariableMessage to cut per-message gRPC overhead #75.

Notes

proto/disciplines.proto is the shared standard, so StartJob, EndJob, JobHandle and the philote-job-id header convention need to land in the Philote-MDO standard repo as well, and the committed stubs regenerate via python utils/compile_proto.py.