Isolate each client's state in a job

[chrislupp]

Closes #76.

Important

Blocked on MDO-Standards/Philote#24. The committed gRPC stubs are generated from the .proto definitions in that PR. Before this merges, #24 has to land and the proto submodule pointer here has to be bumped to the resulting commit — the pointer currently reads v0.8.0 deliberately, so that this branch never references a commit the standard repo cannot serve.

The bug

A server holds one discipline instance and shares it across every client. The protocol has each client run a setup sequence before it computes, and that sequence mutates the shared instance:

1. Client A calls Setup; the server runs _clear_data() and rebuilds _var_meta.
2. A calls GetVariableDefinitions and caches the metadata.
3. Client B calls Setup; _clear_data() discards _var_meta and rebuilds it.
4. A calls ComputeFunction; the server preallocates from B's metadata and reads A's arrays into those buffers.

If the two declared different variables, A either aborts with INTERNAL on a KeyError or — worse — has a short array read into a longer buffer and receives a zero-padded result with no error. An optimizer consumes that happily.

examples/rosenbrock.py is the sharp case, because its variable shape comes from an option, so whichever client called SetOptions last fixed the shapes both clients got. The same collision applies to SetStreamOptions and to the in-place shape edits in SetVariableShapes.

The worst instance is openmdao/group.py: OpenMdaoSubProblem is a server-side discipline holding a live om.Problem, which is not thread-safe. Concurrent clients there are not merely reading stale data.

The change

A job is a session owning one discipline instance. Servers take a factory and build one discipline per job, so state an author keeps on self — a mesh, a solver, an om.Problem — is private to one client.

Jobs are session-shaped rather than request-shaped: few of them, each alive across thousands of evaluations, each holding real resources. That is what selects instance-per-job over a stateless handler with an explicit context argument, and it is why every existing discipline hook signature is unchanged. examples/rosenbrock.py needed zero edits and its collision is gone.

The job id rides in a philote-job-id metadata header, attached by a client-side channel interceptor so no call site passes it. Clients start a job lazily on the first call that needs one, so existing scripts and both OpenMDAO components work unmodified.

Breaking change

ExplicitServer(discipline=Paraboloid())  # before: an instance
ExplicitServer(discipline=Paraboloid)    # after:  the class

The keyword is unchanged but its meaning is not, which reads as a trap. It fails safely, though: a Discipline instance is not callable, so the old call raises TypeError immediately rather than doing something subtly wrong, and the message names both forms —

discipline must be a zero-argument callable returning a Discipline, got an instance of Paraboloid. Pass the class rather than an instance of it — ExplicitServer(discipline=Paraboloid), not ExplicitServer(discipline=Paraboloid()). The server builds one discipline per job, so it needs something it can call.

A class works directly when its initialize() does its own configuration; a discipline configured externally needs a closure or functools.partial. attach_discipline() keeps its name and now takes the same callable.

How to review this

37 files, but the split is lopsided — +1223/−136 source, +1103/−266 tests, plus 85 generated lines.

Start with philote_mdo/general/job.py (new: Job, JobStore), then _resolve_job in discipline_server.py and the self._discipline → job.discipline substitution across the three server files. The client side is one __init__ change plus start_job/end_job/keep_alive.

The test churn is two mechanical patterns, both explained in the new tests/conftest.py: 43 tests called RPC handlers with a Mock() or None context that cannot produce metadata, and ~45 configured server._discipline, which no longer exists. tests/test_jobs.py is the new coverage and is the interesting file.

Concurrency: what this actually buys

Separate jobs may evaluate at the same time, with no global lock in the path, but the GIL decides whether that yields throughput. Measured with four concurrent clients:

discipline does...	1 client	4 clients	speedup
pure Python	278 ms	1107 ms	1.0×
NumPy A @ A	29 ms	138 ms	0.8× (threaded BLAS already saturates cores)
compiled solver releasing the GIL	306 ms	310 ms	4.0×

Jobs buy correctness unconditionally and throughput conditionally. Disciplines wrapping a compiled solver get real parallelism; pure-Python ones — including OpenMdaoSubProblem, whose run_model() is mostly Python — become correct under concurrent clients rather than faster. This is stated plainly in the docs so nobody expects a speedup that cannot exist.

Related: max_jobs is not the cap that binds — the gRPC thread pool is, since every in-flight RPC holds a worker for its whole duration. The server warns at startup when the pool is smaller than the job limit.

Also fixed here

Found while building this, both with regression tests:

• Only the compute calls translated gRPC errors, so an unknown job during run_setup escaped as a raw grpc.RpcError. All base-client calls now translate, and NOT_FOUND / RESOURCE_EXHAUSTED map to distinguishable exceptions so a driver can tell "start over" from "retry later".
• JobStore.create() reclaimed expired slots without running teardown_job(), leaking whatever the job held whenever it beat the sweeper.

Testing

359 tests pass locally on 3.12. Beyond the suite I verified: the reproduction above returns correct results for concurrent clients at dimension=2 and dimension=10; both example server/client pairs and the OpenMDAO example run with unmodified clients; a max_jobs=2 server refuses the third job and frees the slot on EndJob; an idle job is evicted and teardown_job() fires; RSS grew under a megabyte across 300 job cycles.

Not verified locally: Python 3.9. All files parse against the 3.9 grammar and there is no 3.10+ syntax, but the 3.9 interpreter here lacks scipy so the suite could not run. CI settles it.

Also not run: the Docusaurus build. docs/docs/getting-started/quickstart.md gains a Jobs section and tutorials/implicit-disciplines.md is updated for the factory, but a broken admonition would only surface in the docs workflow.

Merged from develop

Three merges from develop while this was in progress, all touching files this branch rewrote: #81–#84 (the four bugs found while planning this work) and #86 (deriving DisciplineServer from the servicer base). Two conflicts, both resolved as the union of intents rather than by choosing a side — see the merge commit messages. After #86 I verified all eleven DisciplineService RPCs resolve to this package's implementations rather than to the base class's UNIMPLEMENTED stubs, since that base now supplies defaults where the old one supplied none.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[chrislupp]

To be clear, the code should be usable/testable because the stubs were generated from the updated protocol. However, regenerating the stubs will lead to failures with potentially confusing error messages. We will need to close out the protocol PR and release it before finishing this PR.