Batch multiple variables per VariableMessage to cut per-message gRPC overhead

[chrislupp]

Summary

_assemble_input_messages emits at least one gRPC message per variable. get_chunk_indices splits within a variable but never packs across variables, so a discipline with many small variables sends a flood of near-empty messages while the 100k-double chunk budget goes almost entirely unused.

The fix is to batch multiple Arrays into a single VariableMessage until the num_double budget is reached, then flush.

Why it matters

Measured over a real gRPC channel (loopback), 1000 scalar variables:

	per round trip
1000 scalars, one message each (today)	58.27 ms
same data, one batched message	0.45 ms

That is ~58 µs of fixed cost per message — dominated by HTTP/2 framing, flow control, the Python generator step, and the thread handoff, not by serialization. For comparison, protobuf serialize+parse for those same 1000 messages is only 0.40 ms, so ~99% of the current cost is per-message overhead that batching removes.

Disciplines with a few large arrays are unaffected — those already fill their chunks. The win is entirely on wide, scalar-heavy disciplines.

Proposed change

Add a third variant to the VariableMessage oneof in proto/data.proto:

message ArrayBatch { repeated Array arrays = 1; }

Client (discipline_client.py, _assemble_input_messages) and servers (explicit_server.py, implicit_server.py) accumulate Arrays and flush when the budget is hit.

Properties worth preserving, all of which this keeps:

• messages stay self-describing (names remain on each Array), so a metadata mismatch is still a loud error rather than silent data corruption
• sparse / partial sends still work
• the encoding.py fast path applies per-Array unchanged

Notes / open questions

• Capability negotiation is required. An old peer receiving the new oneof variant sees WhichOneof("payload") == None and silently skips the data. Batching has to be gated — e.g. a flag on StreamOptions, which is already exchanged before any compute call.
• This is a change to the shared standard, so it needs coordination with the Philote-MDO proto repo and stub regeneration via python utils/compile_proto.py.
• Keep the 100k-double chunk default. A sweep at a 7.6 MiB payload gives 12.68 ms at 25k chunks, 7.99 ms at 100k, 9.78 ms at 500k — the current constant is at the optimum.
• Do not switch these RPCs to unary. Unary saves only a fixed ~0.3–0.5 ms of stream setup and loses to chunked streaming above ~1 MiB (10.85 ms vs 7.99 ms at 7.6 MiB) because streaming pipelines parsing against transfer. It would also reintroduce gRPC's 4 MiB default message ceiling, which the streaming design exists to avoid.
• Moving variable names out of the messages is not worth it. Names are ~45% of the wire bytes for scalar-heavy disciplines but only ~2% of the time (0.40 ms → 0.39 ms). Message count is the bottleneck, not name strings.

Related

Follows on from #74, which raised the chunk budget and bypassed protobuf's boxed-float path in encoding.py. That addressed per-byte cost; this addresses per-message cost.