feat: port the shared-memory transport onto the WorkerChannel abstraction#28
Open
mdashti wants to merge 2 commits into
Open
feat: port the shared-memory transport onto the WorkerChannel abstraction#28mdashti wants to merge 2 commits into
mdashti wants to merge 2 commits into
Conversation
mdashti
force-pushed
the
moe/shm-on-protocol-grpc
branch
from
5b1d213 to
9e7210d
Compare
mdashti
force-pushed
the
moe/embedder-api
branch
from
1728fb5 to
328e142
Compare
mdashti
force-pushed
the
moe/shm-on-protocol-grpc
branch
from
9e7210d to
992673d
Compare
mdashti
force-pushed
the
moe/shm-on-protocol-grpc
branch
from
992673d to
ea63b5e
Compare
stuhood
force-pushed
the
moe/shm-on-protocol-grpc
branch
from
ea63b5e to
aa130c2
Compare
mdashti
force-pushed
the
moe/shm-on-protocol-grpc
branch
from
aa130c2 to
3f1137b
Compare
mdashti
force-pushed
the
moe/shm-on-protocol-grpc
branch
from
3f1137b to
57188a1
Compare
mdashti
force-pushed
the
moe/shm-on-protocol-grpc
branch
2 times, most recently
from
6a488c9 to
34dd3f8
Compare
mdashti
force-pushed
the
moe/shm-on-protocol-grpc
branch
from
34dd3f8 to
c85e6dd
Compare
mdashti
force-pushed
the
moe/protocol-grpc-shm
branch
from
903cd33 to
9270c0f
Compare
mdashti
force-pushed
the
moe/shm-on-protocol-grpc
branch
from
c85e6dd to
eea4f1c
Compare
mdashti
force-pushed
the
moe/protocol-grpc-shm
branch
2 times, most recently
from
9940349 to
be44b9c
Compare
mdashti
force-pushed
the
moe/shm-on-protocol-grpc
branch
from
eea4f1c to
bc12d49
Compare
mdashti
force-pushed
the
moe/protocol-grpc-shm
branch
from
be44b9c to
0dfe291
Compare
The shm ring/DSM/mesh/framing core moves verbatim from the fork; the old WorkerTransport umbrella is gone, so the resolver hands out a channel and execute_task reads the rings per partition. The mesh is the no-gRPC transport, so it builds in both the grpc-on and grpc-off configs. Workers are passive executors: a fragment arrives with its nested stages already Remote, so the worker runs it as-is and its boundary leaves read the mesh; nothing on the worker converts or dispatches. collect_task_metrics reports an empty task-level metric set instead of a missing one, since one failed decode starves the whole store.
ShmWorkerChannel::coordinator_channel now ships each stage's SetPlanRequest to the worker proc that owns its task, as a SetPlan frame the worker takes with take_set_plan, so the embedder no longer routes plans out of band.
mdashti
force-pushed
the
moe/shm-on-protocol-grpc
branch
from
bc12d49 to
c58122a
Compare
mdashti
changed the base branch from
moe/protocol-grpc-shm
to
moe/transport-extension-points
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR ports our shared-memory ("shm") transport onto the
WorkerChannelabstraction.It stacks on the transport-completion PR (#27). The base is that branch, so the diff is only
src/shm/*plus themod shmdeclaration and its two Cargo deps. The transport plugs in with no core changes, which is the point.What
src/shm/*(rings, DSM, mesh, framing) moves over verbatim. The oldShmMqWorkerTransport: WorkerTransport+ShmMqWorkerConnection: WorkerConnectionbecomeShmChannelResolver: ChannelResolver+ aShmWorkerChannel: WorkerChannel:execute_taskreads the rings keyed off the task;coordinator_channelroutes eachSetPlanRequestto the destination worker's inbox over the mesh, so the coordinator's dispatch is the plan-delivery path.Workers are passive executors, same as the gRPC model: they run the ready-to-run per-task plans the coordinator dispatches (task-specialized, nested stages
Remote), whose boundary leaves read the mesh. Nothing on the worker re-plans, converts, or dispatches.Verified
The in-crate
in_processtest runs real distributed queries through the shm rings (gather, shuffle, early-cancel, producer-loss), green in both feature configs, under the no-gRPC CI job inherited from #27. It exercises #27's extension points end to end: the dispatch handoff throughDispatchPlanSource, the push-side placement throughroute_partition, and the metrics drain throughmetrics_store+decode_task_metrics.self_hostedis deferred and gated out; its no-gRPC-default role is the in-process transport's and its ring safety-net role is this test.Consumed by
pg_searchruns its MPP on this transport (paradedb#5516), which is what exercises it end to end.