fix(backends): pin grpcio-tools to installed grpcio in runProtogen (protobuf gencode mismatch)#10735
Open
localai-bot wants to merge 1 commit into
Open
fix(backends): pin grpcio-tools to installed grpcio in runProtogen (protobuf gencode mismatch)#10735localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
runProtogen installed grpcio-tools unpinned, so the protoc it bundles stamped backend_pb2.py with the newest Protobuf gencode (7.35.0). When a backend caps the protobuf runtime lower -- vLLM pins protobuf to 6.33.6 -- the import-time guarantee runtime >= gencode fails: google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions ... gencode 7.35.0 runtime 6.33.6 The backend crashes on `import backend_pb2` before it can serve, which surfaces to the user as "grpc service not ready". It was mis-reported as a ROCm/gfx1201 failure in #10718 but is not GPU-specific and affects every vLLM variant (and any backend that caps protobuf below the latest gencode). Pin grpcio-tools to the grpcio version the backend already installed -- they release in lockstep -- so the generated gencode stays in step with the protobuf runtime. Falls back to unpinned when grpcio isn't present. Closes #10718 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
mudler
approved these changes
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The vLLM backend (all accelerations) fails to start with:
The backend crashes on
import backend_pb2before it can serve, surfacing to the user asfailed to load model with internal loader: grpc service not ready.Reported as a ROCm / R9700 gfx1201 failure (#10718), but it is not GPU-specific — the process never reaches any GPU code. It affects every vLLM variant (and any Python backend that caps the protobuf runtime below the latest grpcio-tools gencode).
Root cause
runProtogen(backend/python/common/libbackend.sh) runs after the backend's requirements are installed, and installsgrpcio-toolsunpinned:uv pip install grpcio-tools # pulls the newest releaseThe
protocbundled in the newest grpcio-tools stampsbackend_pb2.pywith Protobuf gencode 7.35.0. vLLM's dependencies pin the protobuf runtime down to 6.33.6. Protobuf enforcesruntime >= gencodeat import, so the generated module refuses to load.Fix
Pin
grpcio-toolsto thegrpcioversion the backend already installed — the two are released in lockstep, so the bundled protoc's gencode stays in step with the protobuf runtime. Falls back to unpinned whengrpcioisn't present (behaviour unchanged for those backends).Testing
bash -nclean;shellcheck -S warningreports nothing new on the changed lines (pre-existing findings elsewhere untouched).Closes #10718
🤖 Generated with Claude Code