Skip to content

fix(backends): pin grpcio-tools to installed grpcio in runProtogen (protobuf gencode mismatch)#10735

Open
localai-bot wants to merge 1 commit into
masterfrom
fix/protogen-grpcio-tools-pin
Open

fix(backends): pin grpcio-tools to installed grpcio in runProtogen (protobuf gencode mismatch)#10735
localai-bot wants to merge 1 commit into
masterfrom
fix/protogen-grpcio-tools-pin

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Problem

The vLLM backend (all accelerations) fails to start with:

google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf
Gencode/Runtime versions when loading backend.proto: gencode 7.35.0 runtime 6.33.6.
Runtime version cannot be older than the linked gencode version.

The backend crashes on import backend_pb2 before it can serve, surfacing to the user as failed to load model with internal loader: grpc service not ready.

Reported as a ROCm / R9700 gfx1201 failure (#10718), but it is not GPU-specific — the process never reaches any GPU code. It affects every vLLM variant (and any Python backend that caps the protobuf runtime below the latest grpcio-tools gencode).

Root cause

runProtogen (backend/python/common/libbackend.sh) runs after the backend's requirements are installed, and installs grpcio-tools unpinned:

uv pip install grpcio-tools   # pulls the newest release

The protoc bundled in the newest grpcio-tools stamps backend_pb2.py with Protobuf gencode 7.35.0. vLLM's dependencies pin the protobuf runtime down to 6.33.6. Protobuf enforces runtime >= gencode at import, so the generated module refuses to load.

Fix

Pin grpcio-tools to the grpcio version the backend already installed — the two are released in lockstep, so the bundled protoc's gencode stays in step with the protobuf runtime. Falls back to unpinned when grpcio isn't present (behaviour unchanged for those backends).

grpcio_version="$(python -c 'import importlib.metadata as m; print(m.version("grpcio"))' 2>/dev/null || true)"
[ -n "$grpcio_version" ] && grpcio_tools_spec="grpcio-tools==${grpcio_version}"

Testing

  • bash -n clean; shellcheck -S warning reports nothing new on the changed lines (pre-existing findings elsewhere untouched).
  • Fallback path verified (grpcio absent → unpinned, as before).
  • Full runtime validation requires a backend image build (ROCm/vLLM) — CI territory; cannot be reproduced locally.

Closes #10718

🤖 Generated with Claude Code

runProtogen installed grpcio-tools unpinned, so the protoc it bundles
stamped backend_pb2.py with the newest Protobuf gencode (7.35.0). When a
backend caps the protobuf runtime lower -- vLLM pins protobuf to 6.33.6 --
the import-time guarantee runtime >= gencode fails:

  google.protobuf.runtime_version.VersionError: Detected incompatible
  Protobuf Gencode/Runtime versions ... gencode 7.35.0 runtime 6.33.6

The backend crashes on `import backend_pb2` before it can serve, which
surfaces to the user as "grpc service not ready". It was mis-reported as a
ROCm/gfx1201 failure in #10718 but is not GPU-specific and affects every
vLLM variant (and any backend that caps protobuf below the latest gencode).

Pin grpcio-tools to the grpcio version the backend already installed --
they release in lockstep -- so the generated gencode stays in step with
the protobuf runtime. Falls back to unpinned when grpcio isn't present.

Closes #10718

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
@mudler mudler enabled auto-merge (squash) July 7, 2026 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VLLM backend failing GRPC rocm hipblas AI PRO R9700 AI TOP 32G gfx1201

2 participants