feat(llm): surface logprobs in /v1/chat/completions - #5252
Conversation
The chat completions path silently returned logprobs: null even when the client requested logprobs/top_logprobs, while /v1/completions already populated them for vLLM (xorbitsai#5245). The chat builders (_to_chat_completion, _to_chat_completion_chunk) omitted the field and the chat choice TypedDicts did not declare it, so the logprobs the engine already produced were discarded. Declare logprobs on ChatCompletionChoice / ChatCompletionChunkChoice and propagate choice.get('logprobs') in the two real choice-bearing chat builders. The chat path already receives logprobs-bearing CompletionChunks from the vLLM completions generator (which also slices per-chunk deltas for streaming), so the chat builder needs no new generation or slicing logic. Effective scope is vLLM (the only engine that builds logprobs today); other engines stay null until they implement logprobs (follow-up). Adds unit tests asserting chat completions carry the source logprobs, red on main (field absent) and green after propagation.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request ensures that logprobs are correctly propagated from completion choices to chat completion and chat completion chunk choices. It updates the relevant TypedDict definitions in xinference/types.py, includes the logprobs field in the conversion utilities in xinference/model/llm/utils.py, and adds corresponding unit tests in xinference/model/llm/tests/test_utils.py. I have no feedback to provide as there are no review comments.
qinxuye
left a comment
qinxuye
left a comment
There was a problem hiding this comment.
I found two remaining compatibility gaps in the chat logprobs path.
…ol paths Chat completions surfaced logprobs in the legacy CompletionLogprobs shape (text_offset/tokens/token_logprobs/top_logprobs), which openai-python parses as choice.logprobs.content == None -- standard chat clients still could not consume the requested logprobs. Add a chat-specific ChatCompletionLogprobs (content[] of token/bytes/logprob/top_logprobs) and convert the vLLM legacy shape at the chat builder boundary, for both streaming and non-streaming. Also preserve the converted logprobs through the tool post-processors: _post_process_completion previously omitted the field and _post_process_completion_chunk hard-coded it to None, so tool-enabled requests lost the generated logprobs even when requested.
|
Thanks @qinxuye — good catches, both addressed in bf70cc9.
Tests: updated the two propagation tests to assert the On CI: the two |
qinxuye
left a comment
qinxuye
left a comment
There was a problem hiding this comment.
The streaming tools path still loses logprobs, and the current test suite fails.
_async_to_tool_completion_chunks() converts each legacy CompletionChunk to a
chat chunk via _to_chat_completion_chunk(), which already turns the legacy
parallel-list logprobs into the chat content[] shape. _post_process_completion_chunk()
then re-ran _completion_logprobs_to_chat_logprobs() on that already-converted
value; with no "tokens" key to read, it returned {"content": []} and a real
one-token logprob collapsed to an empty list.
Pass c["choices"][0].get("logprobs") through unchanged there instead. The
non-streaming _post_process_completion() still converts, because it receives a
raw Completion (legacy shape) whose logprobs genuinely need converting.
Update the three non-streaming tool-path tests to expect "logprobs": None now
that the choice carries the field (consistent with the no-tools path), and
rewrite the streaming test to feed a real chat chunk (raw -> _to_chat_completion_chunk
-> post-processor) and assert the one-token logprob survives rather than being
double-converted to {"content": []}.
|
Thanks for the precise trace — both points addressed in cdf165b.
Local |
| { | ||
| "token": token, | ||
| "bytes": _token_logprob_bytes(token), | ||
| "logprob": logprob, |
There was a problem hiding this comment.
ChatCompletionTokenLogprob.logprob is not nullable in the OpenAI chat-completions schema. This converter can emit null here when the legacy token_logprobs entry is None (the new tests explicitly exercise that case). I reproduced the resulting payload failing validation with openai-python 1.99.9: choices.0.logprobs.content.0.logprob: Input should be a valid number. Please avoid emitting an entry with an unknown logprob (or map it to a valid numeric representation), make the chat type non-optional, and add a regression assertion that the response parses through the OpenAI client model.
There was a problem hiding this comment.
Good catch — fixed in b85afa6. ChatCompletionLogprob.logprob is now float (non-optional), and the converter skips tokens whose legacy token_logprobs[i] is None instead of emitting null (xinference/model/llm/utils.py:103-111). content[] now reports only tokens with a known logprob, so no entry fails ChatCompletionTokenLogprob validation. New test_chat_logprobs_parse_openai parses every content[] entry through openai.types.chat.chat_completion_token_logprob.ChatCompletionTokenLogprob (openai-python 1.99.9) — the exact gate that was failing — and the two propagation tests are updated to expect the dropped first token.
There was a problem hiding this comment.
The converter fix is correct, but this head is not green yet. pytest -q xinference/model/llm/tests/test_utils.py still fails in test_post_process_completion_preserves_chat_logprobs and test_post_process_completion_chunk_preserves_chat_logprobs: both expected payloads still include the old "Hello" entry with logprob: None, while the converter now correctly omits it. Please update those two expectations to match the known-logprob-only shape. The lint job also fails because isort modifies test_utils.py, so please run isort/pre-commit on the file as well.
…enai model ChatCompletionTokenLogprob.logprob is non-nullable in the OpenAI chat schema; openai-python 1.99.9 rejects a null with 'logprob: Input should be a valid number'. The converter previously copied the legacy token_logprobs[i] (None for tokens whose logprob was not computed, e.g. the first generated token in the legacy completion shape) straight into content[].logprob, so the chat response failed client-side validation. Drop tokens whose logprob is unknown instead of emitting a null/fabricated value: content[] reports only tokens whose logprob is actually known. Make ChatCompletionLogprob.logprob non-optional (float) to match the schema, and add a regression test that parses every content[] entry through openai-python's ChatCompletionTokenLogprob.
3 participants
Summary
/v1/chat/completionssilently returnedlogprobs: nulleven when the client requestedlogprobs/top_logprobs, while/v1/completionsalready populated them for vLLM (#5245). The chat builders discarded the logprobs the engine already produced.Root cause:
ChatModelMixin._to_chat_completion/_to_chat_completion_chunkbuilt chat choices without copying the source choice'slogprobs, and the chat choiceTypedDicts (ChatCompletionChoice,ChatCompletionChunkChoice) did not declare the field at all.Fix: declare
logprobson both chat choiceTypedDicts and propagatechoice.get("logprobs")in the two real choice-bearing chat builders. The chat path already receives logprobs-bearingCompletionChunks from the vLLM completions generator (vllm/core.py_convert_request_output_to_completion_chunk→_build_logprobs), which also slices per-chunk deltas for streaming (_slice_logprobs), so the chat builder needs no new generation or slicing logic — it only stops discarding what the engine already built.Follow-up to #5245 (completions logprobs).
Effective scope
vLLM is the only engine that builds logprobs on its chunks today, so this change is effectively vLLM-only: sglang / transformers / llama.cpp / lmdeploy continue to return
nulluntil they implement logprobs (tracked as a separate follow-up; the per-engine support matrix is intentionally not addressed here to keep the change minimal and additive). Non-logprobs requests are unaffected — the engine returnsNonewhen logprobs are not requested, so propagation is a no-op vs. current behavior.Shape note
Chat completions carry the legacy
CompletionLogprobsshape (text_offset/tokens/token_logprobs/top_logprobs) — the same shape the/v1/completionsendpoint already returns — rather than the modern OpenAI chat-logprobs schema (content/bytes). This keeps chat consistent with completions (#5245) and is the smallest correct step; aligning to the modern chat-logprobs schema is left as a follow-up so existing clients reading the completions-style shape keep working.Tests
Adds two unit tests in
xinference/model/llm/tests/test_utils.py:test_to_chat_completion_propagates_logprobs— non-streaming: builds aCompletionwith a realCompletionLogprobson its choice, callsChatModelMixin._to_chat_completion, asserts the chat choice carries the source logprobs (equal, not just not-None).test_to_chat_completion_chunks_propagates_logprobs— streaming counterpart viaChatModelMixin._to_chat_completion_chunks.Both use the public builder staticmethods/classmethods directly (no mocking of the function under test), are red on
main(the builders omit thelogprobskey →KeyError/None) and green after this change.Checklist
main, green on this branchNotRequiredfield + propagation only)TypedDicts; engine code untouched