Migrate service model base url#3560
Conversation
…/v1`) if `model` has `openai` format. - Add CORS support to the gateway service endppoint (if `model` has `openai` format)
…the same model name
… backward compatibility (with older verions of gateway)
Now that service.model.base_url points to service.url + /v1 for openai-format models, it is no longer deprecated and can be recommended as the model endpoint. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
@jvstme Let me know once you have a look. In principle, we can also merge as is. |
| strip_prefix: bool = True # only used in-server | ||
| replicas: tuple[Replica, ...] | ||
| router: Optional[AnyRouterConfig] = None | ||
| cors_enabled: bool = False # only used on gateways; enabled for openai-format models |
There was a problem hiding this comment.
(nit) Whether CORS is enabled is a decision the gateway takes based on existing data points, not a new data point, so I'd prefer not to persist it in the repo.
But if there isn't a different and easy way to implement that decision, I don't mind keeping it as is
There was a problem hiding this comment.
This CORS policy is more permissive than what we have on the gateway.* endpoint, because this policy allows cross-origin requests to the entire service, not just the model. Consider setting a stricter policy by returning the CORS headers only from /v1 (or a different configured OpenAI API prefix). Should be relatively easy to implement with an if ($request_uri ~ ^/<prefix>) block
There was a problem hiding this comment.
Yes, if you don't strongly insist, I'd prefer to implement it separately from the main PR.
| proxy_hide_header 'Access-Control-Allow-Origin'; | ||
| proxy_hide_header 'Access-Control-Allow-Methods'; | ||
| proxy_hide_header 'Access-Control-Allow-Headers'; | ||
| proxy_hide_header 'Access-Control-Allow-Credentials'; |
There was a problem hiding this comment.
Note that this prevents users from configuring their own CORS policy for the service. For example, users may want to set a stricter Access-Control-Allow-Origin, which is a requirement if they also set Access-Control-Allow-Credentials (which is forbidden when Access-Control-Allow-Origin is *).
But I'm not sure if there's an easy way to set headers conditionally, based on whether a header was returned by the upstream.
Maybe this limitation won't have any realistic impact if we only set our CORS policy on /v1 (or another configured OpenAI API prefix)
3 participants
Summary
Migrate
service.model.base_urlfrom the deprecatedgateway.{domain}model aggregation endpoint to the service endpoint foropenai-format models.service.model.base_url: Foropenaiformat, set toservice_url + prefix(e.g.https://run-name.domain.com/v1). For other formats, keep the existinggateway.{domain}URL.prefixfield instead of hardcoding/v1.service.jinja2nginx template, enabled only for services withopenai-format models. Strips upstream CORS headers to avoid duplicates.cors_enabledonServicemodel: Store CORS flag per-service for correctness with multiple services. Includes a startup migration for gateways with pre-existing state.