feat: Added support for Foundry-Local#649
Conversation
There was a problem hiding this comment.
Mainly spelling mistakes and minor nits. The only major thing that needs fixing is that Foundry Local does support tool use (and tool results), otherwise they wouldn't allow you to add tools in the first place.
EDIT: Pasting the link here so it doesn't get lost in the review stuff: https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-models/how-to/use-chat-completions?pivots=programming-language-python#use-tools
| /// # Panics | ||
| /// - If the reqwest client cannot be built (if the TLS backend cannot be initialized). | ||
| pub fn new() -> Self { | ||
| Self::builder().build().expect("Ollama client should build") |
There was a problem hiding this comment.
Ollama -> Foundry Local
There was a problem hiding this comment.
yeahh i basically took a lot of code from ollama and forgot to rename lol
| where | ||
| Self: Sized, | ||
| { | ||
| let api_base = std::env::var("OLLAMA_API_BASE_URL").expect("OLLAMA_API_BASE_URL not set"); |
There was a problem hiding this comment.
Ollama -> Foundry Local
| } | ||
| } | ||
|
|
||
| // these i took from gemini ( review needed josh) |
There was a problem hiding this comment.
You can find the Azure AI Foundry model catalog here: https://ai.azure.com/catalog/models
I would suggest having a look as there are quite a few of these where the capital letter casing is not quite correct.
| impl TryFrom<CompletionResponse> for completion::CompletionResponse<CompletionResponse> { | ||
| type Error = CompletionError; | ||
| fn try_from(resp: CompletionResponse) -> Result<Self, Self::Error> { | ||
| let mut assitant_contents = Vec::new(); |
There was a problem hiding this comment.
assitant -> assistant
| // foundry only responds with an array of choices which have | ||
| // role and content (role is always "assistant" for responses) | ||
| for choice in resp.choices.clone() { | ||
| assitant_contents.push(completion::AssistantContent::text(&choice.message.content)); |
There was a problem hiding this comment.
assitant -> assistant
| .text() | ||
| .await | ||
| .map_err(|e| CompletionError::ProviderError(e.to_string()))?; | ||
| tracing::debug!(target: "rig", "Foundry chat response: {}", text); |
There was a problem hiding this comment.
nit: Foundry -> Foundry Local (I know this is a bit pedantic, but as Azure AI Foundry itself is already a known service, if we end up adding support for it then it will be a much more obvious differentiation to make)
|
|
||
| let data_line = if let Some(data) = line.strip_prefix("data: "){ | ||
| data | ||
| }else{ |
There was a problem hiding this comment.
very small nit: } else {
| } | ||
|
|
||
| #[derive(Debug, Serialize, Deserialize)] | ||
| pub enum Role { |
There was a problem hiding this comment.
You can use #[serde(rename_all = "lowercase")] here rather than manually renaming each one
| use crate::message::Message as InternalMessage; | ||
| match internal_msg { | ||
| InternalMessage::User { content, .. } => { | ||
| // Foundry doesn't support tool results in messages, so we skip them |
There was a problem hiding this comment.
unfortunately for you, it actually does: https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-models/how-to/use-chat-completions?pivots=programming-language-python#use-tools
There was a problem hiding this comment.
i was looking at their rest api docs ... shouldve digged deeper
| pub mod azure; | ||
| pub mod cohere; | ||
| pub mod deepseek; | ||
| pub mod foundry; |
There was a problem hiding this comment.
Please rename to foundry_local (removes ambiguity in case we decide to add Azure AI Foundry officially later on)
joshua-mo-143
left a comment
joshua-mo-143
left a comment
There was a problem hiding this comment.
Just needs an example (as well as pending changes) then should be good for another review
| streaming::{self, RawStreamingChoice}, | ||
| }; | ||
|
|
||
| const FOUNDRY_API_BASE_URL: &str = "http://localhost:42069"; |
There was a problem hiding this comment.
please use a different port number (yes, I know 42069 is funny)
There was a problem hiding this comment.
alright alright :(
joshua-mo-143
added
the
blocked
|
Marked |
2 participants
closes #630