Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions crates/libsy-llm-client/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,9 +144,9 @@ async fn stream(
### Routing an algorithm

[`run`] takes a libsy algorithm and a [`ClientRouter`], and returns the final response plus
the trace of decisions the algorithm published. The router resolves each offloaded call to
the client for the target the algorithm selected; `ClientRouter::single` is the
single-provider case:
the trace of decisions the algorithm published. Each offloaded `CallModel` carries an ordered
`models` list. The router resolves and tries those candidates in order; `ClientRouter::single`
is the single-provider case:

```rust
use std::sync::Arc;
Expand Down Expand Up @@ -249,9 +249,12 @@ fn build_multi_format_client(
transport failures are retried; streaming body failures are not replayed after
the response has been returned.

Retries replay the same upstream request. A transport failure can therefore
duplicate a request that the provider processed but did not finish returning,
and the retry budget plus capped `Retry-After` delays determines total latency.
Retries replay the same upstream request to the same model. Each candidate's
`max_retries` budget is exhausted before candidate fallback advances to the next
model. The worst case is `candidates × (max_retries + 1)` upstream requests, and
total latency includes every candidate's capped `Retry-After` backoff. A transport
failure can duplicate a request that the provider processed but did not finish
returning.

## Errors

Expand Down
1 change: 0 additions & 1 deletion crates/libsy-llm-client/src/observability.rs
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,6 @@ fn client_call_error_type(error: &LibsyError) -> Cow<'static, str> {
LibsyError::AlgorithmError { .. } => Cow::Borrowed("algorithm_error"),
LibsyError::Driver(_) => Cow::Borrowed("driver_error"),
LibsyError::MissingFinalResponse => Cow::Borrowed("missing_final_response"),
LibsyError::AllTargetsExcluded => Cow::Borrowed("context_window_exceeded"),
LibsyError::External { .. } => Cow::Borrowed("_OTHER"),
}
}
Expand Down
Loading
Loading