Repository navigation
feat: fail fast locally on endpoints answered with 429 - #45
Draft
gracehopper-vt wants to merge 6 commits into
Draft
gracehopper-vt wants to merge 6 commits into
gracehopper-vt wants to merge 6 commits into
Conversation
Requests answered with 429 are now retried up to 2 times by default (WithMaxRetries), waiting what the Retry-After header asks for when it's at most 60 seconds (WithMaxRetryWait), or using exponential backoff with jitter if the header is missing. When the server asks to wait longer (e.g. an hourly or daily quota is exhausted), the error is returned and the client remembers it: requests to the same endpoint family fail locally with the same QuotaExceededError until Retry-After has elapsed, without reaching the API. This can be disabled with WithQuotaFailFast(false). Requests whose body can't be re-sent (streamed uploads) are not retried, but they honor the local block. Co-authored-by: Oscar Torreno <torreno@google.com>
Drop expired blocks whenever a new one is added and cap the map at 256 endpoint families (evicting the block that expires first), so it can't grow unbounded even if a caller hits many different paths. Co-authored-by: Oscar Torreno <torreno@google.com>
Most API endpoints share the general API requests quota, which is exhausted for all of them at once, so a 429 there now blocks every endpoint using that quota. Intelligence downloads (file/behaviour downloads, zip files), Intelligence searches and private scanning have their own quotas and are blocked independently. With a fixed set of four families the bounding logic is no longer needed and has been removed. Co-authored-by: Oscar Torreno <torreno@google.com>
Drop automatic retries: a 429 is returned to the caller right away. When it comes with a Retry-After header, further requests to the same endpoint (method + path with object IDs replaced by "*") fail locally until that time has passed, so a client looping over the API can't keep hitting it. This replaces the hard-coded quota family table, and the WithMaxRetries, WithMaxRetryWait and WithQuotaFailFast options are removed. Co-authored-by: Oscar Torreno <torreno@google.com>
gracehopper-vt
changed the title
Rate limits at the edge (Cloud Armor) answer 429 with a generic HTML body and no Retry-After header, so a client looping over the API wouldn't be stopped. Block the endpoint for 60 seconds in that case. QuotaExceededError responses without Retry-After are not blocked, since the API only omits the header for quotas that don't reset over time (e.g. storage). Co-authored-by: Oscar Torreno <torreno@google.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #44 (base branch
feat/expose-retry-after), which exposesError.RetryAfter.Today vt-go returns an error on every
429, but nothing stops a loop in user code from sending the same call again and again while the quota is exhausted. With this PR the client stops sending requests to an endpoint that has already been rejected.429is returned to the caller right away, as it is today.429comes with aRetry-Afterheader, the client remembers it. Until then, requests to the same endpoint fail locally with aQuotaExceededErrorErrorand the remainingRetryAfter, without reaching the API.*. Any path segment that isn't only[a-z_]is treated as an ID. For example,GET /files/*andGET /files/*/downloadare separate endpoints. If a segment is misclassified, the block only gets more specific, never wider.Retry-After, the endpoint is blocked for 60 seconds (e.g. rate limits at the edge/Cloud Armor, which answer 429 with a generic HTML body and no header), except forQuotaExceededError: the API only omits the header for quotas that don't reset over time (storage, object counts), where blocking makes no sense.No new client options. This is a behaviour change, so it should go in a minor release with a changelog entry.
Tests
New tests in
vt_test.go: no retry on 429, per-endpoint block withRetry-After(other endpoints unaffected), 60s block on a 429 withoutRetry-After(edge HTML body), no block forQuotaExceededErrorwithout it, block expiry,endpointKeycases.go test ./...andgo vet ./...pass;gofmtis clean on the changed files.