feat!(objectstore): add Azure and GCS backends with provider auto-detection - #291
Draft
jplbrun wants to merge 9 commits into
Draft
feat!(objectstore): add Azure and GCS backends with provider auto-detection#291jplbrun wants to merge 9 commits into
jplbrun wants to merge 9 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds Azure Blob Storage and Google Cloud Storage backends to the
objectstoremodule and makescreate_client()auto-detect the provider from the service binding at runtime. Previously the module only supported S3/MinIO.The three backends are unified behind a single
ObjectStoreClientprotocol — a fixed contract of 8 methods (put_object_from_bytes,put_object,put_object_from_file,get_object,delete_object,list_objects,head_object,object_exists). Every backend ships all 8 from day one, so all providers are at parity and no caller has to branch on the provider.create_client()returns something satisfying this protocol; the concrete class you get back (S3Client,AzureClient,GcsClient) is an internal detail.How a client is built:
create_client("object-store-1")reads the binding for that instance from the secret mount / env vars, inspects which keys are present, infers the provider, then loads and validates the full config.S3Config,AzureConfig,GcsConfigand the factory routes on its type — no binding read, no detection.Key pieces added:
_protocol.py— theObjectStoreClientprotocol plus anObjectReaderprotocol (the managed binary streamget_objectreturns)._azure.py/_gcs.py— the two new backends._detect.py— reads whatever keys the binding presents and maps them to a provider (case-insensitive; providers are verified disjoint so a binding can't match two)._factory.py— housescreate_client(moved out of__init__.py; the public import path is unchanged because__init__.pyre-exports it).config.py— the three public*Configdataclasses plus internal*BindingData(what the secret resolver fills) with avalidate()/to_config()step._validation.py— argument validation extracted out of_s3.pyso all backends share it.Related Issue
N/A
Type of Change
How to Test
Unit tests
uv run pytest tests/objectstore/unit -v— full unit suite for the module, including the new backends, detection, and config resolution:test_detect.py— provider inference from binding keys + disjointness.test_config.py— binding → config resolution and validation errors.test_azure_client.py/test_gcs_client.py— the two new backends against mocked provider SDKs.test_create_client.py— auto-detection, explicit-config routing, and error paths.test_s3_client.py,test_models.py,test_protocol.py— updated for the rename/protocol.uv run pytest tests/ -m "not integration"— full unit suite. Expected: all pass.Integration tests
uv run pytest tests/objectstore/integration -m integration -v.Checklist
Breaking Changes
This PR reshapes the module's public surface. The impact on real consumers was measured against the agent-repo corpus; the affected population is small but non-zero, so each break is called out below with a migration.
1.
ObjectStoreClientis now a Protocol, not an instantiable classBefore,
ObjectStoreClientwas the concrete S3 client and callers could construct it directly. It is now atyping.Protocoldescribing the shared interface, and the concrete S3 implementation moved to the privateS3Client.ObjectStoreClient(...)directly (bypassingcreate_client).ObjectStoreClientpurely as a type annotation —create_client()still returns something that satisfies it, sox: ObjectStoreClient = create_client(...)type-checks unchanged.create_client(instance)(auto-detection) orcreate_client(instance, config=S3Config(...))(explicit). Do not instantiate the client class yourself.2.
ObjectStoreBindingDatais renamed and no longer publicThe single S3-shaped
ObjectStoreBindingDatadataclass is gone from the public API. It is replaced by three provider-specific public config types —S3Config,AzureConfig,GcsConfig— which are whatcreate_client(config=...)accepts and what__all__exports. (Provider-specific*BindingDatatypes still exist internally as secret-resolver targets, but they are not part of the public surface.)ObjectStoreBindingData, whether from the public re-export or reaching intosap_cloud_sdk.objectstore._models.*Config. The S3 field names are unchanged (access_key_id,secret_access_key,bucket,host), withdisable_sslnow living onS3Configinstead of oncreate_client(see chore(deps): update setuptools requirement from ~=80.9.0 to >=80.9,<82.1 #4).3.
create_client(config=...)now takes a typed union, notObjectStoreBindingDataThe
configkeyword's type changed fromOptional[ObjectStoreBindingData]toUnion[S3Config, AzureConfig, GcsConfig, None]. The factory routes on the concrete config type to pick the backend. This is a direct consequence of #2 — a caller passingObjectStoreBindingDatano longer compiles/runs.4.
disable_sslmoved offcreate_clientontoS3Configdisable_sslwas a keyword oncreate_client, but it only ever mapped to MinIO's plain-HTTP mode — it does nothing for Azure (URI fixes the scheme tohttps) or GCS (always HTTPS). It's now a field onS3Config.create_client(instance, disable_ssl=True)— the kwarg no longer exists, so this raisesTypeErrorat runtime.disable_ssl=Truerequires the explicit-config path — there is no way to enable it while relying on auto-detection, because auto-detection builds the config from the binding and the binding resolver only handlesstrfields. You must supply full credentials viaS3Configwhen you need plaintext mode (this is a local-dev / MinIO-only scenario in practice).5.
get_object()return type changed:http.client.HTTPResponse→ObjectReaderget_objectpreviously returned MinIO's rawurllib3/HTTPResponseobject; it now returns anObjectReader— a small protocol exposingread(),close(), and context-manager support. This was necessary to give all three providers a common, provider-agnostic return type.HTTPResponse, or that relied onHTTPResponse-specific attributes/methods beyondread()/close()(e.g..status,.getheaders(),.release_conn()).with:Additional Notes
N/A