Document S3-compatible remote logging in the Amazon provider#69190
Draft
goanpeca wants to merge 1 commit into
Draft
Document S3-compatible remote logging in the Amazon provider#69190goanpeca wants to merge 1 commit into
goanpeca wants to merge 1 commit into
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
Signed-off-by: Gonzalo Peña-Castellanos <goanpeca@gmail.com>
b33f995 to
eec9d36
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a docs recipe and a system-test example Dag showing how to reuse the existing Amazon provider S3 integration against any S3-compatible object store.
The Amazon provider already talks to S3-compatible endpoints: pointing an
awsconnection at a customendpoint_urlmakes the S3 task handler andObjectStoragePath("s3://...")write to that endpoint instead of AWS S3. That capability is not documented from the "bring your own S3-compatible bucket" angle, so users rediscover the connection extras (endpoint_url,region_name, path-style addressing) by trial and error. This change writes that path down. Amazon S3 is the baseline, and the recipe notes that the same steps work against other S3-compatible services (for example Amazon S3, Backblaze B2, Cloudflare R2, and MinIO), using placeholder endpoints so the steps are copy-pasteable against any of them.The change is intentionally small and additive: it adds documentation and one example, and touches no provider or core code.
What is included:
providers/amazon/docs/logging/s3-compatible-remote-logging.rst: a recipe that maps S3-compatible credentials onto the AWS connection fields, configures[logging]remote logging through that connection, verifies the logs land in the bucket, and reuses the same connection forObjectStoragePathDag data. It cross-references the existingwrite-logs-amazon-s3handler doc rather than restating it, and it is picked up automatically by the globbed toctree inproviders/amazon/docs/logging/index.rst(no manual index edit needed).providers/amazon/tests/system/amazon/aws/example_s3_compatible_object_storage.py: an input to transform to output example Dag built onObjectStoragePath, sitting alongside the otherexample_s3*.pysystem tests and wired through the standardget_test_run(dag)harness. Bucket and connection id are read from the environment so the example carries no secrets.Scope notes for reviewers:
provider.yamlentry: this documents existing behavior, so there is nothing new for the registry to pick up.providers/changelogs are regenerated fromgit logby the release managers and do not consume per-PR newsfragments.Testing:
example_s3*.pystructure; the module imports and parses cleanly and exposestest_runviaget_test_run(dag).Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.