Skip to content

Document S3-compatible remote logging in the Amazon provider#69190

Draft
goanpeca wants to merge 1 commit into
apache:mainfrom
goanpeca:b2-remote-logging-docs
Draft

Document S3-compatible remote logging in the Amazon provider#69190
goanpeca wants to merge 1 commit into
apache:mainfrom
goanpeca:b2-remote-logging-docs

Conversation

@goanpeca

@goanpeca goanpeca commented Jul 1, 2026

Copy link
Copy Markdown

Add a docs recipe and a system-test example Dag showing how to reuse the existing Amazon provider S3 integration against any S3-compatible object store.

The Amazon provider already talks to S3-compatible endpoints: pointing an aws connection at a custom endpoint_url makes the S3 task handler and ObjectStoragePath("s3://...") write to that endpoint instead of AWS S3. That capability is not documented from the "bring your own S3-compatible bucket" angle, so users rediscover the connection extras (endpoint_url, region_name, path-style addressing) by trial and error. This change writes that path down. Amazon S3 is the baseline, and the recipe notes that the same steps work against other S3-compatible services (for example Amazon S3, Backblaze B2, Cloudflare R2, and MinIO), using placeholder endpoints so the steps are copy-pasteable against any of them.

The change is intentionally small and additive: it adds documentation and one example, and touches no provider or core code.

What is included:

  • providers/amazon/docs/logging/s3-compatible-remote-logging.rst: a recipe that maps S3-compatible credentials onto the AWS connection fields, configures [logging] remote logging through that connection, verifies the logs land in the bucket, and reuses the same connection for ObjectStoragePath Dag data. It cross-references the existing write-logs-amazon-s3 handler doc rather than restating it, and it is picked up automatically by the globbed toctree in providers/amazon/docs/logging/index.rst (no manual index edit needed).
  • providers/amazon/tests/system/amazon/aws/example_s3_compatible_object_storage.py: an input to transform to output example Dag built on ObjectStoragePath, sitting alongside the other example_s3*.py system tests and wired through the standard get_test_run(dag) harness. Bucket and connection id are read from the environment so the example carries no secrets.

Scope notes for reviewers:

  • No new operator, hook, bundle, or provider.yaml entry: this documents existing behavior, so there is nothing new for the registry to pick up.
  • The framing is generic S3-compatible object storage. AWS S3 is the baseline; other providers are named only as interchangeable examples with placeholder endpoints.
  • No newsfragment: per the contributor docs, providers/ changelogs are regenerated from git log by the release managers and do not consume per-PR newsfragments.

Testing:

  • Built the provider docs locally and confirmed the new page renders and appears under the Amazon provider "Logging for Tasks" section, with the cross-reference to the S3 task handler resolving.
  • The example Dag is a system test (network access to a real bucket required to execute against a live endpoint) and follows the existing example_s3*.py structure; the module imports and parses cleanly and exposes test_run via get_test_run(dag).
  • Ran the repo static checks (ruff / ruff-format) on the new Python file.

Was generative AI tooling used to co-author this PR?
  • Yes (Claude Code, Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg

boring-cyborg Bot commented Jul 1, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

Signed-off-by: Gonzalo Peña-Castellanos <goanpeca@gmail.com>
@goanpeca goanpeca force-pushed the b2-remote-logging-docs branch from b33f995 to eec9d36 Compare July 1, 2026 14:05
@goanpeca goanpeca changed the title Add S3-compatible remote logging guide to the Amazon provider Document S3-compatible remote logging in the Amazon provider Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant