Support Object Lock feature #479

Activity

1. moroine

   added

   waiting-for-triage

   on Jun 16, 2026
2. github-project-automation

   added this to Fluentd Kanbanon Jun 16, 2026
3. Watson1978

   added

   enhancement

   and removed

   waiting-for-triage

   on Jun 16, 2026
4. Watson1978

   removed this from Fluentd Kanbanon Jun 16, 2026
5. Watson1978 commented on Jun 16, 2026

   

   Watson1978

   on Jun 16, 2026

   Contributor

   More actions

   @moroine
   Thanks for the proposal! I'd like to understand the use case better, since this may already be doable on the S3 side.

   Enabling Object Lock with a default retention on the bucket locks every uploaded object at PUT time without any plugin change, and already-locked objects aren't affected by later changes to the default. The concern about the default being modified can also be handled by denying s3:PutBucketObjectLockConfiguration via IAM/SCP, so only a limited role can change it.

   Could you share where these fall short for you? Specifically:

   1. Is the fluentd config managed more strictly than the bucket itself (e.g. IaC/CI), so setting the lock app-side is genuinely harder to tamper with than the bucket config?
   2. Do you need different retention per object/data type within one bucket?

   That would help us decide whether this adds real defense-in-depth beyond bucket-level config + IAM. Thanks!
6. moroine commented on Jun 16, 2026

   

   moroine

   on Jun 16, 2026

   Author

   More actions

   Hi,

   Yeah, that's mainly for a defense in depth. IAM configuration can be hard to manage, and scenario where an admin get his credentials stolen then this could be use to modify IAM policies and add let's say a lifecycle removal policy.

   As we use this plugin to persist long-time the logs, then the server will directly use the COMPLIANCE mode so even an admin won't be able to delete neither the logs nor the bucket.

   But I agree that's an extreme use case ^^ I was just wondering if this could make sense.
7. Watson1978 commented on Jun 17, 2026

   

   Watson1978

   on Jun 17, 2026

   Contributor

   More actions

   Thanks, that makes the intent much clearer.

   Two things to align on first:

   1. The lifecycle attack doesn't actually work against locked logs. Object Lock retention overrides Lifecycle expiration — a lifecycle rule can't delete an object version that's still within its retention period (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-managing.html). So a stolen-admin credential can't wipe logs that are already locked, no matter how the lock was applied.

   2. For already-written logs, a bucket default retention in COMPLIANCE mode is equivalent to locking per-object from the plugin: in both cases no one (incl. root) can delete the objects, and the bucket can't be deleted while locked objects remain.

   The one genuine delta for doing it plugin-side is independence from the bucket's default config for future writes — if the default gets changed/removed, app-side locking keeps protecting new objects. That's real defense-in-depth, though it's also closable by denying s3:PutBucketObjectLockConfiguration (ideally via an Organizations SCP a single compromised account admin can't override).

   So my recommendation for most users would be bucket default retention + an SCP/IAM deny — it achieves the goal without extra config surface, and I'd want to document that as the primary approach.

   That said, I do see the value of the plugin asserting the lock itself when the fluentd config is managed more strictly than the bucket (IaC/CI), so I'm not opposed to adding it as opt-in (defaulting to off).

   If you'd like to pursue that, a PR is welcome — the main things I'd want are: clear docs that the bucket needs Object Lock + versioning enabled, and an explicit warning that COMPLIANCE is irreversible (objects can't be deleted until expiry even by mistake or for cost), which is especially important for long-retention logs.

   Does that work for you?
8. moroine commented on Jun 17, 2026

   

   moroine

   on Jun 17, 2026

   Author

   More actions

   Yup, that works for me !

   I agree the scenario is very hard to hit. And yeah this should never become default on !!

   I've never written ruby, but looks like the change is very small. Do you have any existing doc on how to test / run locally the plugin ?
9. Watson1978 commented on Jun 18, 2026

   

   Watson1978

   on Jun 18, 2026

   Contributor

   More actions

   Awesome, glad to have you give it a shot! Don't worry about the Ruby side — the change is small and the existing tests will guide you. Here's the local setup.

   Prereqs

   A recent Ruby and Bundler (gem install bundler).

   Setup & run tests

   git clone https://github.com/<your-fork>/fluent-plugin-s3.git
   cd fluent-plugin-s3
   bundle install
   bundle exec rake test        # runs the whole suite
   

   Run just the output-plugin tests:

   bundle exec ruby -Ilib -Itest test/test_out_s3.rb
   # single test method:
   bundle exec ruby -Ilib -Itest test/test_out_s3.rb -n test_your_new_case
   

   Write tests

   You don't need a real S3 bucket or AWS credentials.
   The tests stub the AWS SDK client (via the rr mocking lib), so they assert on the parameters the plugin would send to S3.
   You'd add a test that sets your new object_lock_* options and checks that the put_object call receives the matching object_lock_mode / object_lock_retain_until_date params.

   Where things live

   • Output plugin code: lib/fluent/plugin/out_s3.rb (look for the config_param declarations near the top, and the put_object call in the write path)
   • Tests: test/test_out_s3.rb