Repository navigation
fluent-plugin-s3 it fail to auto clean up the buffer's files after pushing to S3 #339
Description
I also encountered the problem that the files in the buffer could not be automatically deleted after the push。
<match>
<buffer>
@type file
path /etc/fluentd/temp/moons
timekey_wait 5
timekey 1
chunk_limit_size 256m
</buffer>
time_slice_format %Y%m%d%H
</match>
Activity
ZP-AlwaysWin commentedon Jul 9, 2020
/assign @@repeatedly
jychen7 commentedon Jul 17, 2020
I find it too, so I change to use
memorybufferAnd my test of
file to s3shows memory buffer (200MB per minute) have 4x throughput than disk buffer (50MB), with gzipstevehipwell commentedon Apr 6, 2021
This continues to be an issue is there any more information on this?
github-actions commentedon Jul 6, 2021
This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days
prabhpreet332 commentedon Aug 18, 2022
Anyone resolved this issue?
It still doesn't work for me.
I get buffer size of ~900MB in s3 output plugin. Flush interval is 10min.
Any suggestions are welcome.
jychen7 commentedon Aug 19, 2022
I switch to memory buffer and works well
#339 (comment)
jatinder10 commentedon Oct 31, 2022
Disk space of EKS pods are filling fast when we are using fluentd to s3