bug: Memory leak when logger log the traffic with large (about 128K) response body

[caogw279]

Current Behavior

When configured kafka-logger in global rules with include_resp_body set to true and upstream server response 128K body, the memory will increase during the test and never get down, if the test continue, the memory usage will continue increase until the OOM happen.

Expected Behavior

The memory usage will keep in a range during the test and will return to normal after the stress stop.

Error Logs

No error logs, please have look at output of "free" and the RES of very worker process

Steps to Reproduce

1. Run apisix in CentOS
2. APISIX Config route with Admin API
3. APISIX Config kafka-logger to globale_rules with include_resp_body set to true
4. Stress the APISIX (with Avalanche, server response 128K )
5. Monitor the system memory usage (such as via order free -h)
6. Check the memory usage of every worker process (such as via top)

Environment

• APISIX version (run apisix version): apisix 3.8.0
• Operating system (run uname -a): Linux localhost.localdomain 3.10.0-1160.105.1.el7.x86_64 change: added doc of how to load plugin. #1 SMP Thu Dec 7 15:39:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.21.4.2
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5.0
• APISIX Dashboard version, if relevant: No APISIX Dashboard used.
• Plugin runner version, for issues related to plugin runners: no external plugin is used
• LuaRocks version, for installation issues (run luarocks --version): No luarocks is installed.

[caogw279]

Addition: I test it with wrk as client and take the nginx as the upstream server, and change it's index.html with 512k bytes characters, can't reproduce the issue, resident memory of every openresty process will keep at 1 - 1.5 G no matter how long wrk stress it.

[caogw279]

From the document [gc public API] (https://apisix.apache.org/docs/apisix/control-api/) I found a method which will trigger a full garbage collection in the HTTP subsystem, it's works, the memory not leak ? just hold by GC ? but why doesn't GC collection work by itself, why the memory is exhausted and GC still not work ?

[shreemaan-abhishek]

@caogw279, please share the details on how you stress test APISIX.

[Baoyuantop]

After local testing with

wrk -t10 -c100 -d300s http://127.0.0.1:9080/

The memory grows by about 2G and doesn't drop at the end of the test, it doesn't drop when /v1/gc is called, and it recovers after restarting the container.

[shreemaan-abhishek]

@Baoyuantop, can you try load testing with this patch: https://github.com/apache/apisix/pull/12035/files

last time I can confirm I was able to fix the memory leak bug by this patch. But could not document and prove it as I did not have proper wrk tool setup in my development machine.