Skip to content

Mutual TLS on multi-worker /aggregated_metrics still uses VERIFY_NONE #269

Open

Description

@addresswork038-sys

Describe the bug

When in_prometheus uses TLS and workers > 1, /aggregated_metrics fetches each worker on loopback with OpenSSL::SSL::VERIFY_NONE. External /metrics can require a client certificate ( and client_cert_auth). The worker-aggregation fetch does not verify the worker certificate, so that path is not mutual TLS.

The setting is in do_request in lib/fluent/plugin/in_prometheus.rb:

if secure
http.use_ssl = true

target is our child process. so it's secure.

http.verify_mode = OpenSSL::SSL::VERIFY_NONE
end
The async-http path in lib/fluent/plugin/in_prometheus/async_wrapper.rb has the same gap.

To Reproduce

Start Fluentd with workers 2 and the configuration below.
Request https://127.0.0.1:24231/aggregated_metrics with a valid client certificate.
The parent worker scrapes the other worker with verify_mode = OpenSSL::SSL::VERIFY_NONE.
A worker certificate that would fail VERIFY_PEER is still accepted on that internal fetch.

Expected behavior

The loopback fetch should use VERIFY_PEER and the same CA as the TLS transport, on both the Net::HTTP path and the async-http path. A bad worker certificate should be rejected.

Your Environment

Rocky8

- Fluentd version: 1.16.9
- TD Agent/Fluent Package version: fluent-package 5.0.7 (fluent-package-5.0.7-1.el8.x86_64)
- fluent-plugin-prometheus version: 2.1.0
- prometheus-client version: 2.1.0
- Operating system: Rocky Linux 8.10 (Green Obsidian)
- Kernel version: 5.15.167.4-microsoft-standard-WSL2

Your Configuration

workers 2 @type prometheus bind 0.0.0.0 port 24231 metrics_path /metrics cert_path /path/to/server.crt private_key_path /path/to/server.key ca_path /path/to/ca.pem client_cert_auth true

Your Error Log

No error is logged. The loopback scrape succeeds because verification is disabled. That is the bug: a certificate that VERIFY_PEER would reject is accepted

Additional context

No response

Activity

  1. Watson1978 commented on Oct 1, 2026

    @Watson1978
    Contributor

    A proper fix needs a bit more than switching to VERIFY_PEER:

    • When client_cert_auth is enabled, the loopback client should present cert_path / private_key_path as its client certificate (this would also fix the aggregation failure described above).
    • Server certificates often do not include 127.0.0.1 / ::1 in their SAN, so hostname verification on the loopback address needs to be handled while still verifying the chain against ca_path.
    • The behavior with system CA (no ca_path), generated self-signed certificates, and the deprecated <ssl> section needs to stay consistent.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions