Fix: Spring 7 Webflux NoSuchElementError on HttpHeaders#entrySet()

[mtomik]

What does this PR do?

Since the original PR is stuck for some time,
I created another one with the same commits + a bit better solution for the core of this issue.

Also when I was running these newer tests I found one more issue:

Spring7ServerFunctionalInstrumentationTest#dispatchError test was failing due to TracedSubscriber not removing the reference of the subscription on cancel() from contextMap. so the solution might be the CancellationAwareSubscription that is just the wrapper calling the discardIf() after cancel.

Is it possible that the issue was there even with older versions of reactor (from older Spring version), but now in that Spring 7 ( reactor 3.8.6 ) it was always failing.

Checklist

• This is an enhancement of existing features, or a new feature in existing plugins
  • I have updated CHANGELOG.next-release.md
  • I have added tests that prove my fix is effective or that my feature works
  • Added an API method or config option? Document in which version this will be introduced
  • I have made corresponding changes to the documentation
• This is a bugfix
  • I have updated CHANGELOG.next-release.md
  • I have added tests that would fail without this fix
• This is a new plugin
  • I have updated CHANGELOG.next-release.md
  • My code follows the style guidelines of this project
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have updated supported-technologies.md
  • Added an API method or config option? Document in which version this will be introduced
  • Added an instrumentation plugin? Describe how you made sure that old, non-supported versions are not instrumented by accident.
• This is something else
  • I have updated CHANGELOG.next-release.md

[cla-checker-service]

💚 CLA has been signed

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mtomik]

thanks for the review 👍

the failed build was due to incompatibility of that sub cancel with java 7. I reworked that using Instrumentation -SubscriptionCancelInstrumentation that will call the discardIf on cancelled subscription

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[SylvainJuge]

Is there a common name to all the implementations we need to instrument here ? the hasSuperType is very expensive without getTypeMatcherPreFilter, so if possible adding getTypeMatcherPreFilter would be relevant here.

[mtomik]

only I can think of is something like this, but that probably will not narrow it much, right?

    @Override
    public ElementMatcher<? super NamedElement> getTypeMatcherPreFilter() {
        return nameContains("Subscription").or(nameContains("Subscriber"));
    }

or maybe have multiple these instrumentations by package/library that implements the CoreSubscriber ? or have just one for the implementations from reactor.core and the rest will be cleaned potentially by the GC later?

[SylvainJuge]

I think this should be fine as a first step, do you have a list of all the classes that are being currently instrumented here to verify all of them match ? If those are spring classes I would expect them to stay consistently named. If those are user-provided then it might be too narrow and we need another approach.

[mtomik]

the problem is, that there can come any implementation of CoreSubscriber. if we miss some thanks to this filter, the TracedSubscriber will not remove it immediately from its contextMap / subscriptionMap.

if I understand it correctly, the memory leak caused by this was always handled by GC. in this newer spring 7 came newer reactor or other library that allowed us to spot it more easily? that was at least my thought...

I had here another solution to solve it - the wrapper around that subscription (ecde26e) , but I had there compile issue with the older java versions. so maybe just try to solve that to avoid using this instrumentation?

[SylvainJuge]

I am definitely not familiar with the implementation details of webflux here, so I'll rely on your expertise to know if it's fine or not to add such filter.

What I know though is that the matching can become time-expensive, when the agent is started with debug logging there is an ascii-art table displayed at the JVM shutdown that prints the matching time per instrumentation. With that, you should be able to evaluate if the matching is actually expensive or not relative to other instrumentations. With a bit of luck the classes that are being instrumented here do not have very deep class hierarchies (unlike servlets for example), so the matching might not be as-expensive as it could be with such matcher.

In order to evaluate the impact in practice I would thus suggest to:

• run the webflux test with this instrumentation disabled and log_level=debug, store the output as baseline
• run the webflux test with this instrumentation enabled and log_level=debug, compare the output and report if there is any significant difference.

[mtomik]

run the benchmarks as you mentioned -

without instrumentation

2026-09-03 00:17:05,567 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 239,064,579ns
| Advice name                                        | Type ns         | Method ns       |
| ExecutorInstrumentation$ExecutorRunnableInstrumentation |      49,737,495 |       5,401,685 |
| JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation |      26,006,870 |               0 |
| JakartaFilterInstrumentation                       |      25,307,667 |               0 |

instrumentation without name filter

2026-09-02 23:56:42,555 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 716,530,294ns
| Advice name                                        | Type ns         | Method ns       |
| SubscriptionCancelInstrumentation                  |     501,739,464 |      18,140,762 |
| ExecutorInstrumentation$ExecutorRunnableInstrumentation |      33,892,075 |       5,670,217 |
| JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation |      16,321,812 |               0 |
| JakartaFilterInstrumentation                       |      15,601,878 |               0 |

instrumentation with name filter

2026-09-02 23:58:28,541 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 280,546,991ns
| Advice name                                        | Type ns         | Method ns       |
| ExecutorInstrumentation$ExecutorRunnableInstrumentation |      54,391,782 |       7,226,246 |
| SubscriptionCancelInstrumentation                  |      27,428,938 |      11,004,509 |
| JakartaFilterInstrumentation                       |      30,907,346 |               0 |

instrumentation with name filter + small type matcher optimization

2026-09-03 00:00:57,965 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 266,680,947ns
| Advice name                                        | Type ns         | Method ns       |
| ExecutorInstrumentation$ExecutorRunnableInstrumentation |      52,177,027 |       5,465,724 |
| SubscriptionCancelInstrumentation                  |      24,501,340 |       8,916,960 |

so the name matcher made huge difference.

I also solved the java7 compile issue with the "non-instrumentation" solution, but I couldn't find cleaner solution than that - you can find it in last commits.

so if that 10% increase in time spent is not much I would pick that

[SylvainJuge]

we probably don't need to have DEBUG by default here, this will make the test execution very verbose.

[mtomik]

it seems like it is just copy from the existing "testapp" module. so should I remove it also from there?

[SylvainJuge]

yes, please do that.

[mtomik]

done

[jackshirazi]

run docs-build

[jackshirazi]

@mtomik are you able to complete the remaining suggested feedback, I was waiting for this to do a release but if it's not going to be completed soon I'll push this to the next release