Flaky test report: committed-code failures on 2026-06-11

[andrross]

Flaky test report: committed-code failures on 2026-06-11

Automated scan of the OpenSearch metrics cluster for gradle-check test failures against committed code (invoke_type:Timer + git_reference:main, plus Post Merge Action "push trigger main") over the 24 hours preceding 2026-06-11T10:00Z.

This report identifies and characterizes flaky tests; it does not attempt to root-cause any failure. No source changes were made.

Method

• Recent failures: queried gradle-check-* for committed-code failures in the last 24h.
• Historical flake rate: for each distinct test, aggregated all build types (including PR builds) by calendar month, counting unique builds (cardinality on build_number).
• Reproduction: each distinct test was run once locally with the exact suite seed extracted from the Jenkins stack trace (__randomizedtesting.SeedInfo.seed([...])).

Reproduction environment caveats

• Local checkout was on branch fix-arrow-flight-rpc-hardcoded-ports at commit 40e0098 (2026-05-28), roughly two weeks behind the main commits that failed. Results are not a perfect mirror of CI HEAD.
• JDK 25; single-machine dev desktop (more cores / faster than the CI runner — see the note about timing-amplified flakes).
• Mixed-cluster tests were reproduced against the v3.4.0 ↔ 3.7.0 (current) pairing that was cached locally; the specific failing CI build may have used a different BWC pairing.
• A single seeded run is not a deterministic-reproduction proof for concurrency/timing-sensitive integration tests; "did not reproduce" below means the seeded run passed locally, not that the test is healthy.

---

Summary table (sorted by unique builds affected, all-time, all build types)

#	Test	Unique builds affected	First seen	Recent failing build	Reproduced locally w/ seed?	Trend
1	MixedClusterClientYamlTestSuiteIT.test {p0=search/310_match_bool_prefix/multi_match multiple fields partial term}	392	2024-03-25	80096	No	Chronic, spiky; recent uptick
2	MixedClusterClientYamlTestSuiteIT.test {p0=search/310_match_bool_prefix/multi_match multiple fields complete term}	379	2024-03-25	80096	No	Chronic, spiky; recent uptick
3	FullRollingRestartIT.testFullRollingRestart_withNoRecoveryPayloadAndSource {p0={replication.strategy=SEGMENT}}	141	2024-10-11	80067	No	Chronic; resurgent in 2026
4	ClusterShardLimitIT.testOpenIndexOverLimit {p0={writable_warm_index.enabled=true}}	71	2025-10-15	80111	No	Worsening (May–Jun 2026 spike)
5	WarmIndexSegmentReplicationIT.testReplicaAlreadyAtCheckpoint	7	2025-04-29	80096	No	Rare; small Apr 2026 cluster
6	WarmToHotTieringServiceIT.classMethod	3	2026-05-19	80162	No	New; low frequency

None of the six reproduced locally with the original seed.

---

Details

1 & 2. MixedClusterClientYamlTestSuiteIT — search/310_match_bool_prefix (partial term & complete term)

• Recent build: 80096 (Timer/main, 2026-06-10 18:55Z)
• Error: AssertionError: Failure at [search/310_match_bool_prefix:245/222]: hits.hits.0._id didn't match expected value: expected String [4] but was String [1] — top-hit ordering differs from the YAML expectation.
• Seed: 7E5E289390B42D59:F60A17493E4840A1 (suite 7E5E289390B42D59)
• Reproduction: Did not reproduce. Ran :qa:mixed-cluster:v3.4.0#mixedClusterTest for each param with the suite seed; a mixed 3.4.0/3.7.0 cluster was launched and both passed.
• History: Chronic since 2024-03-25. 392 / 379 unique builds all-time. Failures are spiky — large clusters in 2024-04 (~40) and 2024-09 (~135), then a long low-rate tail of ~3–22/month, with a recent uptick in 2026-05 (23) and 7 so far in 2026-06. The two params almost always fail together (same suite). Stable-chronic with periodic spikes.

Monthly unique builds (partial term):

2024-03:5 2024-04:41 2024-05:7 2024-06:6 2024-07:2 2024-08:24 2024-09:137 2024-10:5 2024-11:4 2024-12:3
2025-01:20 2025-02:4 2025-03:7 2025-04:7 2025-05:3 2025-06:6 2025-07:12 2025-08:8 2025-09:5 2025-10:6 2025-11:9 2025-12:5
2026-01:16 2026-02:6 2026-03:9 2026-04:5 2026-05:23 2026-06:7

3. FullRollingRestartIT.testFullRollingRestart_withNoRecoveryPayloadAndSource {p0={replication.strategy=SEGMENT}}

• Recent build: 80067 (Timer/main, 2026-06-10 10:55Z)
• Error: AssertionError: replica shards haven't caught up with primary expected:<18> but was:<15>
• Seed: 31A3C1F12C242F40:2EAABB185694E18 (suite 31A3C1F12C242F40)
• Reproduction: Did not reproduce (:server:internalClusterTest, passed in ~22s).
• History: Chronic since 2024-10-11. 141 unique builds all-time. A large burst in 2025-07 (47) and 2025-08 (24), nearly silent through late 2025, then resurgent across 2026-02 → 2026-06 (17, 13, 9, 19, 9). Chronic with a clear 2026 resurgence.

4. ClusterShardLimitIT.testOpenIndexOverLimit {p0={writable_warm_index.enabled=true}}

• Recent build: 80111 (Timer/main, 2026-06-10 22:55Z)
• Error: IllegalStateException: Some shards are still open after the threadpool terminated. Something is leaking index readers or store references. (suite teardown leak check)
• Seed: B1BD0F1D1A72218C:E0127FAD9488A6FA (suite B1BD0F1D1A72218C)
• Reproduction: Did not reproduce. Both param variants (enabled=false and enabled=true) ran and passed.
• History: First seen 2025-10-15. 71 unique builds all-time. Steady ~6–9/month through early 2026, then a notable jump to 19 in 2026-05 and 11 already in 2026-06. Worsening recently.

Monthly unique builds:

2025-10:2 2025-11:9 2025-12:3 2026-01:7 2026-02:7 2026-03:6 2026-04:7 2026-05:19 2026-06:11

5. WarmIndexSegmentReplicationIT.testReplicaAlreadyAtCheckpoint

• Recent build: 80096 (Timer/main, 2026-06-10 18:55Z)
• Error: bare java.lang.AssertionError (no message)
• Seed: 7E5E289390B42D59
• Reproduction: Did not reproduce (:server:internalClusterTest, passed in ~13s).
• History: Rare. 7 unique builds all-time since 2025-04-29: 2025-04 (1), 2026-04 (4), 2026-05 (1), 2026-06 (1). Low-frequency, with a small cluster in Apr 2026.

6. WarmToHotTieringServiceIT.classMethod

• Recent build: 80162 (Timer/main, 2026-06-11 08:55Z)
• Error: java.io.IOException: Could not remove the following files ... temp/...WarmToHotTieringServiceIT_257B0F59E3AE7A21-001/... — classMethod indicates a suite-level (afterClass) cleanup failure, not a single test method.
• Seed: 257B0F59E3AE7A21
• Reproduction: Did not reproduce (ran whole class via :server:internalClusterTest, passed in ~2m6s).
• History: New. 3 unique builds all-time, first seen 2026-05-19: 2026-05 (2), 2026-06 (1). Newly appearing, low frequency.

---

Notes for triage

• All six failed only on a single recent committed-code build each and none reproduced under a single seeded local run — consistent with timing/concurrency-sensitive flakiness rather than seed-deterministic logic bugs. Seed reproduction for cluster/disruption tests is known to be unreliable.
• Tests Bump com.diffplug.spotless from 5.6.1 to 6.2.2 #3 and Bump opensearch-core from current to 1.2.4 in /buildSrc/src/testKit/opensearch.build #4 show 2026 growth; CI runner hardware/throughput changes can amplify timing-window flakes, but that hypothesis was not investigated here.
• Root-causing was intentionally out of scope for this report.

_{Generated automatically. Raw query output and per-test reproduction logs were retained locally under /tmp/flaky-report/logs/.}