Flaky test report: committed-code failures on 2026-06-17

[andrross]

Flaky test report: committed-code failures on 2026-06-17

This report summarizes test failures observed on main (Timer + Post Merge Action runs) in the OpenSearch metrics cluster over the last 24 hours, with historical context and a local reproduction attempt for each.

Methodology

• Source of failures: the gradle-check-* indices on metrics.opensearch.org, filtered to invoke_type=Timer & git_reference=main or invoke_type=Post Merge Action & pull_request_title="push trigger main..." (the canonical "against committed code" filter from the steering docs).
• Window: now-24h to current time (2026-06-17 ~10:00 UTC).
• Historical pattern: monthly aggregations with cardinality(build_number) to count unique builds affected, broken down for all build types (PR + Timer + Post Merge) and for committed-code only. This avoids inflating numbers when the same build emits multiple records.
• Seeds: extracted from __randomizedtesting.SeedInfo.seed([SUITE:METHOD]) in the Jenkins test report stack traces. Most recent failure events do not include tests.seed= directly, so the SeedInfo annotation is the source of truth.
• Local repro: ran each test once with the captured seed using the appropriate gradle task in this checkout. A single deterministic-pass with the seed is not proof the seed reproduces the flake — these are integration / concurrent / environment-sensitive tests for which the steering docs explicitly call out that seeds are typically not reliable reproducers. We did not iterate (-Dtests.iters=N) here; this is a triage pass, not a root-cause investigation.

Caveats

• Local repo state: working copy on commit 90fdbf95a0f with four staged files in server/.../concurrent/ and ThreadPool.java. Unrelated to any of the tests run, but worth noting.
• JDK 25 (Temurin 25.0.2) on a dev desktop. CI runs on m7a.8xlarge runners; relative CPU speed and timing differ.
• One test (IndexingIT.testIndexingWithSegRep) is in qa/rolling-upgrade and runs against a BWC version matrix; reproducing in isolation requires building old version distributions and orchestrating multi-stage upgrades, so this one was not attempted locally.
• One test (GoogleCloudStorageBlobContainerRetriesTests) initially failed to build due to docker-compose buildx being too old on the dev host; reran successfully with -x composeUp -x composeBuild -x buildFixture since the actual test class is a unit test that uses an in-process mock HTTP server.

Summary table

Sorted by total unique builds affected historically (all build types), descending.

#	Test	Failing build	Local repro?	First seen	Total unique builds (all types)	Trend
1	IndexingIT.testIndexingWithSegRep	80636	not attempted (BWC matrix)	2024-03	274	chronic, stable to slightly worsening
2	FullRollingRestartIT.testFullRollingRestart_withNoRecoveryPayloadAndSource	80608	no	2024-10	145	worsening — recent surge in 2026
3	RemoteStoreReplicationSourceTests.testGetMergedSegmentFilesDownloadTimeout	80698	no (suite seed only)	2025-08	104	recent uptick after months of quiet
4	Netty4HttpRequestSizeLimitIT.testDoesNotLimitExcludedRequests	80652	no	2025-04	49	chronic, slightly improving
5	GoogleCloudStorageBlobContainerRetriesTests.testWriteLargeBlob	80698	no	2024-04	30	chronic, low rate
6	ClusterShardLimitIT.testRestoreSnapshotOverLimit	80671	no	2025-10	19	stable
7	OsProbeTests.testGetProcessNativeMemoryBytes_returnsDifferenceWhenRssAnonExceedsCommitted	80665	no	2026-05	14	new and worsening
8	RemoteCloseIndexIT.testCloseWhileIndexingDocuments	80614	no	2026-05	2	new, very low rate

None of the captured seeds produced a deterministic reproduction locally. Per the steering guidance, that is the expected outcome for IT and concurrency-sensitive tests (the seed controls Random streams but not thread scheduling, network timing, GC, etc.), so this is a signal about the kind of flake, not whether the failures are real.

---

Per-test details

1. IndexingIT.testIndexingWithSegRep

• Failing build: 80636 (Timer, main)
• Module / suite: qa/rolling-upgrade, org.opensearch.upgrades.IndexingIT
• Seed: E1F0BDEAA9582338:6E89ABA75D01D6F9
• Failure (CI): java.lang.AssertionError: expected:<0> but was:<1> from waitForSearchableDocs (IndexingIT.java:121 → :324). Test ran against v2.19.6 BWC stage.
• Local repro: not attempted. Rolling-upgrade tests orchestrate multi-version clusters across oldClusterTest / oneThirdUpgradedTest / twoThirdsUpgradedTest / upgradedClusterTest; reproducing in isolation requires building old version distributions and is impractical for a triage pass.
• First seen: 2024-03-25
• Total unique builds: 274 (PR: 194, Timer: 40, Post Merge: 28)
• Committed-code monthly counts: consistent presence — 2025-08: 8, 2025-11: 6, 2026-01: 5, 2026-02: 7, 2026-05: 7, 2026-06: 3 (so far)
• Pattern: highly chronic, present in nearly every month for two years, with a stable to slightly worsening rate. By far the most-impactful flaky test in this list.

2. FullRollingRestartIT.testFullRollingRestart_withNoRecoveryPayloadAndSource

• Failing build: 80608 (Timer, main)
• Module / suite: server internalClusterTest, org.opensearch.recovery.FullRollingRestartIT
• Parameter: {p0={"cluster.indices.replication.strategy":"SEGMENT"}}
• Seed: 2C364803B7379115:1F7F22431E7AF04D
• Failure (CI): replica shards haven't caught up with primary expected:<18> but was:<15> inside waitForReplication → assertBusy.
• Local repro: NO. ./gradlew :server:internalClusterTest --tests "...FullRollingRestartIT.testFullRollingRestart_withNoRecoveryPayloadAndSource" -Dtests.seed=2C364803B7379115:1F7F22431E7AF04D → BUILD SUCCESSFUL.
• First seen: 2024-10-11
• Total unique builds: 145 (PR: 88, Timer: 47, Post Merge: 10)
• Committed-code monthly counts: 2025-07: 14, 2025-08: 10, 2026-02: 9, 2026-03: 4, 2026-04: 5, 2026-05: 5, 2026-06: 7
• Pattern: two distinct surges — Jul-Aug 2025 (47+24 unique builds across all build types) and again from Feb 2026 onward (17, 13, 9, 19, 13). Currently worsening on main.

3. RemoteStoreReplicationSourceTests.testGetMergedSegmentFilesDownloadTimeout (and classMethod)

• Failing build: 80698 (Timer, main)
• Module / suite: server test, org.opensearch.indices.replication.RemoteStoreReplicationSourceTests
• Seed (suite-level only): 92ABBEC2AD1ADD02
• Failure (CI): Test abandoned because suite timeout was reached (1200000 msec). The companion classMethod failure on the same build is the same root event, recorded twice.
• Local repro: NO. ./gradlew :server:test --tests "...RemoteStoreReplicationSourceTests.testGetMergedSegmentFilesDownloadTimeout" -Dtests.seed=92ABBEC2AD1ADD02 completed in seconds → BUILD SUCCESSFUL. Note: the CI failure is suite-timeout, so only the suite seed was captured (no method seed available); local one-shot run cannot definitively rule out a hang under suite-level interleaving.
• First seen: 2025-08-04
• Total unique builds: 104 (PR: 55, Timer: 32, Post Merge: 17)
• Committed-code monthly counts: 2025-08: 18, 2025-09: 13, 2025-10: 9, then 1/mo through 2026-04, then 2026-05: 3, 2026-06: 2
• Pattern: large initial surge Aug-Oct 2025, mostly quiet through early 2026, then a clear uptick in May-June 2026. Likely back-on the radar.

4. Netty4HttpRequestSizeLimitIT.testDoesNotLimitExcludedRequests

• Failing build: 80652 (Post Merge Action, main)
• Module / suite: modules/transport-netty4 internalClusterTest, org.opensearch.http.netty4.Netty4HttpRequestSizeLimitIT
• Seed: FF778A71215778B3:50633E82CA922F3
• Failure (CI): Expected all requests with status [200 OK] but [112] requests had a different one — Expected: <0L> but: was <112L>.
• Local repro: NO. ./gradlew :modules:transport-netty4:internalClusterTest --tests "...Netty4HttpRequestSizeLimitIT.testDoesNotLimitExcludedRequests" -Dtests.seed=FF778A71215778B3:50633E82CA922F3 → BUILD SUCCESSFUL.
• First seen: 2025-04-29
• Total unique builds: 49 (PR: 37, Timer: 9, Post Merge: 3)
• Committed-code monthly counts: 1/mo from 2025-05 through 2026-04 with a 2026-03 spike (PR-heavy: 9 total unique builds that month), then 2026-06: 1
• Pattern: chronic but low rate; net trend slightly improving since the March 2026 spike.

5. GoogleCloudStorageBlobContainerRetriesTests.testWriteLargeBlob

• Failing build: 80698 (Timer, main)
• Module / suite: plugins/repository-gcs test (unit), org.opensearch.repositories.gcs.GoogleCloudStorageBlobContainerRetriesTests
• Seed: 92ABBEC2AD1ADD02:EDFE8681DEC2AEBA
• Failure (CI): com.google.cloud.storage.StorageException: Unknown Error thrown from JsonResumableSession.put during writeBlobResumable (the test simulates GCS errors against an in-process mock HTTP server, so this is a test-internal timing race, not a real GCS issue).
• Local repro: NO. After working around a docker-compose/buildx version mismatch on the dev host with -x composeUp -x composeBuild -x buildFixture (the unit test itself does not need the gcs-fixture container), the test passed → BUILD SUCCESSFUL.
• First seen: 2024-04-16
• Total unique builds: 30 (PR: 23, Timer: 4, Post Merge: 2)
• Committed-code monthly counts: 1 hit in 2025-03, 2025-10, 2025-11, 2026-03, 2026-06. About one committed-code failure every couple of months.
• Pattern: very chronic, very low rate. The same unit test class appears repeatedly in this report's index because it has multiple flaky methods.

6. ClusterShardLimitIT.testRestoreSnapshotOverLimit

• Failing build: 80671 (Post Merge Action, main)
• Module / suite: server internalClusterTest, org.opensearch.cluster.shards.ClusterShardLimitIT
• Parameter: {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
• Seed: C199E0970DD06B71:135A8B9F94D1FC76
• Failure (CI): Expected: <16> but: was <15> at ClusterShardLimitIT.java:596 — a shard-count assertion off by one, likely an early-snapshot-state read race.
• Local repro: NO. ./gradlew :server:internalClusterTest --tests "...ClusterShardLimitIT.testRestoreSnapshotOverLimit" -Dtests.seed=C199E0970DD06B71:135A8B9F94D1FC76 → BUILD SUCCESSFUL. Note: local run is not parameterized with the writable-warm feature flag — that may matter; we did not retest with the flag set.
• First seen: 2025-10-15
• Total unique builds: 19 (PR: 11, Post Merge: 4, Timer: 4)
• Committed-code monthly counts: 1 in 2025-11, 1 in 2026-02, 1 in 2026-03, 3 in 2026-04, 1 in 2026-06
• Pattern: stable low-rate flake. Slight elevation correlated with writable-warm feature work.

7. OsProbeTests.testGetProcessNativeMemoryBytes_returnsDifferenceWhenRssAnonExceedsCommitted

• Failing build: 80665 (Timer, main)
• Module / suite: server test (unit), org.opensearch.monitor.os.OsProbeTests
• Seed: FBBF90AE8E4B39:702DF1DAF76CC496
• Failure (CI): expected:<67108864> but was:<67043328> — the two values differ by exactly 65,536 (one 64K page), consistent with platform/page-size dependent memory accounting.
• Local repro: NO. ./gradlew :server:test --tests "...OsProbeTests.testGetProcessNativeMemoryBytes_returnsDifferenceWhenRssAnonExceedsCommitted" -Dtests.seed=FBBF90AE8E4B39:702DF1DAF76CC496 → BUILD SUCCESSFUL. The test reads system-level memory metrics so it's environment-sensitive rather than seed-deterministic.
• First seen: 2026-05-25 (very recent)
• Total unique builds: 14 (PR: 9, Timer: 3, Post Merge: 2)
• Committed-code monthly counts: 2026-05: 1, 2026-06: 4 (in only ~half the month)
• Pattern: new and worsening. Worth prioritizing — the rapid growth suggests either a recent change to the production code under test, or a recently-changed test invariant.

8. RemoteCloseIndexIT.testCloseWhileIndexingDocuments

• Failing build: 80614 (Timer, main)
• Module / suite: server internalClusterTest, org.opensearch.indices.state.RemoteCloseIndexIT
• Parameter: {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}
• Seed: B554F7E5E2E76E18:39F396858831487
• Failure (CI): Shard [tfospottup][1] is still locked after 5 sec waiting in InternalTestCluster.assertAfterTest — shard-lock leak on test teardown. Test passed but did not clean up cleanly.
• Local repro: NO. ./gradlew :server:internalClusterTest --tests "...RemoteCloseIndexIT.testCloseWhileIndexingDocuments" -Dtests.seed=B554F7E5E2E76E18:39F396858831487 → BUILD SUCCESSFUL. Local run not parameterized with writable-warm feature flag.
• First seen: 2026-05-05
• Total unique builds: 2 (Timer: 2)
• Committed-code monthly counts: 2026-05: 1, 2026-06: 1
• Pattern: very new, only seen on Timer runs against main, both times with the writable-warm parameter. Single-digit-occurrence test — flag for monitoring rather than action yet.

---

Notes on local-reproduction methodology

For every test where BUILD SUCCESSFUL was reported above, only one run was performed with the original seed. Per the project's flaky-test investigation guidance, a single pass with a captured seed is not evidence the seed is non-reproducing — the steering docs are explicit that seeds for IT and disruption-style tests are typically not deterministic reproducers, and that ten or more iterations would be needed to characterize the actual rate. This report does not draw causal conclusions; it ranks the failures by historical impact and flags which are recent/worsening.

If anyone picks one of these up for a real investigation, the next step on the most-impactful ones (IndexingIT.testIndexingWithSegRep, FullRollingRestartIT.testFullRollingRestart_withNoRecoveryPayloadAndSource, RemoteStoreReplicationSourceTests.testGetMergedSegmentFilesDownloadTimeout) would be -Dtests.iters=20 over a long window with targeted logging, not seed-replay.