Fix monthly quota baseline recovery after aggregation gaps

[crliao]

Summary

Fixes AMBRY-14502.

During the Aug 2026 prod-lva1 incident, account 1085's monthly STORAGE_IN_GB baseline was about 18 TB below peer fabrics. Once current aggregation caught up, the apparent monthly delta jumped from about 13 TB to 31.5 TB and uploads received 429s.

The task writes AggregatedAccountReports before handling the monthly baseline. AggregatedAccountReports.updatedAt is therefore always fresh after the write and cannot identify a pre-write gap. Row timestamps also cannot prove that a host report is complete because unchanged rows are not rewritten and report upserts/deletes span multiple transactions.

This change adds durable completion state and gates baseline recovery on measurable report coverage:

• every host publication marks HostAccountReportsState in progress before changing AccountReports, then complete only after all upserts and deletes succeed;
• recovery reads a host only when its completed report timestamp is at or after the persisted recovery marker and the publication state is unchanged before/after the row read;
• reported partition IDs are persisted with the marker, including empty partitions;
• a pending recovery remains armed for any number of aggregation cycles until completed post-gap reports cover every numeric partition in Helix ideal state;
• current aggregation continues to be written every cycle regardless of recovery readiness;
• once coverage is complete, the recovered aggregate is written and the monthly baseline/state/version are replaced in one transaction;
• same-month gaps do not reset legitimate monthly usage, initial null state is rollout bootstrap rather than an inferred outage, and optimistic state transitions prevent repeated resets across retries/restarts/concurrent tasks;
• quota readers reload when month or snapshot version changes and atomically replace/retry local backup persistence.

The recovery reset can slightly under-report usage for the remainder of that month because the corrected baseline starts near zero. That is intentional: avoiding customer-facing 429s from a stale baseline is the higher-priority failure mode.

Schema, deployment, and rollback

The required schema counterpart is AmbryLI#4141. It adds the three backward-compatible AggregatedAccountReportsMonth state columns plus HostAccountReportsState. No backfill is required or safe: monthly state retains NULL/NULL/0, and each host state row is created by its next successful report publication.

Required rollout order:

1. merge and apply AmbryLI#4141 through Pretzel in EI, CORP, then every PROD fabric; verify all columns and HostAccountReportsState exist;
2. deploy all server publishers/aggregators and drain old aggregation tasks;
3. allow completed reports to cover all Helix partitions; verify monthlyBaselineRecoveryMonth clears and snapshotVersion advances if recovery is armed;
4. deploy quota readers last.

This mechanism is intentionally non-retroactive when lastAggregationTimeMs is initially null, so rollout cannot silently infer or repair a baseline corrupted before initialization. For rollback, roll back readers first and aggregators/publishers second; retain the additive columns and table. Old binaries ignore them, and retaining state avoids destructive rollback and supports redeploy.

Durability risk

This does not change blob write/delete paths or client success callbacks. It changes quota/account-report metadata only. A failed host publication remains marked incomplete and is excluded from recovery; the next successful publication supersedes it. Monthly snapshot replacement and state/version advancement are transactional, so SQL failures cannot expose an empty or partially replaced monthly table. Current aggregation writes are never suppressed by an unready recovery gate.

Testing Done

• JAVA_HOME=$(/usr/libexec/java_home -v 17) ./gradlew :ambry-mysql:test --tests com.github.ambry.accountstats.HostAccountReportsStateDaoTest :ambry-mysql:compileIntTestJava :ambry-clustermap:test --tests com.github.ambry.clustermap.MySqlReportAggregatorTaskTest --quiet
• JAVA_HOME=$(/usr/libexec/java_home -v 17) ./gradlew :ambry-mysql:test --quiet
• JAVA_HOME=$(/usr/libexec/java_home -v 17) ./gradlew :ambry-clustermap:test --quiet
• license checks for changed ambry-api, ambry-mysql, and ambry-clustermap source sets
• git diff --check
• MySQL-backed integration execution was not run locally because no ambry_container_storage_stats MySQL service is available; integration test sources compile and CI provides the database-backed run.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 45.62118% with 267 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.63%. Comparing base (52ba813) to head (24d3040).
⚠️ Report is 421 commits behind head on master.

Files with missing lines	Patch %	Lines
...ub/ambry/clustermap/MySqlReportAggregatorTask.java	0.00%	94 Missing ⚠️
...mbry/accountstats/AggregatedAccountReportsDao.java	53.61%	71 Missing and 6 partials ⚠️
...ambry/accountstats/HostAccountReportsStateDao.java	65.16%	24 Missing and 7 partials ⚠️
.../ambry/accountstats/InmemoryAccountStatsStore.java	0.00%	14 Missing ⚠️
...ry/accountstats/AggregatedAccountReportsState.java	42.10%	11 Missing ⚠️
...mbry/quota/storage/MySqlStorageUsageRefresher.java	78.43%	7 Missing and 4 partials ⚠️
...hub/ambry/accountstats/AccountStatsMySqlStore.java	72.97%	7 Missing and 3 partials ⚠️
.../ambry/clustermap/AccountStorageStatsIterator.java	0.00%	9 Missing ⚠️
...ithub/ambry/clustermap/MySqlClusterAggregator.java	0.00%	9 Missing ⚠️
...m/github/ambry/accountstats/AccountStatsStore.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #3298       +/-   ##
=============================================
- Coverage     64.24%   50.63%   -13.61%     
+ Complexity    10398     8736     -1662     
=============================================
  Files           840      942      +102     
  Lines         71755    81222     +9467     
  Branches       8611     9811     +1200     
=============================================
- Hits          46099    41128     -4971     
- Misses        23004    36668    +13664     
- Partials       2652     3426      +774     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.