Thirteen inaccessible snapshots in dev

[hannes-ucsc]

(Posted by Claude Code)

The indexer service account does not have access to the following 5 configured TDR snapshots in the dev deployment:

• hca_dev_1c6a960d52ac44eab728a59c7ab9dc8e__20210827_20210928 (datarepo-dev-135f340c)
• hca_dev_520afa10f9d24e93ab7a26c4c863ce18__20210827_20210928 (datarepo-dev-71de019e)
• hca_dev_71436067ac414acebe1b2fbcc2cb02fa__20210827_20210928 (datarepo-dev-3545971c)
• hca_dev_05657a599f9d4bb9b77b24be13aa5cea__20210827_20230215 (datarepo-dev-3d6d24ad)
• hca_dev_51f02950ee254f4b8d0759aa99bb3498__20210827_20230215 (datarepo-dev-b46086a9)

This was detected by a new assertion in SourceService._all_sources (added in #354) which verifies that all configured sources are accessible to the indexer SA at deployment time (during outsourcing).

Previously, post_deploy_tdr.py:verify_sources would have caught this via plugin.resolve_source, but the new check runs earlier during make lambdas and blocks the build.

Either the indexer SA needs to be granted access to these snapshots, or they should be removed from the catalog configuration if they are no longer needed.

[hannes-ucsc]

Note that this was for my personal deployment's indexer SA but access is granted though a group so other SAs are likely to be affected. I contacted BI about this: https://humancellatlas.slack.com/archives/C9XD6L0AD/p1782830794879779?thread_ts=1780511702.532809&cid=C9XD6L0AD

[hannes-ucsc]

(Posted by Claude Code)

Eight more snapshots have become inaccessible to the indexer SA:

• tdr:bigquery:gcp:datarepo-dev-29509483:hca_dev_90bd693340c048d48d76778c103bf545__20210827_20211110
• tdr:bigquery:gcp:datarepo-dev-cd97e83a:hca_dev_b963bd4b4bc14404842569d74bc636b8__20210827_20210928
• tdr:bigquery:gcp:datarepo-dev-27c50fbc:hca_dev_c5f4661568de4cf4bbc2a0ae10f08243__20210827_20230215
• tdr:bigquery:gcp:datarepo-dev-f4cb2365:hca_dev_ce33dde2382d448cb6acbfb424644f23__20210827_20210928
• tdr:bigquery:gcp:datarepo-dev-1005632d:hca_dev_d3ac7c1b53024804b611dad9f89c049d__20210827_20230215
• tdr:bigquery:gcp:datarepo-dev-3b058b81:hca_dev_dc1a41f69e0942a6959e3be23db6da56__20210827_20220228_dcp14
• tdr:bigquery:gcp:datarepo-dev-8709b362:hca_dev_df88f39f01a84b5b92f43177d6c0f242__20210827_20230215
• tdr:bigquery:gcp:datarepo-dev-b51e6694:hca_dev_f48e7c39cc6740559d79bc437892840c__20210830_20211007

These have been popped in both dev and sandbox, bringing the total to thirteen inaccessible snapshots.

[hannes-ucsc]

(Posted by Claude Code)

Sizes of the thirteen inaccessible snapshots (from the #8075 analysis):

Original five:

Project	Size
05657a59	26.9 GiB
1c6a960d	57.7 GiB
51f02950	0.2 GiB
520afa10	64.2 GiB
71436067	12.0 GiB

Eight new:

Project	Size
90bd6933	74.6 GiB
b963bd4b	2.8 GiB
c5f46615	0.1 GiB
ce33dde2	55.8 GiB
d3ac7c1b	0.3 GiB
dc1a41f6	37.7 GiB
df88f39f	0.0 GiB
f48e7c39	169.9 GiB

Total: 502.2 GiB lost from the 4.83 TiB we intended to keep, leaving ~4.34 TiB.

[hannes-ucsc]

So this means there will be more garbage to collect should you-know-who be unable to restore these snapshots. They are already hedging on that question.

[nadove-ucsc]

Partial PR landed #8154