Verbatim PFB manifest fails for some datasets in anvil14

[nadove-ucsc]

[added by @dsotirho-ucsc]

Datasets with a large amount of metadata in their snapshot tables fail to generate a verbatim PFB manifest due to timeout and/or OutOfMemory.

Using dataset information gathered for #8122 (comment), I tested generating a Verbatim PFB manifest for the largest datasets added or updated in anvil14. "Largest" here refers to the total size of the snapshot as JSON. Note that there may be other large datasets in anvil14, inherited from anvil13, that were not tested here.

Failed to generate Verbatim PFB manifest

JSON (MiB)  Files   Dataset                             Result
-------     -----   ----------------------------        ------------------------------------------------------
5202        109     ANVIL_eMERGE_GWAS_HMB_MDS           REPORT RequestId: 95e3ea4e-ef30-4493-b65d-3b7e3d749676 Duration: 403937.52 ms Billed Duration: 403938 ms Memory Size: 3009 MB Max Memory Used: 3009 MB Status: error Error Type: Runtime.OutOfMemory
2838        109     ANVIL_eMERGE_GWAS_GRU               REPORT RequestId: 5914bf31-6456-421c-8d3d-6d0cc76cabc0 Duration: 384158.63 ms Billed Duration: 387257 ms Memory Size: 3009 MB Max Memory Used: 3009 MB Init Duration: 3098.10 ms Status: error Error Type: Runtime.OutOfMemory
2451        109     ANVIL_eMERGE_GWAS_HMB_GSO           REPORT RequestId: 1b2425f2-a7f3-45c1-9fe8-c781e81ca95e Duration: 399135.35 ms Billed Duration: 399136 ms Memory Size: 3009 MB Max Memory Used: 3008 MB Status: error Error Type: Runtime.OutOfMemory
1670        109     ANVIL_eMERGE_GWAS_HMB_PUB_GSO       REPORT RequestId: 7339366e-4073-418f-b953-b72a5f523e2d Duration: 900000.00 ms Billed Duration: 902913 ms Memory Size: 3009 MB Max Memory Used: 3009 MB Init Duration: 2912.42 ms Status: timeout
988         109     ANVIL_eMERGE_GWAS_HMB               REPORT RequestId: 10149e16-6b98-4c38-b474-a280593a83df Duration: 900000.00 ms Billed Duration: 900000 ms Memory Size: 3009 MB Max Memory Used: 3009 MB Status: timeout
721         109     ANVIL_eMERGE_GWAS_GRU_IRB_NPU       REPORT RequestId: 63ed177c-5e59-48da-a882-1885a1059c42 Duration: 858708.21 ms Billed Duration: 858709 ms Memory Size: 3009 MB Max Memory Used: 3009 MB Status: error Error Type: Runtime.OutOfMemory
657         109     ANVIL_eMERGE_GWAS_GRU_IRB_PUB       REPORT RequestId: 4c08327f-bafb-4509-9d1a-ebb011c5c044 Duration: 900000.00 ms Billed Duration: 900000 ms Memory Size: 3009 MB Max Memory Used: 3009 MB Status: timeout
556         109     ANVIL_eMERGE_GWAS_DS_DEM            REPORT RequestId: 0c806513-7d1f-4ec5-9573-c6f99087d146 Duration: 323436.51 ms Billed Duration: 323437 ms Memory Size: 3009 MB Max Memory Used: 3009 MB Status: error Error Type: Runtime.OutOfMemory

Successfully generated Verbatim PFB manifest

JSON (MiB)  Files   Dataset                             Result
-------     -----   ----------------------------        ------------------------------------------------------
355         5953    ANVIL_eMERGE_eMERGEseq_GRU_NPU      REPORT RequestId: 47580e7e-8ce1-48c5-ad2e-e85aeea861a6 Duration: 263989.92 ms Billed Duration: 263990 ms Memory Size: 3009 MB Max Memory Used: 2345 MB
302         109     ANVIL_eMERGE_GWAS_GRU_IRB_PUB_GSO   REPORT RequestId: bec484fc-ef92-41a0-a544-4f3bc353d7ca Duration: 245622.20 ms Billed Duration: 245623 ms Memory Size: 3009 MB Max Memory Used: 2345 MB
268         2520    ANVIL_eMERGE_eMERGEseq_HMB_NPU      REPORT RequestId: 136dbf5d-543f-4c1b-9e8e-cc558fe50973 Duration: 192148.03 ms Billed Duration: 192149 ms Memory Size: 3009 MB Max Memory Used: 2345 MB
228         21      ANVIL_eMERGE_PGRNseq_GRU            REPORT RequestId: d19eb48a-dedb-45d4-905c-4d711bced6a4 Duration: 155341.81 ms Billed Duration: 158144 ms Memory Size: 3009 MB Max Memory Used: 1606 MB Init Duration: 2801.53 ms
164         49608   ANVIL_GREGoR_R05_GRU                REPORT RequestId: e533e26c-79aa-4632-abb0-18196fbd24c9 Duration: 106157.04 ms Billed Duration: 106158 ms Memory Size: 3009 MB Max Memory Used: 1606 MB
155         3042    ANVIL_eMERGE_eMERGEseq_GRU_IRB_PUB_NPU  REPORT RequestId: 2061d7ea-e248-458e-b041-a24b3b7582e1 Duration: 101471.88 ms Billed Duration: 101472 ms Memory Size: 3009 MB Max Memory Used: 1606 MB
145         3013    ANVIL_eMERGE_eMERGEseq_GRU          REPORT RequestId: cc7ca7ea-6a5b-42e4-9de4-54d71c8a3ff8 Duration: 96215.53 ms Billed Duration: 96216 ms Memory Size: 3009 MB Max Memory Used: 1606 MB

[nadove-ucsc]

Assignee to provide description.

[hannes-ucsc]

The problematic anvil14 snapshots caused #8017 and that issue is assigned to identify them. Assignee test the verbatim PFB manifest for the snapshots identified there.

For the failures listed above, and any new ones resulting from the test described in the previous paragraph, the fix will likely be to increase the memory size of the manifest Lambda function. While there is a mix of timeouts and OOM's, we know that excessive garbage collection prior to an OOM can significantly slow down a process. Once the test described in the previous paragraph concludes, spike to repeat the test for all problematic datasets while temporarily increasing the Lambda memory to the maximum.

[dsotirho-ucsc]

The problematic anvil14 snapshots caused #8017 and that issue is #8017 (comment) to identify them. Assignee test the verbatim PFB manifest for the snapshots identified there.

The four snapshots identified in #8017 are a subset of the eight snapshots previously tested and found to fail in this ticket, specifically the four largest:

• ANVIL_eMERGE_GWAS_HMB_MDS
• ANVIL_eMERGE_GWAS_GRU
• ANVIL_eMERGE_GWAS_HMB_GSO
• ANVIL_eMERGE_GWAS_HMB_PUB_GSO

[dsotirho-ucsc]

Once the test described in the previous paragraph concludes, spike to repeat the test for all problematic datasets while temporarily increasing the Lambda memory to the maximum.

I temporarily increased manifest Lambda memory to maximum (3009 -> 10240MB) and tested the snapshots listed in the description in order of largest first.

Result summary:

• The four largest still failed due to timeout
• The next four largest successfully generated (in memory) however failed when writing. (The Lambda's ephemeral storage is set at 512MB and can be increased up to 10 GB, however I did not modify this setting for this test.)
• The next two largest successfully generated.

Four largest: timeout

JSON (MiB)  Files   Dataset                             Result
-------     -----   ----------------------------        ------------------------------------------------------
5202        109     ANVIL_eMERGE_GWAS_HMB_MDS           REPORT RequestId: 587b706d-d6c6-4f1a-9265-f50180f20a67 Duration: 900000.00 ms Billed Duration: 903263 ms Memory Size: 10240 MB Max Memory Used: 8141 MB Init Duration: 3262.07 ms Status: timeout
2838        109     ANVIL_eMERGE_GWAS_GRU               REPORT RequestId: cf72d4aa-15f7-4780-9254-bdb41170b835 Duration: 900000.00 ms Billed Duration: 900000 ms Memory Size: 10240 MB Max Memory Used: 9441 MB Status: timeout
2451        109     ANVIL_eMERGE_GWAS_HMB_GSO           REPORT RequestId: a7ddc962-04be-4afc-8e78-c12afa007597 Duration: 900000.00 ms Billed Duration: 900000 ms Memory Size: 10240 MB Max Memory Used: 8971 MB Status: timeout
1670        109     ANVIL_eMERGE_GWAS_HMB_PUB_GSO       REPORT RequestId: e17bea6a-185d-4ed6-885d-d157181c496c Duration: 900000.00 ms Billed Duration: 900000 ms Memory Size: 10240 MB Max Memory Used: 9778 MB Status: timeout

Next four largest: Logs show success however also OSError (see below)

JSON (MiB)  Files   Dataset                             Result
-------     -----   ----------------------------        ------------------------------------------------------
988         109     ANVIL_eMERGE_GWAS_HMB               REPORT RequestId: 702c85b1-1aa3-449c-a679-948be0091c43 Duration: 624543.48 ms Billed Duration: 624544 ms Memory Size: 10240 MB Max Memory Used: 6287 MB
721         109     ANVIL_eMERGE_GWAS_GRU_IRB_NPU       REPORT RequestId: 9c4632a7-c89b-4116-9303-8808d5f5b120 Duration: 357198.34 ms Billed Duration: 357199 ms Memory Size: 10240 MB Max Memory Used: 6525 MB
657         109     ANVIL_eMERGE_GWAS_GRU_IRB_PUB       REPORT RequestId: 50a0cc0d-f21b-4e26-9f73-bcc1cae8e9d2 Duration: 310351.48 ms Billed Duration: 310352 ms Memory Size: 10240 MB Max Memory Used: 6525 MB
556         109     ANVIL_eMERGE_GWAS_DS_DEM            REPORT RequestId: 81bc22b5-a7b6-4543-99ef-0abf6480dbc2 Duration: 253785.98 ms Billed Duration: 253786 ms Memory Size: 10240 MB Max Memory Used: 6525 MB

Successful

JSON (MiB)  Files   Dataset                             Result
-------     -----   ----------------------------        ------------------------------------------------------
355         5953    ANVIL_eMERGE_eMERGEseq_GRU_NPU      REPORT RequestId: 62acb886-e7a7-4f7d-8545-50762bc6c4ff Duration: 202950.63 ms Billed Duration: 205987 ms Memory Size: 10240 MB Max Memory Used: 2350 MB Init Duration: 3036.01 ms
302         109     ANVIL_eMERGE_GWAS_GRU_IRB_PUB_GSO   REPORT RequestId: 1034d73a-ef60-4993-bca6-0dd5d92df44a Duration: 218.89 ms Billed Duration: 219 ms Memory Size: 768 MB Max Memory Used: 191 MB

OSError details

[ERROR] OSError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/var/task/azul/chalice.py", line 193, in patched_event_source_handler
    return old_handler(self_, event, context)
  File "/var/task/chalice/app.py", line 1761, in __call__
    return self.handler(event_obj)
  File "/var/task/chalice/app.py", line 1707, in __call__
    return self._original_func(event.to_dict(), event.context)
  File "/var/task/azul/service/manifest_controller.py", line 422, in generate_manifest
    return self.generate(event)
  File "/var/task/azul/service/manifest_controller.py", line 695, in generate
    result = self._service.get_manifest(format=manifest_key.format,
  File "/var/task/azul/service/manifest_service.py", line 633, in get_manifest
    return self._generate_manifest(generator, manifest_key, partition)
  File "/var/task/azul/service/manifest_service.py", line 642, in _generate_manifest
    partition = generator.write(manifest_key, partition)
  File "/var/task/azul/service/manifest_service.py", line 1459, in write
    file_path, base_name = self.create_file()
  File "/var/task/azul/service/manifest_service.py", line 2219, in create_file
    avro_pfb.write_pfb_entities(pfb_entities(), pfb_schema, path)
  File "/var/task/azul/service/avro_pfb.py", line 79, in write_pfb_entities
    with open(path, 'w+b') as fh:

I have reverted all my changes to the manifest Lambda made for this testing.

[dsotirho-ucsc]

@hannes-ucsc: "Assignee to continue experiment with the partially successful ones using 10GB of ephemeral storage. Assignee to also to get an idea of how far the unsuccessful ones got to finishing by looking at the search-after parameter that we send to OpenSearch."

[dsotirho-ucsc]

Verbatim PFB results with Lambda memory & ephemeral storage increased to maximum (both 10240 MB)

Dataset                         JSON (MiB)  Result
----------------------------    -------     ------------------------------------------
ANVIL_eMERGE_GWAS_HMB_MDS       5202        Timeout 28.9% (search_after 49fa71f7)
ANVIL_eMERGE_GWAS_GRU           2838        Timeout 58.9% (search_after 9667556e)
ANVIL_eMERGE_GWAS_HMB_GSO       2451        Timeout 65.6% (search_after a744b89e)
ANVIL_eMERGE_GWAS_HMB_PUB_GSO   1670        Timeout 99.9% (search_after fffffc6e)
ANVIL_eMERGE_GWAS_HMB           988         File Size: 654M, Duration: 735552.73 ms Memory Size: 10240 MB Max Memory Used: 6536 MB
ANVIL_eMERGE_GWAS_GRU_IRB_NPU   721         File Size: 476M, Duration: 487693.14 ms Memory Size: 10240 MB Max Memory Used: 6536 MB
ANVIL_eMERGE_GWAS_GRU_IRB_PUB   657         File Size: 424M, Duration: 400378.15 ms Memory Size: 10240 MB Max Memory Used: 6536 MB
ANVIL_eMERGE_GWAS_DS_DEM        556         File Size: 363M, Duration: 339432.91 ms Memory Size: 10240 MB Max Memory Used: 3653 MB

[hannes-ucsc]

I think we should look into generating the manifest in two passes by using OpenSearch's reindex feature. The first pass would use that feature to copy matching replicas into a new, temporary index, eliminating the need to deduplicate replicas in Lambda memory. The second pass would prepare the PFB by reading from that temporary index. The first pass could even be partitioned, eliminating the 15 min limit. Whether the second pass could be partitioned remains to be determined. It hinges upon the question as to whether we can write a PFB in parts.

[hannes-ucsc]

Assignee to file a PR that increases memory and ephemeral storage of the manifest Lambda function in stable AnVIL deployments (avoid hardcoding anvilprod) to the maximum allowed by AWS. Assignee to estimate the expected increase in Lambda billing based on the duration of all executions of the manifest Lamnda function over the past six months. Assignee to verify the calculation by comparing the estimated billing with the actual cost over that timeframe.