feat(IDX): introduce the ci/githubstats/query.py tool for answering FAQs on GitHub Actions

[basvandijk]

This adds the ci/githubstats/query.py tool for running "Frequently Asked Queries" on the github stats PostgreSQL database which stores information on all GitHub Action workflow runs of the dfinity/ic repository including bazel invocations.

This is an efficient CLI alternative to running the same queries on Apache Superset (https://superset.idx.dfinity.network/) and will allow AI agents to run the same or similar queries (either using the query.py tool or directly using:

psql -h githubstats.idx.dfinity.network -U githubstats_read -d github

Supported Queries

Currently the following two queries are supported (more will follow):

$ ci/githubstats/query.py top 10 non_success_rate --week
                                    label                                    | total_count | non_success_count | non_success_rate | flaky_count | flaky_rate | timeout_count | timeout_rate | fail_count | fail_rate | p90_duration
-----------------------------------------------------------------------------+-------------+-------------------+------------------+-------------+------------+---------------+--------------+------------+-----------+--------------
 //rs/tests/nested/nns_recovery:nr_large                                     |           8 |                 5 |            62.50 |           1 |      12.50 |             0 |         0.00 |          4 |     50.00 | 00:46:20
 //rs/tests/message_routing:state_sync_malicious_chunk_test                  |          11 |                 5 |            45.45 |           5 |      45.45 |             0 |         0.00 |          0 |      0.00 | 00:13:49
 //rs/tests/nns:rent_subnet_test                                             |         115 |                22 |            19.13 |          10 |       8.70 |             0 |         0.00 |         12 |     10.43 | 00:14:05
 //rs/tests/nested:guestos_upgrade_from_latest_release_to_current_head_nns   |         114 |                19 |            16.67 |           2 |       1.75 |             0 |         0.00 |         17 |     14.91 | 00:14:16
 //rs/tests/nested:guestos_upgrade_from_latest_release_to_current            |         145 |                24 |            16.55 |           3 |       2.07 |             0 |         0.00 |         21 |     14.48 | 00:14:35
 //rs/tests/nested:guestos_upgrade_smoke_test_head_nns                       |         114 |                17 |            14.91 |           0 |       0.00 |             0 |         0.00 |         17 |     14.91 | 00:14:30
 //rs/nns/integration_tests:upgrade_canisters_with_golden_nns_state          |           7 |                 1 |            14.29 |           0 |       0.00 |             0 |         0.00 |          1 |     14.29 | 00:17:31
 //rs/nns/integration_tests:node_provider_remuneration_with_golden_nns_state |           7 |                 1 |            14.29 |           0 |       0.00 |             0 |         0.00 |          1 |     14.29 | 00:13:44
 //rs/tests/nested:guestos_upgrade_smoke_test                                |         145 |                20 |            13.79 |           0 |       0.00 |             0 |         0.00 |         20 |     13.79 | 00:14:40
 //rs/tests/boundary_nodes:api_bn_integration_test                           |         305 |                34 |            11.15 |          32 |      10.49 |             0 |         0.00 |          2 |      0.66 | 00:03:58
(10 rows)

$ ci/githubstats/query.py last-runs --flaky //rs/tests/nns:rent_subnet_test --week --verbose
# Last flaky runs of //rs/tests/nns:rent_subnet_test in the last week:
psql -h githubstats.idx.dfinity.network -U githubstats_read -d github << EOF
SELECT
  bi.build_date
, bt.total_run_duration / 60 AS total_run_duration_min
, CASE
      WHEN bt.overall_status = 1 THEN 'SUCCESS'
      WHEN bt.overall_status = 2 THEN 'FLAKY'
      WHEN bt.overall_status = 3 THEN 'TIMEOUT'
      WHEN bt.overall_status = 4 THEN 'FAILED'
  END AS status
, 'https://dash.idx.dfinity.network/invocation/' || bi.build_id AS buildbuddy_url
, bi.head_branch
, 'https://github.com/dfinity/ic/commit/' || bi.head_sha AS commit
FROM bazel_tests       AS bt
JOIN bazel_invocations AS bi ON bt.build_id = bi.build_id
WHERE
   bt.label = '//rs/tests/nns:rent_subnet_test'
   AND bt.overall_status IN (2)
   AND
     CASE
       WHEN 'week' <> '' THEN bi.build_date > now() - ('1 week'::interval)
       ELSE TRUE
     END
ORDER BY bi.build_date DESC;
EOF

          build_date           | total_run_duration_min | status |                                  buildbuddy_url                                  |           head_branch           |                                    commit
-------------------------------+------------------------+--------+----------------------------------------------------------------------------------+---------------------------------+-------------------------------------------------------------------------------
 2026-01-16 15:36:54.411406+00 |                     13 | FLAKY  | https://dash.idx.dfinity.network/invocation/052b6cae-2529-4084-aeba-bb2143eb8c2c | nm-ic-gateway                   | https://github.com/dfinity/ic/commit/4df3e55cbe1c26724941e3998644838225af577e
 2026-01-15 08:45:08.912011+00 |                     12 | FLAKY  | https://dash.idx.dfinity.network/invocation/95e9b1a0-ac69-40d1-8763-1e0a58380d9d | master                          | https://github.com/dfinity/ic/commit/1b744b485be1197080b1de029ab02cbc285755ab
 2026-01-14 15:32:30.392309+00 |                     14 | FLAKY  | https://dash.idx.dfinity.network/invocation/1e154ad2-712b-47eb-8ffa-7a6c4b270834 | master                          | https://github.com/dfinity/ic/commit/33fb61d28024ef33e0e8aba50c42ddc08ba75eb5
 2026-01-14 14:57:39.857046+00 |                     12 | FLAKY  | https://dash.idx.dfinity.network/invocation/16410371-8d1f-470f-9127-84b21bbe8304 | rjb/remove-old-prefixes-from-fw | https://github.com/dfinity/ic/commit/a2946890d9fbfeae66062daec08c79ba7eec3df9
 2026-01-14 13:40:08.762126+00 |                     12 | FLAKY  | https://dash.idx.dfinity.network/invocation/4409ebaf-d1d5-4391-b061-a5b593577fe3 | master                          | https://github.com/dfinity/ic/commit/5cfda155427558233c882bf2104e71bad2c22734
 2026-01-13 11:56:19.512686+00 |                     14 | FLAKY  | https://dash.idx.dfinity.network/invocation/9c4ba4df-4070-44e7-8d0e-e47d2df9de86 | nm-dep-dict                     | https://github.com/dfinity/ic/commit/e6ee3a42594551179c3ff6c840f1e1e99895f1ff
 2026-01-13 10:34:53.425971+00 |                     12 | FLAKY  | https://dash.idx.dfinity.network/invocation/a85024e7-29ae-492d-b8fc-ac481f90bdb2 | delete-bazel-files              | https://github.com/dfinity/ic/commit/ea0ab952a1cb5754a3f2a90e8038430ada9315af
 2026-01-12 16:56:15.924008+00 |                     13 | FLAKY  | https://dash.idx.dfinity.network/invocation/ebba0833-c7e5-4be4-bd75-6ebf957ada79 | master                          | https://github.com/dfinity/ic/commit/4e2d08b10ca96716942e622bd589825e225f788d
 2026-01-09 23:28:26.212729+00 |                     13 | FLAKY  | https://dash.idx.dfinity.network/invocation/d9bc758f-be16-4457-ad0e-6c97d4a40b25 | shuo/synced_marker              | https://github.com/dfinity/ic/commit/5763cb32ff54034fd935a00da8237b1b7ce6b9b7
(9 rows)

Future Work

We'll install psql in the dev container once the dev container as been split from the ci container (#8346).

For now get psql in other ways like:

nix shell nixpkgs#postgresql