Skip to content

Read disposable performance from JSON - #38

Merged
marmarek merged 1 commit into
QubesOS:mainfrom
ben-grande:preload-perf-json
Sep 5, 2025
Merged

marmarek merged 1 commit into
QubesOS:mainfrom
ben-grande:preload-perf-json

Conversation

@ben-grande

Copy link
Copy Markdown
Contributor

Comment thread utils/lib/openqa_api.py Outdated
for key, value in data.items():
name = value["name"]
mean = value["mean"]
key = name + "(mean:" + mean + ")"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TypeError: can only concatenate str (not "float") to str

Comment thread utils/lib/openqa_api.py Outdated
name = value["name"]
mean = value["mean"]
key = name + "(mean:" + mean + ")"
result[key] = float(value)

@marmarek marmarek Aug 26, 2025 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here value is a dict - float() doesn't work on it

@marmarek

Copy link
Copy Markdown
Member

And finally, existing past jobs have the old format and the script loads those too for comparison - dropping old format support breaks that.

@marmarek

marmarek commented Aug 26, 2025 •

Copy link
Copy Markdown
Member

BTW, you can run the script yourself, it uses public API (except actually posting the comment, but with --show-results-only option it's skipped).
For example for recent PR jobs, comparing to earlier updates job the call would be:

python3 utils/github_reporting.py --package-list utils/github_package_mapping.json --latest --build 2025082613-4.3 --flavor pull-requests --version 4.3 --compare-to-build 2025081011-4.3 --show-results-only

@ben-grande

Copy link
Copy Markdown
Contributor Author

For example for recent PR jobs, comparing to earlier updates job the call would be:

Yes, I never actually run this as I didn't knew, I should have asked for help/instructions.

Is the command known to take long (more than 5 min)? I am using verbose and there are network connections for the first minute and then it hangs. Maybe there is something happening but not being logged.

@marmarek

Copy link
Copy Markdown
Member

It does take a while with the --instability option, but should be quicker without it. But still, it makes quite a few API calls (all results for jobs in at least two test runs).

@ben-grande

Copy link
Copy Markdown
Contributor Author

I didn't get a report from the public API yet (15 minutes hanging) but did commit on a best effort.

@ben-grande

Copy link
Copy Markdown
Contributor Author
~/openqa-tests-qubesos$ time python3 utils/github_reporting.py --package-list utils/github_package_mapping.json --latest --build 2025082613-4.3 --flavor pull-requests --version 4.3 --compare-to-build 2025081011-4.3 --show-results-only --verbose
2025-08-27 13:59:17,364 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:19,631 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs?build=2025082613-4.3&version=4.3&flavor=pull-requests HTTP/1.1" 200 11377
2025-08-27 13:59:20,454 root: getting job 150652
2025-08-27 13:59:20,487 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:22,982 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150652/details HTTP/1.1" 200 21376
2025-08-27 13:59:23,593 root: creating orphan job for 150652
2025-08-27 13:59:23,606 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:25,099 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150652/details HTTP/1.1" 200 21376
2025-08-27 13:59:25,510 root: Flushing orphan_job 150652 to the database
2025-08-27 13:59:25,703 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:29,004 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150653/details HTTP/1.1" 200 10895
2025-08-27 13:59:29,514 root: Flushing child_job 150653 to the database
2025-08-27 13:59:29,581 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:30,350 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150654/details HTTP/1.1" 200 64710
2025-08-27 13:59:32,111 root: Flushing child_job 150654 to the database
2025-08-27 13:59:32,195 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:34,376 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150655/details HTTP/1.1" 200 12963
2025-08-27 13:59:36,393 root: Flushing child_job 150655 to the database
2025-08-27 13:59:36,425 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:37,549 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150656/details HTTP/1.1" 200 85498
2025-08-27 13:59:50,956 root: Flushing child_job 150656 to the database
2025-08-27 13:59:50,991 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:53,111 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150657/details HTTP/1.1" 200 79799
2025-08-27 13:59:58,239 root: Flushing child_job 150657 to the database
2025-08-27 13:59:58,273 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:59,564 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150658/details HTTP/1.1" 200 61593
2025-08-27 14:00:03,363 root: Flushing child_job 150658 to the database
2025-08-27 14:00:03,402 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 14:00:06,140 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150659/details HTTP/1.1" 200 19809
2025-08-27 14:00:06,419 root: Flushing child_job 150659 to the database
2025-08-27 14:00:06,449 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 14:00:08,184 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150660/details HTTP/1.1" 200 88471
2025-08-27 14:00:18,305 root: Flushing child_job 150660 to the database
2025-08-27 14:00:18,340 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 14:00:19,018 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150661/details HTTP/1.1" 200 76813

@marmarek

Copy link
Copy Markdown
Member

After making all those API calls, it should finish relatively quickly...

Comment thread utils/lib/openqa_api.py
value = data[1]
data_dict = dict(v.split("=", maxsplit=1) for v in data[2:])
key = name + "(avg:" + data_dict["average"] + ")"
result[key] = float(total)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

total is not set in this scope... value?

@ben-grande
ben-grande force-pushed the preload-perf-json branch 3 times, most recently from 4feb3e6 to b37303b Compare August 27, 2025 14:50
@ben-grande

Copy link
Copy Markdown
Contributor Author
  • debian-13-xfce_dispvm (mean:6.428): 102.84
  • debian-13-xfce_dispvm-gui (mean:7.359): 117.75 🟢 ( previous job: 119.40, improvement: 98.62%)
  • debian-13-xfce_dispvm-concurrent (mean:3.09): 49.44
  • debian-13-xfce_dispvm-gui-concurrent (mean:3.917): 62.68 🟢 ( previous job: 64.59, improvement: 97.03%)
  • debian-13-xfce_dispvm-dom0 (mean:7.078): 113.25
  • debian-13-xfce_dispvm-dom0-gui (mean:8.054): 128.87 🔺 ( previous job: 127.44, degradation: 101.12%)
  • debian-13-xfce_dispvm-dom0-concurrent (mean:3.384): 54.15
  • debian-13-xfce_dispvm-dom0-gui-concurrent (mean:4.147): 66.36 🔺 ( previous job: 65.60, degradation: 101.16%)
  • debian-13-xfce_dispvm-preload (mean:3.73): 59.68
  • debian-13-xfce_dispvm-preload-gui (mean:4.607): 73.71
  • debian-13-xfce_dispvm-preload-concurrent (mean:2.85): 45.60
  • debian-13-xfce_dispvm-preload-gui-concurrent (mean:3.806): 60.89
  • debian-13-xfce_dispvm-preload-dom0 (mean:3.84): 61.43
  • debian-13-xfce_dispvm-preload-dom0-gui (mean:5.08): 81.29
  • debian-13-xfce_dispvm-preload-dom0-concurrent (mean:3.279): 52.46
  • debian-13-xfce_dispvm-preload-dom0-gui-concurrent (mean:4.037): 64.59
  • debian-13-xfce_dispvm-api (mean:6.876): 110.01
  • debian-13-xfce_dispvm-gui-api (mean:7.965): 127.44 🟢 ( previous job: 127.48, improvement: 99.97%)
  • debian-13-xfce_dispvm-concurrent-api (mean:3.335): 53.36
  • debian-13-xfce_dispvm-gui-concurrent-api (mean:4.171): 66.74 🔺 ( previous job: 65.39, degradation: 102.06%)
  • debian-13-xfce_dispvm-preload-api (mean:3.868): 61.89
  • debian-13-xfce_dispvm-preload-less-api (mean:6.414): 102.62
  • debian-13-xfce_dispvm-preload-more-api (mean:3.938): 63.00
  • debian-13-xfce_dispvm-preload-gui-api (mean:5.488): 87.81
  • debian-13-xfce_dispvm-preload-concurrent-api (mean:3.172): 50.75
  • debian-13-xfce_dispvm-preload-gui-concurrent-api (mean:3.995): 63.92
  • debian-13-xfce_vm (mean:0.043): 0.69
  • debian-13-xfce_vm-gui (mean:0.04): 0.65 🟢 ( previous job: 7.40, improvement: 8.72%)
  • debian-13-xfce_vm-concurrent (mean:0.028): 0.45
  • debian-13-xfce_vm-gui-concurrent (mean:0.026): 0.41 🟢 ( previous job: 7.33, improvement: 5.65%)
  • debian-13-xfce_vm-api (mean:0.038): 0.60
  • debian-13-xfce_vm-gui-api (mean:0.045): 0.72 🟢 ( previous job: 2.17, improvement: 33.26%)
  • debian-13-xfce_vm-concurrent-api (mean:0.029): 0.47
  • debian-13-xfce_vm-gui-concurrent-api (mean:0.032): 0.51 🟢 ( previous job: 1.82, improvement: 27.86%)
  • fedora-42-xfce_dispvm (mean:7.202): 115.24 🔺 ( previous job: 111.99, degradation: 102.90%)
  • fedora-42-xfce_dispvm-gui (mean:8.523): 136.36 🔺 ( previous job: 131.63, degradation: 103.60%)
  • fedora-42-xfce_dispvm-concurrent (mean:3.681): 58.90 🔺 ( previous job: 57.25, degradation: 102.87%)
  • fedora-42-xfce_dispvm-gui-concurrent (mean:4.418): 70.69 🟢 ( previous job: 74.72, improvement: 94.60%)
  • fedora-42-xfce_dispvm-dom0 (mean:7.76): 124.16 🟢 ( previous job: 124.92, improvement: 99.39%)
  • fedora-42-xfce_dispvm-dom0-gui (mean:8.862): 141.79 🟢 ( previous job: 147.17, improvement: 96.35%)
  • fedora-42-xfce_dispvm-dom0-concurrent (mean:3.812): 60.98 🟢 ( previous job: 64.09, improvement: 95.16%)
  • fedora-42-xfce_dispvm-dom0-gui-concurrent (mean:4.669): 74.70 🟢 ( previous job: 75.59, improvement: 98.82%)
  • fedora-42-xfce_dispvm-preload (mean:4.35): 69.59 🟢 ( previous job: 69.72, improvement: 99.81%)
  • fedora-42-xfce_dispvm-preload-gui (mean:4.951): 79.21 🔺 ( previous job: 79.21, degradation: 100.01%)
  • fedora-42-xfce_dispvm-preload-concurrent (mean:3.027): 48.43 🟢 ( previous job: 49.89, improvement: 97.09%)
  • fedora-42-xfce_dispvm-preload-dom0 (mean:4.37): 69.91 🟢 ( previous job: 72.35, improvement: 96.63%)
  • fedora-42-xfce_dispvm-preload-dom0-gui (mean:5.785): 92.55 🔺 ( previous job: 91.39, degradation: 101.27%)
  • fedora-42-xfce_dispvm-preload-dom0-concurrent (mean:3.578): 57.25 🔺 ( previous job: 57.13, degradation: 100.22%)
  • fedora-42-xfce_dispvm-preload-dom0-gui-concurrent (mean:4.329): 69.26 🔺 ( previous job: 67.89, degradation: 102.02%)
  • fedora-42-xfce_dispvm-api (mean:7.992): 127.87 🟢 ( previous job: 128.15, improvement: 99.78%)
  • fedora-42-xfce_dispvm-gui-api (mean:9.206): 147.29 🟢 ( previous job: 149.03, improvement: 98.83%)
  • fedora-42-xfce_dispvm-concurrent-api (mean:3.919): 62.71 🟢 ( previous job: 66.32, improvement: 94.55%)
  • fedora-42-xfce_dispvm-gui-concurrent-api (mean:4.535): 72.56 🟢 ( previous job: 77.33, improvement: 93.84%)
  • fedora-42-xfce_dispvm-preload-api (mean:4.798): 76.78 🔺 ( previous job: 73.95, degradation: 103.82%)
  • fedora-42-xfce_dispvm-preload-less-api (mean:7.434): 118.94 🔺 ( previous job: 116.39, degradation: 102.19%)
  • fedora-42-xfce_dispvm-preload-more-api (mean:4.441): 71.05 🟢 ( previous job: 71.33, improvement: 99.61%)
  • fedora-42-xfce_dispvm-preload-gui-api (mean:5.622): 89.94 🟢 ( previous job: 92.06, improvement: 97.70%)
  • fedora-42-xfce_dispvm-preload-concurrent-api (mean:3.506): 56.09 🟢 ( previous job: 61.60, improvement: 91.05%)
  • fedora-42-xfce_dispvm-preload-gui-concurrent-api (mean:4.226): 67.62 🟢 ( previous job: 77.66, improvement: 87.07%)
  • fedora-42-xfce_vm (mean:0.022): 0.36 🟢 ( previous job: 9.19, improvement: 3.87%)
  • fedora-42-xfce_vm-gui (mean:0.029): 0.46 🟢 ( previous job: 9.01, improvement: 5.09%)
  • fedora-42-xfce_vm-concurrent (mean:0.02): 0.33 🟢 ( previous job: 8.88, improvement: 3.66%)
  • fedora-42-xfce_vm-gui-concurrent (mean:0.021): 0.34 🟢 ( previous job: 9.15, improvement: 3.75%)
  • fedora-42-xfce_vm-api (mean:0.033): 0.53 🟢 ( previous job: 2.24, improvement: 23.76%)
  • fedora-42-xfce_vm-gui-api (mean:0.039): 0.62 🟢 ( previous job: 2.33, improvement: 26.49%)
  • fedora-42-xfce_vm-concurrent-api (mean:0.027): 0.44 🟢 ( previous job: 1.62, improvement: 27.09%)
  • fedora-42-xfce_vm-gui-concurrent-api (mean:0.032): 0.51 🟢 ( previous job: 2.20, improvement: 23.27%)
  • whonix-workstation-17_dispvm (mean:7.641): 122.26 🟢 ( previous job: 123.87, improvement: 98.69%)
  • whonix-workstation-17_dispvm-gui (mean:8.861): 141.78 🟢 ( previous job: 148.68, improvement: 95.36%)
  • whonix-workstation-17_dispvm-concurrent (mean:4.307): 68.92 🟢 ( previous job: 77.00, improvement: 89.51%)
  • whonix-workstation-17_dispvm-gui-concurrent (mean:5.032): 80.51 🟢 ( previous job: 89.18, improvement: 90.27%)
  • whonix-workstation-17_dispvm-dom0 (mean:8.466): 135.46 🔺 ( previous job: 135.24, degradation: 100.16%)
  • whonix-workstation-17_dispvm-dom0-gui (mean:9.852): 157.63 🟢 ( previous job: 159.23, improvement: 99.00%)
  • whonix-workstation-17_dispvm-dom0-concurrent (mean:4.574): 73.18 🟢 ( previous job: 76.91, improvement: 95.16%)
  • whonix-workstation-17_dispvm-dom0-gui-concurrent (mean:5.138): 82.21 🟢 ( previous job: 87.45, improvement: 94.01%)
  • whonix-workstation-17_dispvm-preload (mean:7.882): 126.11 🔺 ( previous job: 124.31, degradation: 101.45%)
  • whonix-workstation-17_dispvm-preload-gui (mean:8.68): 138.88 🔺 ( previous job: 138.71, degradation: 100.12%)
  • whonix-workstation-17_dispvm-preload-concurrent (mean:3.936): 62.98 🟢 ( previous job: 66.00, improvement: 95.41%)
  • whonix-workstation-17_dispvm-preload-gui-concurrent (mean:4.679): 74.87 🟢 ( previous job: 77.46, improvement: 96.65%)
  • whonix-workstation-17_dispvm-preload-dom0 (mean:4.871): 77.94 🟢 ( previous job: 81.24, improvement: 95.93%)
  • whonix-workstation-17_dispvm-preload-dom0-gui (mean:6.24): 99.84 🟢 ( previous job: 102.71, improvement: 97.21%)
  • whonix-workstation-17_dispvm-preload-dom0-concurrent (mean:4.112): 65.80 🟢 ( previous job: 72.84, improvement: 90.33%)
  • whonix-workstation-17_dispvm-preload-dom0-gui-concurrent (mean:5.006): 80.09 🟢 ( previous job: 87.14, improvement: 91.92%)
  • whonix-workstation-17_dispvm-api (mean:8.358): 133.73 🟢 ( previous job: 140.50, improvement: 95.18%)
  • whonix-workstation-17_dispvm-gui-api (mean:9.79): 156.64 🟢 ( previous job: 157.40, improvement: 99.52%)
  • whonix-workstation-17_dispvm-concurrent-api (mean:4.291): 68.65 🟢 ( previous job: 76.06, improvement: 90.26%)
  • whonix-workstation-17_dispvm-gui-concurrent-api (mean:4.921): 78.73 🟢 ( previous job: 87.18, improvement: 90.30%)
  • whonix-workstation-17_dispvm-preload-api (mean:5.07): 81.12 🟢 ( previous job: 84.09, improvement: 96.47%)
  • whonix-workstation-17_dispvm-preload-less-api (mean:7.886): 126.18 🔺 ( previous job: 126.04, degradation: 100.12%)
  • whonix-workstation-17_dispvm-preload-more-api (mean:5.397): 86.35 🟢 ( previous job: 89.12, improvement: 96.90%)
  • whonix-workstation-17_dispvm-preload-gui-api (mean:6.32): 101.12 🟢 ( previous job: 101.72, improvement: 99.41%)
  • whonix-workstation-17_dispvm-preload-concurrent-api (mean:4.176): 66.82 🟢 ( previous job: 71.58, improvement: 93.35%)
  • whonix-workstation-17_dispvm-preload-gui-concurrent-api (mean:5.025): 80.41 🟢 ( previous job: 88.24, improvement: 91.12%)
  • whonix-workstation-17_vm (mean:0.026): 0.41 🟢 ( previous job: 9.27, improvement: 4.48%)
  • whonix-workstation-17_vm-gui (mean:0.025): 0.40 🟢 ( previous job: 9.82, improvement: 4.02%)
  • whonix-workstation-17_vm-concurrent (mean:0.018): 0.29 🟢 ( previous job: 8.93, improvement: 3.29%)
  • whonix-workstation-17_vm-gui-concurrent (mean:0.025): 0.41 🟢 ( previous job: 9.38, improvement: 4.33%)
  • whonix-workstation-17_vm-api (mean:0.043): 0.69 🟢 ( previous job: 2.56, improvement: 27.07%)
  • whonix-workstation-17_vm-gui-api (mean:0.036): 0.58 🟢 ( previous job: 2.53, improvement: 22.89%)
  • whonix-workstation-17_vm-concurrent-api (mean:0.032): 0.50 🟢 ( previous job: 1.81, improvement: 27.83%)
  • whonix-workstation-17_vm-gui-concurrent-api (mean:0.024): 0.39 🟢 ( previous job: 2.57, improvement: 14.96%)

- btrfs-progs
- python3-nose2
- python3-objgraph
- python3-deepmerge

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrap this in {% if grains['osrelease'] >= '4.3' %} (or just != '4.2') - the package is not available in F37 (R4.2 dom0).

@marmarek
marmarek merged commit 17a410e into QubesOS:main Sep 5, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants