[feat] Update report schema, add more loggable fields and update JUnit report - #3722
Open
vkarak wants to merge 1 commit into
Open
[feat] Update report schema, add more loggable fields and update JUnit report#3722vkarak wants to merge 1 commit into
vkarak wants to merge 1 commit into
Conversation
Signed-off-by: Vasileios Karakasis <vkarak@gmail.com>
vkarak
added
prio: normal
enhancement
reporting
vkarak
changed the title
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #3722 +/- ##
===========================================
- Coverage 91.92% 91.89% -0.03%
===========================================
Files 62 62
Lines 13808 13913 +105
===========================================
+ Hits 12693 12786 +93
- Misses 1115 1127 +12 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reworks the run report schema to give test cases proper identity fields and complete job/build-job reporting, adds output capture for failed tests (surfaced in both the JSON report and the JUnit XML report), and cleans up a number of inconsistencies uncovered along the way in the logging and storage layers.
DATA_VERSIONbumps from4.2to5.0— this is a breaking schema change; see below.TL;DR
Jobloggable attribute can now be logged and it doesn't need to be re-exposed in theRegressionTest. This makes the design cleaner and adds flexibility.ignore_keys).prebuild_cmds,prerun_cmdsetc.). The only exception ismodules, which can be useful standalone.slurmbackend it's derived from the job accounting data, so in combination with the completion time, the exact elapsed time can be calculated.RegressionTestorJobattribute is loggable.Report schema changes (
reframe/schemas/runreport.json)dependencies_actual,dependencies_conceptual,session_uuidanduuid(per-testcase),time_compile,time_performance,time_run,time_sanity,time_setup,time_total.A test case's identity can be reconstructed from
session_info.uuid+run_index+ the newtestcase_index.build_jobid→build_job_id,build_stdout/build_stderr→build_job_stdout/build_job_stderr.job_*/build_job_*prefixes (job_state,job_nodelist,job_scheduler,job_script_contents,build_job_exitcode, etc.), driven directly offJob's loggable attributes rather than a hand-picked subset.basename,sysenv,testcase_indexper test case;start_time_us/start_timestamp/end_time_us/end_timestamponsession_info(the oldtime_start/time_start_unix/time_end/time_end_unixare kept, but only for the SQLite storage backend).job_stdout_contents,job_stderr_contents,build_job_stdout_contents,build_job_stderr_contents— the tail of the captured build/run output for failed tests (size controlled by the existinggeneral/failure_inspect_linesoption).requiredon bothsession_infoandtestcase_typeis trimmed down to the minimal identifying fields (data_version/hostname/uuid, andname/resultrespectively) rather than an exhaustive list, since almost everything else can legitimately be absent depending on how far a test got before failing.JUnit XML report (
--report-junit)<system-out>/<system-err>(previously only emitted, empty, at the<testsuite>level) and afileattribute pointing at the test's source file.reframe/schemas/junit.xsdis extended to accept both additions — it's now a superset of the original Apache Ant JUnit XSD it started from. This matches what GitLab's Test Reports feature parses; stricter/differently-shaped JUnit consumers (e.g. Jenkins' own XSD) may not accept thefileattribute.Logging /
fileloghandlerfilelog'sprefix,format,format_perfvarsandignore_keysare now schema defaults (reframe/schemas/config.json) instead of being hardcoded in the builtinreframe/core/settings.py. A user defining their ownfileloghandler no longer has to redefine all of these from scratch to get sensible behavior —{'type': 'filelog'}alone now works.prefixis no longer a required option forfilelog.%(check_jobid)s,%(check_job_submit_time)s— silently resolved tonullafter thejob_*rework) are given a real backward-compatible alias to their replacements (%(check_job_id)s,%(check_job_submit_time_us)s), with a one-time deprecation warning instead of silent data loss.%(check_basename)s,%(check_sysenv)s,%(check_filename)s,%(check_fixture)s, the full%(check_job_NAME)s/%(check_build_job_NAME)sfamilies.Bug fixes found along the way
reframe/frontend/reporting/storage.py: session/testcase timestamps were being read from the new RFC3339 string fields (start_timestamp, etc.) and inserted intoREALSQLite columns compared against numeric epoch bounds — silently made every time-range query return zero rows. Fixed to use the numeric fields._job_contents()(report generation) resolved the test'sstdout/stderrinstead of the job's, so build failures reported the run job's (non-existent) output filename instead of the build job's._TCProxy._job_nodelistand a couple of report/printer code paths indexedjob_*fields unconditionally; any test case without a job (e.g. failing beforesetup()) wouldKeyError. Now uses.get()consistently for everything that's conditionally present.Documentation
docs/manpage.rst: new5.0entry in the run-report schema changelog;--report-junitupdated for thesystem-out/system-err/fileadditions and the JUnit-consumer compatibility note.docs/config_reference.rst:fileloghandler options updated (defaults, no-longer-requiredprefix); log placeholder table brought up to date; deprecation note for the renamed placeholders.reframe/core/schedulers/__init__.py(Job) andreframe/core/pipeline.py(RegressionTest): every variable/property that participates in the loggable-attribute mechanism now documents whether it's loggable, and under what name if different from the attribute (:loggable: Yes (logged as \`id``)`).Testing
unittests/test_logging.py,unittests/test_perflogging.py,unittests/test_reporting.py,unittests/test_pipeline.py,unittests/test_meta.py,unittests/test_config.pyall pass.--list-stored-sessions/--list-stored-testcases, and JUnit XML validated against the extended XSD.Example JSON report with failures
{ "session_info": { "data_version": "5.0", "hostname": "tresa.local", "uuid": "d19d9e8a-7464-429e-9b71-2777023f9f51", "start_time_us": 1788563638891562, "start_timestamp": "20260905T011358+0200", "end_time_us": 1788563640134085, "end_timestamp": "20260905T011400+0200", "time_start": "20260905T011358+0200", "time_start_unix": 1788563638.891562, "time_end": "20260905T011400+0200", "time_end_unix": 1788563640.1340852, "time_elapsed": 1.242523193359375, "cmdline": "/Users/user/reframe/.venv/bin/reframe -c unittests/resources/checks/hellocheck.py -r -S postbuild_cmds=false", "config_files": [ "<builtin>" ], "log_files": [ "/var/folders/h7/k7cgrdl13r996m4dmsvjq7v80000gp/T/rfm-2pq3oykm.log" ], "prefix_output": "/Users/user/reframe/output", "prefix_stage": "/Users/user/reframe/stage", "user": "karakasv", "version": "4.11.0-dev3+73d1c13e", "workdir": "/Users/user/reframe", "num_cases": 2, "num_failures": 2, "num_aborted": 0, "num_skipped": 0 }, "runs": [ { "num_cases": 2, "num_failures": 2, "num_aborted": 0, "num_skipped": 0, "run_index": 0, "testcases": [ { "environ": "builtin", "fail_phase": "compile_wait", "fail_reason": "build error: build job failed with exit code: 1", "partition": "default", "result": "fail", "run_index": 0, "testcase_index": 0, "scheduler": "local", "fail_info": { "exc_type": "BuildError", "exc_value": "build job failed with exit code: 1", "traceback": [ " File \"/Users/user/reframe/reframe/frontend/executors/__init__.py\", line 422, in _safe_call\n return fn(*args, **kwargs)\n", " File \"/Users/user/reframe/reframe/core/hooks.py\", line 109, in _fn\n func(obj, *args, **kwargs)\n ~~~~^^^^^^^^^^^^^^^^^^^^^^\n", " File \"/Users/user/reframe/reframe/core/pipeline.py\", line 2771, in compile_wait\n raise BuildError(\n f'build job failed with exit code: {self._build_job.exitcode}'\n )\n" ] }, "fail_severe": false, "basename": "HelloTest", "filename": "/Users/user/reframe/unittests/resources/checks/hellocheck.py", "fixture": false, "sysenv": "generic:default+builtin", "system": "generic", "build_locally": true, "build_time_limit": null, "descr": "C Hello World test", "display_name": "HelloTest", "env_vars": {}, "exclusive_access": false, "hashcode": "2b3e4546", "keep_files": [], "local": false, "maintainers": [ "VK" ], "max_pending_time": null, "modules": [], "name": "HelloTest", "num_cpus_per_task": null, "num_gpus_per_node": null, "num_tasks": 1, "num_tasks_per_core": null, "num_tasks_per_node": null, "num_tasks_per_socket": null, "outputdir": "/Users/user/reframe/output/generic/default/builtin/HelloTest", "perfvalues": {}, "prefix": "/Users/user/reframe/unittests/resources/checks", "readonly_files": [], "short_name": "HelloTest", "stagedir": "/Users/user/reframe/stage/generic/default/builtin/HelloTest", "strict_check": true, "tags": [ "bar", "foo" ], "time_limit": null, "unique_name": "HelloTest", "use_multithreading": null, "variables": {}, "build_job_scheduler": "local", "build_job_completion_time_us": 1788563639895499, "build_job_completion_timestamp": "20260905T011359+0200", "build_job_exitcode": 1, "build_job_id": "74734", "build_job_nodelist": [ "tresa.local" ], "build_job_nodelist_folded": "tresa.local", "build_job_script_contents": "#!/bin/bash\n\n_onerror()\n{\n exitcode=$?\n echo \"-reframe: command \\`$BASH_COMMAND' failed (exit code: $exitcode)\"\n exit $exitcode\n}\n\ntrap _onerror ERR\n\ncc hello.c -o ./HelloTest\nFalse\n", "build_job_script_filename": "rfm_build.sh", "build_job_start_time_us": 1788563639022853, "build_job_start_timestamp": "20260905T011359+0200", "build_job_state": "FAILURE", "build_job_stderr": "rfm_build.err", "build_job_stdout": "rfm_build.out", "build_job_submit_time_us": 1788563639022853, "build_job_submit_timestamp": "20260905T011359+0200", "job_scheduler": "local", "job_completion_time_us": null, "job_completion_timestamp": "20260905T011400+0200", "job_exitcode": null, "job_id": null, "job_nodelist": [], "job_nodelist_folded": "", "job_script_contents": "", "job_script_filename": "rfm_job.sh", "job_start_time_us": null, "job_start_timestamp": "20260905T011400+0200", "job_state": null, "job_stderr": "rfm_job.err", "job_stdout": "rfm_job.out", "job_submit_time_us": null, "job_submit_timestamp": "20260905T011400+0200", "build_job_stdout_contents": "-reframe: command `False' failed (exit code: 1)\n", "build_job_stderr_contents": "" }, { "environ": "builtin", "fail_phase": "compile_wait", "fail_reason": "build error: build job failed with exit code: 1", "partition": "default", "result": "fail", "run_index": 0, "testcase_index": 1, "scheduler": "local", "fail_info": { "exc_type": "BuildError", "exc_value": "build job failed with exit code: 1", "traceback": [ " File \"/Users/user/reframe/reframe/frontend/executors/__init__.py\", line 422, in _safe_call\n return fn(*args, **kwargs)\n", " File \"/Users/user/reframe/reframe/core/hooks.py\", line 109, in _fn\n func(obj, *args, **kwargs)\n ~~~~^^^^^^^^^^^^^^^^^^^^^^\n", " File \"/Users/user/reframe/reframe/core/pipeline.py\", line 2771, in compile_wait\n raise BuildError(\n f'build job failed with exit code: {self._build_job.exitcode}'\n )\n" ] }, "fail_severe": false, "basename": "CompileOnlyHelloTest", "filename": "/Users/user/reframe/unittests/resources/checks/hellocheck.py", "fixture": false, "sysenv": "generic:default+builtin", "system": "generic", "build_locally": true, "build_time_limit": null, "descr": "Compile-only C Hello World test", "display_name": "CompileOnlyHelloTest", "env_vars": {}, "exclusive_access": false, "hashcode": "ecfc0900", "keep_files": [], "local": false, "maintainers": [], "max_pending_time": null, "modules": [], "name": "CompileOnlyHelloTest", "num_cpus_per_task": null, "num_gpus_per_node": null, "num_tasks": 1, "num_tasks_per_core": null, "num_tasks_per_node": null, "num_tasks_per_socket": null, "outputdir": "/Users/user/reframe/output/generic/default/builtin/CompileOnlyHelloTest", "perfvalues": {}, "prefix": "/Users/user/reframe/unittests/resources/checks", "readonly_files": [], "short_name": "CompileOnlyHelloTest", "stagedir": "/Users/user/reframe/stage/generic/default/builtin/CompileOnlyHelloTest", "strict_check": true, "tags": [], "time_limit": null, "unique_name": "CompileOnlyHelloTest", "use_multithreading": null, "variables": {}, "build_job_scheduler": "local", "build_job_completion_time_us": 1788563640016950, "build_job_completion_timestamp": "20260905T011400+0200", "build_job_exitcode": 1, "build_job_id": "74736", "build_job_nodelist": [ "tresa.local" ], "build_job_nodelist_folded": "tresa.local", "build_job_script_contents": "#!/bin/bash\n\n_onerror()\n{\n exitcode=$?\n echo \"-reframe: command \\`$BASH_COMMAND' failed (exit code: $exitcode)\"\n exit $exitcode\n}\n\ntrap _onerror ERR\n\ncc hello.c -o ./CompileOnlyHelloTest\nFalse\n", "build_job_script_filename": "rfm_build.sh", "build_job_start_time_us": 1788563639075075, "build_job_start_timestamp": "20260905T011359+0200", "build_job_state": "FAILURE", "build_job_stderr": "rfm_build.err", "build_job_stdout": "rfm_build.out", "build_job_submit_time_us": 1788563639075075, "build_job_submit_timestamp": "20260905T011359+0200", "build_job_stdout_contents": "-reframe: command `False' failed (exit code: 1)\n", "build_job_stderr_contents": "" } ] } ], "restored_cases": [] }Supersedes #3702.