Skip to content

Add pcp metrics logging#58

Merged
malucius-rh merged 6 commits intomasterfrom
add_pcp_metrics_logging
Feb 17, 2026
Merged

Add pcp metrics logging#58
malucius-rh merged 6 commits intomasterfrom
add_pcp_metrics_logging

Conversation

@malucius-rh
Copy link
Collaborator

Description

This PR adds logging of fio "settings knobs," run results, and some disk usage metrics to the run's PCP archive
This PR updates the results.csv format from colon to comma separated value and adds timestamps to support opensearch integration

Before/After Comparison

Before:
No logging of results and disk metrics with PCP
Sample from results.csv: op:
blocksize_KiB:njobs:ndisks:iodepth:bw_KiB_s:iops:clat_us:lat_us:slat_us
read:1024:1:1:1:2058648.875000:1996:126277:456:500:43

After:
Results are logged with PCP
op,blocksize_KiB,njobs,ndisks,iodepth,bw_KiB_s,iops,clat_us,lat_us,slat_us,Start_Date,End_Date
read,1024,1,1,1,2058648.875000,1996.126277,456,500,43,2026-01-28T13:41:27Z,2026-01-28T13:43:33Z

Clerical Stuff

This closes #56
This closes #57

Relates to JIRA: RPOPC-807
Relates to JIRA: RPOPC-808

Supporting docs:
results_fio.csv
pcp_openmetrics.txt
pcp_disk.txt
fio.out.txt

Adding fio-specific pmlogger.cfg and openmetrics config files
Switch the results files (interim and final) from colon to comma separated values to support OpenSearch integration
@malucius-rh malucius-rh requested a review from a team January 30, 2026 18:44
@malucius-rh malucius-rh self-assigned this Jan 30, 2026
@malucius-rh malucius-rh added the enhancement New feature or request label Jan 30, 2026
@dvalinrh
Copy link
Contributor

dvalinrh commented Feb 3, 2026

Please check the format of the csv file, appears the header for the csv is out of place. It is appearing in the meta data, not with the block of data.

Copy link
Contributor

@dvalinrh dvalinrh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pcp header is
o.w.iterat o.w.runnin o.w.numthr o.w.runtim o.w.throug o.w.latenc o.w.ndisks o.w.njobs o.w.iodept o.w.operat o.w.blksiz o.w.bw o.w.iops o.w.clat o.w.slat o.w.lat

This will cause the backend tool to have knowledge of what the values of operation etc is. Better solution is to have that identifying information in pcp metric, similar to uperf
o.w.iteration o.w.running o.w.numthreads o.w.runtime o.w.throughput o.w.latency o.w.streams_test o.w.rr_test o.w.biderec_test o.w.maerts o.w.numb_networks o.w.tcp_packet o.w.udp_packet o.w.packet_size o.w.number_procs o.w.GB_Sec o.w.trans_sec o.w.lat_usec

it will make the file a bit larger, but it places all the relevant test information in the log, and will make it easier to take this data and convert into the dbase (they donot have worry about what the value in the field means, just if it is 0 it is not used, if > 0 it has meaning.

Change from "1-10" logging of operation type to a 0/1 metric for each potential type. Supposed to be better for downstream database and AI work
@malucius-rh
Copy link
Collaborator Author

malucius-rh commented Feb 11, 2026

Adding updated pcp-openmetrics output post-code revew

pcp_openmetrics_3.txt

@malucius-rh malucius-rh requested a review from dvalinrh February 12, 2026 13:09
Copy link
Contributor

@dvalinrh dvalinrh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dvalinrh dvalinrh added the group_review_lgtm Indicates approval after a group review meeting label Feb 17, 2026
@malucius-rh malucius-rh merged commit 515a477 into master Feb 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request group_review_lgtm Indicates approval after a group review meeting

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update results.csv for OpenSearch migration [ENH] Add PCP/OM metrics collection

2 participants