Report actual wire bytes in COPY (cumulative) Transfer column#46
Merged
Conversation
…imitri#258) (dimitri#1002) * fix: report actual wire bytes in COPY (cumulative) Transfer column (dimitri#258) The top-level summary has always had a Transfer column in the COPY (cumulative) row but it was populated with sourceTable->bytes — the on-disk size from pg_table_size() on the source — rather than the actual bytes sent over the wire during COPY. The real wire byte count is already tracked buffer-by-buffer in CopyTableSummary.bytesTransmitted (pgsql.c) and stored in the summary table after each table copy (table-data.c:1467/1502). The only gap was that summary_increment_timing for TIMING_SECTION_COPY_DATA was reading the catalog size instead of the transmitted count. Changing that one argument makes the COPY (cumulative) row display the true amount of data moved across the wire, e.g.: COPY (cumulative) both 12h01m 47 GB 4 Source catalog size, target relation size, and wire bytes legitimately differ due to bloat, TOAST, protocol framing, and storage version differences, so the catalog value was never a useful proxy. Closes dimitri#258 * feat: show bytes transferred in list progress (dimitri#258) pgcopydb list progress now prints a Bytes row alongside Tables and Indexes: Tables | 42 | 3 | 39 Indexes | 150 | 12 | 138 Bytes | 100 GB | | 95 GB The 'Done' column accumulates bytes from: - completed tables: their final bytesTransmitted from summary_finish_table - in-progress tables: the last periodic flush from summary_update_table_copy_stats That hook already wrote bytesTransmitted to summary.bytes every 5 s; what was missing was reading it back in copydb_update_progress. Jitter is added to the periodic flush so N table workers do not all wake at the same second. Each worker's first write fires between 1 and WRITE_INTERVAL_SECS seconds after startup, offset by getpid() % WRITE_INTERVAL_SECS. Subsequent writes remain every WRITE_INTERVAL_SECS seconds, but staggered. rand() is not used (it is banned); getpid() is a deterministic, non-repeating source adequate for this purpose. The JSON output gains a 'bytes' object with 'total', 'transferred', 'total-pretty', and 'transferred-pretty' fields. (cherry picked from commit fe9efbc)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
pgcopydb list progressreported only table/index counts (no byte volume), andlist progress --summaryshowed a blank Transfer column for theCOPY (cumulative)row — it was populated fromsourceTable->bytes(catalogpg_table_size()) rather than actual wire bytes.Fix (ports upstream dimitri/pgcopydb dimitri#1002)
table-data.c: passsummary.bytesTransmitted(actual wire bytes) tosummary_increment_timing(TIMING_SECTION_COPY_DATA, …)instead ofsourceTable->bytes.summary.bytes) inlist progress.Observability/progress-accuracy fix; no change to what data gets copied.
Tests
Full
PGVERSION=18 make testsgreen (all suites, exit 0). Clean cherry-pick.