fix: harden client telemetry control flow - #588
Closed
xiaofan-luan wants to merge 1 commit into
Closed
Conversation
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: xiaofan-luan The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #588 +/- ##
===========================================
+ Coverage 53.47% 87.44% +33.96%
===========================================
Files 52 379 +327
Lines 4432 15854 +11422
Branches 0 1729 +1729
===========================================
+ Hits 2370 13863 +11493
+ Misses 2062 1991 -71
🚀 New features to boost your workflow:
|
yhmo
reviewed
Aug 26, 2026
| const auto chunk = std::min(delay, kMaxWaitChunkMs); | ||
| if (condition.wait_for( | ||
| lock, std::chrono::milliseconds(static_cast<int64_t>(chunk)), | ||
| [this, observed_revision]() { return stopped || wait_revision != observed_revision; })) { |
Collaborator
There was a problem hiding this comment.
src/impl/ClientTelemetry.cpp line:664
Medium ---- A server-pushed heartbeat_interval_ms change cannot wake the new chunked wait: the wait predicate only reacts to stopped or wait_revision, but the push_config handler calls condition.notify_all() without bumping wait_revision. With this PR's INT64_MAX interval support chunked into 24h waits, a corrective push (e.g. back to 10s) is deferred until the current chunk elapses. Consider bumping wait_revision in the push_config handler so an interval re-push is applied on the next loop iteration.
yhmo
reviewed
Aug 26, 2026
| SaturatingWindowStart(int64_t end_time, uint64_t interval_ms) { | ||
| const auto signed_interval = static_cast<int64_t>(std::min(interval_ms, kMaxHeartbeatIntervalMs)); | ||
| if (end_time < std::numeric_limits<int64_t>::min() + signed_interval) { | ||
| return std::numeric_limits<int64_t>::min(); |
Collaborator
There was a problem hiding this comment.
src/impl/ClientTelemetry.cpp line:107
Low ---- SaturatingWindowStart saturates to INT64_MIN at the maximum heartbeat interval, and that value is later serialized verbatim as the snapshot "timestamp" in show_latency_history replies. A consumer expecting an epoch-millis timestamp receives -9223372036854775808. Consider capping the window start at the retained-history floor (e.g. end_time - kMaxHistoryRangeMs) so the reported timestamp stays interpretable.
yhmo
reviewed
Aug 26, 2026
| EXPECT_EQ(payload["ignored"], nlohmann::json({"ttl_seconds", "unknown_a", "unknown_b"})); | ||
| } | ||
|
|
||
| TEST(ClientTelemetryTest, ShowErrorsBoundsOversizedNonMessageFields) { |
Collaborator
There was a problem hiding this comment.
test/ut/TestClientTelemetry.cpp line:1309
Low ---- The show_errors truncation tests cover a single oversized field, but the selection loop that alternates across the four truncatable fields and the "still exceeds 1MB after all fields are emptied" throw path are only reachable when one retained error carries multiple oversized fields (e.g. both a huge collection and a huge error_msg). A test with two oversized fields would exercise the next-longest-field selection and the monotonic-progress fallback.
Collaborator
|
this pr is merged into #601 |
yhmo
added a commit
to yhmo/milvus-sdk-cpp
that referenced
this pull request
Sep 9, 2026
- Treat ttl_seconds as an ignored push_config field (milvus-io#587) - Harden client telemetry control flow: make very large heartbeat intervals safe and cancellable across platforms, fence command batches across telemetry transport rebinding, canonicalize ACK IDs, wake the unsupported backoff on channel replacement, and bound show_errors truncation including UTF-8 payloads (milvus-io#588) - Wake the chunked heartbeat wait on a corrective interval decrease so a server push takes effect immediately instead of after the current chunk; an increase still applies at the next natural wake - Swap the accumulated collectors out of the heartbeat lock in O(1) and build the snapshot outside the lock, so the per-bucket sort and serialization no longer block RecordOperation with many collections; note the request serialization in SendHeartbeat as a follow-up Signed-off-by: yhmo <yihua.mo@zilliz.com>
yhmo
added a commit
to yhmo/milvus-sdk-cpp
that referenced
this pull request
Sep 9, 2026
- Treat ttl_seconds as an ignored push_config field (milvus-io#587) - Harden client telemetry control flow: make very large heartbeat intervals safe and cancellable across platforms, fence command batches across telemetry transport rebinding, canonicalize ACK IDs, wake the unsupported backoff on channel replacement, and bound show_errors truncation including UTF-8 payloads (milvus-io#588) - Wake the chunked heartbeat wait on a corrective interval decrease so a server push takes effect immediately instead of after the current chunk; an increase still applies at the next natural wake - Swap the accumulated collectors out of the heartbeat lock in O(1) and build the snapshot outside the lock, so the per-bucket sort and serialization no longer block RecordOperation with many collections; note the request serialization in SendHeartbeat as a follow-up - Re-queue the original reply for fenced redeliveries instead of minting a success ACK, so an already-reported failure is not contradicted; clamp the pushed heartbeat interval as uint64 and cap the reported latency-history window start at the retained range Signed-off-by: yhmo <yihua.mo@zilliz.com>
yhmo
added a commit
to yhmo/milvus-sdk-cpp
that referenced
this pull request
Sep 9, 2026
- Treat ttl_seconds as an ignored push_config field (milvus-io#587) - Harden client telemetry control flow: make very large heartbeat intervals safe and cancellable across platforms, fence command batches across telemetry transport rebinding, canonicalize ACK IDs, wake the unsupported backoff on channel replacement, and bound show_errors truncation including UTF-8 payloads (milvus-io#588) - Wake the chunked heartbeat wait on a corrective interval decrease so a server push takes effect immediately instead of after the current chunk; an increase still applies at the next natural wake - Swap the accumulated collectors out of the heartbeat lock in O(1) and build the snapshot outside the lock, so the per-bucket sort and serialization no longer block RecordOperation with many collections; note the request serialization in SendHeartbeat as a follow-up - Re-queue the original reply for fenced redeliveries instead of minting a success ACK, so an already-reported failure is not contradicted; clamp the pushed heartbeat interval as uint64 and cap the reported latency-history window start at the retained range - Cap the unsupported-server re-probe cadence at the 30-minute backoff ceiling even when the configured heartbeat interval is larger, so an UNIMPLEMENTED server is re-probed within the cap instead of waiting out the full interval Signed-off-by: yhmo <yihua.mo@zilliz.com>
yhmo
added a commit
to yhmo/milvus-sdk-cpp
that referenced
this pull request
Sep 10, 2026
- Treat ttl_seconds as an ignored push_config field (milvus-io#587) - Harden client telemetry control flow: make very large heartbeat intervals safe and cancellable across platforms, fence command batches across telemetry transport rebinding, canonicalize ACK IDs, wake the unsupported backoff on channel replacement, and bound show_errors truncation including UTF-8 payloads (milvus-io#588) - Wake the chunked heartbeat wait on a corrective interval decrease so a server push takes effect immediately instead of after the current chunk; an increase still applies at the next natural wake - Swap the accumulated collectors out of the heartbeat lock in O(1) and build the snapshot outside the lock, so the per-bucket sort and serialization no longer block RecordOperation with many collections; note the request serialization in SendHeartbeat as a follow-up - Re-queue the original reply for fenced redeliveries instead of minting a success ACK, so an already-reported failure is not contradicted; clamp the pushed heartbeat interval as uint64 and cap the reported latency-history window start at the retained range - Cap the UNIMPLEMENTED re-probe cadence at the backoff ceiling even for larger configured intervals, and reject negative pushed intervals instead of wrapping them into a maximum cadence Signed-off-by: yhmo <yihua.mo@zilliz.com>
yhmo
added a commit
to yhmo/milvus-sdk-cpp
that referenced
this pull request
Sep 10, 2026
- Treat ttl_seconds as an ignored push_config field (milvus-io#587) - Harden client telemetry control flow: make very large heartbeat intervals safe and cancellable across platforms, fence command batches across telemetry transport rebinding, canonicalize ACK IDs, wake the unsupported backoff on channel replacement, and bound show_errors truncation including UTF-8 payloads (milvus-io#588) - Wake the chunked heartbeat wait on a corrective interval decrease so a server push takes effect immediately instead of after the current chunk; an increase still applies at the next natural wake - Swap the accumulated collectors out of the heartbeat lock in O(1) and build the snapshot outside the lock, so the per-bucket sort and serialization no longer block RecordOperation with many collections; note the request serialization in SendHeartbeat as a follow-up - Re-queue the original reply for fenced redeliveries instead of minting a success ACK, so an already-reported failure is not contradicted; clamp the pushed heartbeat interval as uint64 and cap the reported latency-history window start at the retained range - Keep the UNIMPLEMENTED backoff from probing more often than the configured interval (floor preserved), reject negative pushed intervals instead of wrapping them, and use underscore-suffixed Impl member names consistent with the rest of the repository Signed-off-by: yhmo <yihua.mo@zilliz.com>
yhmo
added a commit
to yhmo/milvus-sdk-cpp
that referenced
this pull request
Sep 10, 2026
- Treat ttl_seconds as an ignored push_config field (milvus-io#587) - Harden client telemetry control flow: make very large heartbeat intervals safe and cancellable across platforms, fence command batches across telemetry transport rebinding, canonicalize ACK IDs, wake the unsupported backoff on channel replacement, and bound show_errors truncation including UTF-8 payloads (milvus-io#588) - Wake the chunked heartbeat wait on a corrective interval decrease so a server push takes effect immediately instead of after the current chunk; an increase still applies at the next natural wake - Swap the accumulated collectors out of the heartbeat lock in O(1) and build the snapshot outside the lock, so the per-bucket sort and serialization no longer block RecordOperation with many collections; note the request serialization in SendHeartbeat as a follow-up - Re-queue the original reply for fenced redeliveries instead of minting a success ACK, so an already-reported failure is not contradicted; clamp the pushed heartbeat interval as uint64 and cap the reported latency-history window start at the retained range - Keep the UNIMPLEMENTED backoff from probing more often than the configured interval (floor preserved), reject negative pushed intervals instead of wrapping them, and use underscore-suffixed Impl member names consistent with the rest of the repository Signed-off-by: yhmo <yihua.mo@zilliz.com>
yhmo
added a commit
to yhmo/milvus-sdk-cpp
that referenced
this pull request
Sep 10, 2026
- Treat ttl_seconds as an ignored push_config field (milvus-io#587) - Harden client telemetry control flow: make very large heartbeat intervals safe and cancellable across platforms, fence command batches across telemetry transport rebinding, canonicalize ACK IDs, wake the unsupported backoff on channel replacement, and bound show_errors truncation including UTF-8 payloads (milvus-io#588) - Wake the chunked heartbeat wait on a corrective interval decrease so a server push takes effect immediately instead of after the current chunk; an increase still applies at the next natural wake - Swap the accumulated collectors out of the heartbeat lock in O(1) and build the snapshot outside the lock, so the per-bucket sort and serialization no longer block RecordOperation with many collections; note the request serialization in SendHeartbeat as a follow-up - Re-queue the original reply for fenced redeliveries instead of minting a success ACK, so an already-reported failure is not contradicted; clamp the pushed heartbeat interval as uint64 and cap the reported latency-history window start at the retained range - Keep the UNIMPLEMENTED backoff from probing more often than the configured interval (floor preserved), reject negative pushed intervals instead of wrapping them, and use underscore-suffixed Impl member names consistent with the rest of the repository Signed-off-by: yhmo <yihua.mo@zilliz.com>
yhmo
added a commit
to yhmo/milvus-sdk-cpp
that referenced
this pull request
Sep 10, 2026
- Treat ttl_seconds as an ignored push_config field (milvus-io#587) - Harden client telemetry control flow: make very large heartbeat intervals safe and cancellable across platforms, fence command batches across telemetry transport rebinding, canonicalize ACK IDs, wake the unsupported backoff on channel replacement, and bound show_errors truncation including UTF-8 payloads (milvus-io#588) - Wake the chunked heartbeat wait on a corrective interval decrease so a server push takes effect immediately instead of after the current chunk; an increase still applies at the next natural wake - Swap the accumulated collectors out of the heartbeat lock in O(1) and build the snapshot outside the lock, so the per-bucket sort and serialization no longer block RecordOperation with many collections; note the request serialization in SendHeartbeat as a follow-up - Re-queue the original reply for fenced redeliveries instead of minting a success ACK, so an already-reported failure is not contradicted; clamp the pushed heartbeat interval as uint64 and cap the reported latency-history window start at the retained range - Keep the UNIMPLEMENTED backoff from probing more often than the configured interval (floor preserved), reject negative pushed intervals instead of wrapping them, and use underscore-suffixed Impl member names consistent with the rest of the repository Signed-off-by: yhmo <yihua.mo@zilliz.com>
sre-ci-robot
pushed a commit
that referenced
this pull request
Sep 10, 2026
- Treat ttl_seconds as an ignored push_config field (#587) - Harden client telemetry control flow: make very large heartbeat intervals safe and cancellable across platforms, fence command batches across telemetry transport rebinding, canonicalize ACK IDs, wake the unsupported backoff on channel replacement, and bound show_errors truncation including UTF-8 payloads (#588) - Wake the chunked heartbeat wait on a corrective interval decrease so a server push takes effect immediately instead of after the current chunk; an increase still applies at the next natural wake - Swap the accumulated collectors out of the heartbeat lock in O(1) and build the snapshot outside the lock, so the per-bucket sort and serialization no longer block RecordOperation with many collections; note the request serialization in SendHeartbeat as a follow-up - Re-queue the original reply for fenced redeliveries instead of minting a success ACK, so an already-reported failure is not contradicted; clamp the pushed heartbeat interval as uint64 and cap the reported latency-history window start at the retained range - Keep the UNIMPLEMENTED backoff from probing more often than the configured interval (floor preserved), reject negative pushed intervals instead of wrapping them, and use underscore-suffixed Impl member names consistent with the rest of the repository Signed-off-by: yhmo <yihua.mo@zilliz.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary\n\n- make very large heartbeat intervals safe and cancellable across platforms\n- fence command batches across telemetry transport rebinding and canonicalize ACK IDs\n- wake unsupported backoff on channel replacement and bound show_errors truncation, including UTF-8 payloads\n\nFollow-up hardening for #585.\n\n## Validation\n\n- 854/854 unit tests passed\n- 291/291 mocked integration tests passed\n- final telemetry-focused suite: 33/33 passed\n- clang-format and git diff --check passed\n