fix(tracing): preserve dd.internal.entity_id tag on runtime metrics flush - #19602
Conversation
…lush RuntimeWorker.flush() reassigned the dogstatsd client's constant_tags on every flush, wholesale replacing whatever the client derived at construction (e.g. dd.internal.entity_id from DD_ENTITY_ID). This permanently dropped Kubernetes origin detection tags for runtime.python.* metrics starting from the very first flush. Snapshot the client's original constant_tags at construction and merge them back in on every flush, deduping exact-string repeats for customers who applied the documented DD_TAGS workaround. Fixes GH-19526.
Codeowners resolved asResolved from the full PR diff against No remaining files require a CODEOWNERS review. |
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 76c3180 | Docs | View more details | Give us feedback! |
Circular import analysis
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0880de8863
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
BenchmarksBenchmark execution time: 2026-08-18 11:07:52 Comparing candidate commit 76c3180 in PR branch Found 0 performance improvements and 4 performance regressions! Performance is the same for 383 metrics, 9 unstable metrics.
|
… metrics RuntimeWorker snapshotted the dogstatsd client's entire constant_tags at construction to preserve dd.internal.entity_id across flushes. But that client also derives service/env/version tags from DD_SERVICE/DD_ENV/ DD_VERSION at construction time, and those are already recomputed fresh from ddtrace.config on every flush via TracerTags(). Re-merging the whole snapshot meant a stale env-derived value (e.g. service:env-service) could be sent alongside the current one (e.g. service:override-service) whenever config.service/env/version changed after the worker started, since the dedup only caught exact-string repeats, not same-key conflicts. Narrow the snapshot to only dd.internal.entity_id, the one tag with no other fresh source in flush(). Co-Authored-By: Codex <noreply@openai.com> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dependency direction analysis
|
|
/merge |
|
View all feedbacks in Devflow UI.
It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.
devflow unqueued this merge request: It did not become mergeable within the expected time |
There was a problem hiding this comment.
Pull request overview
Fixes runtime-metrics tag handling so the DogStatsD dd.internal.entity_id (and related constant tags) are not lost when RuntimeWorker.flush() refreshes tags, restoring Kubernetes origin-detection attribution for runtime.python.* metrics.
Changes:
- Preserve client-derived constant tags across
RuntimeWorker.flush()calls while keeping service/env/version up-to-date. - Add regression tests asserting entity-id tag preservation and deduplication behavior in emitted packets.
- Add a release note documenting the fix for Kubernetes origin-detection tagging.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
ddtrace/internal/runtime/runtime_metrics.py |
Adjusts runtime-metrics tag computation to preserve important DogStatsD constant tags across flushes. |
tests/tracer/runtime/test_runtime_metrics.py |
Adds regression tests validating entity-id tag preservation/deduplication and tag freshness behavior. |
releasenotes/notes/fix-runtime-metrics-entity-id-tag-f401edcebf35c495.yaml |
Documents the runtime-metrics entity-id tagging fix in release notes. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Address review feedback from @mabdinur to trim the explanatory comment in RuntimeWorker.__init__ down to the essential rationale. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
Build pipeline has failing jobs for 1341082: What to do next?
DetailsSince those jobs are not marked as being allowed to fail, the pipeline will most likely fail. |
test_runtime_worker_flush_does_not_leak_stale_service_tag hard-set ddtrace.config.service back to None in its finally block instead of restoring the value it actually had before the test ran. Since config.service is a plain attribute computed once at process start (not a lazily-recomputed property), this permanently pinned it to None for the rest of the pytest session, breaking test_start_span_service_default (and any other test relying on the default-inferred service name) whenever it ran afterward in the same process -- which is what broke dd-gitlab/core/tracer 8/19 on this PR. Use the repo's override_global_config() helper instead, which snapshots and restores the real prior value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
/merge |
|
View all feedbacks in Devflow UI.
It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.
The expected merge time in
|
…lush (#19602) ## Description `RuntimeWorker.flush()` reassigned the dogstatsd client's `constant_tags` wholesale on every flush, using only `service`/`env`/`version`/platform/process tags. This discarded whatever the client derived at construction — most notably `dd.internal.entity_id` (from `DD_ENTITY_ID`), which the Datadog Agent uses for Kubernetes origin detection (`pod_name`/`kube_namespace` attribution). The tag was gone starting from the very first flush, for the life of the process. The fix snapshots the client's original `constant_tags` at construction (`RuntimeWorker.__init__`) and merges that snapshot back into the tag list on every `flush()`, deduping exact-string repeats — this matters for customers who applied the documented `DD_TAGS=dd.internal.entity_id:...` workaround, since that string can now arrive from two sources. Fixes #19526. ## Testing Added two regression tests to `tests/tracer/runtime/test_runtime_metrics.py`: - `test_runtime_worker_flush_preserves_entity_id_tag`: asserts `dd.internal.entity_id` appears exactly once on every metric line actually sent (via a mocked socket), across multiple `flush()` calls. - `test_runtime_worker_flush_dedupes_entity_id_tag`: asserts no duplicate tag when both `DD_ENTITY_ID` and the `DD_TAGS` workaround are set. Both tests were verified to fail against the pre-fix code (missing tag after the first flush) and pass against the fix. Full `tracer` suite run locally (all three venv variants: `tracer`, `tracer-python-optimize`, `tracer-uwsgi`) with no regressions. ## Risks Low. The change only affects tag computation in `RuntimeWorker.flush()`; no public API changes. Verified no fork-safety impact (the worker instance persists across fork; the snapshot survives via normal copy-on-fork semantics) and confirmed via grep that this is the only place in the non-vendored codebase that mutates `constant_tags`. ## Additional Notes None. Co-authored-by: venus.parfait <venus.parfait@datadoghq.com>
Description
RuntimeWorker.flush()reassigned the dogstatsd client'sconstant_tagswholesale on every flush, using onlyservice/env/version/platform/process tags. This discarded whatever the client derived at construction — most notablydd.internal.entity_id(fromDD_ENTITY_ID), which the Datadog Agent uses for Kubernetes origin detection (pod_name/kube_namespaceattribution). The tag was gone starting from the very first flush, for the life of the process.The fix snapshots the client's original
constant_tagsat construction (RuntimeWorker.__init__) and merges that snapshot back into the tag list on everyflush(), deduping exact-string repeats — this matters for customers who applied the documentedDD_TAGS=dd.internal.entity_id:...workaround, since that string can now arrive from two sources.Fixes #19526.
Testing
Added two regression tests to
tests/tracer/runtime/test_runtime_metrics.py:test_runtime_worker_flush_preserves_entity_id_tag: assertsdd.internal.entity_idappears exactly once on every metric line actually sent (via a mocked socket), across multipleflush()calls.test_runtime_worker_flush_dedupes_entity_id_tag: asserts no duplicate tag when bothDD_ENTITY_IDand theDD_TAGSworkaround are set.Both tests were verified to fail against the pre-fix code (missing tag after the first flush) and pass against the fix. Full
tracersuite run locally (all three venv variants:tracer,tracer-python-optimize,tracer-uwsgi) with no regressions.Risks
Low. The change only affects tag computation in
RuntimeWorker.flush(); no public API changes. Verified no fork-safety impact (the worker instance persists across fork; the snapshot survives via normal copy-on-fork semantics) and confirmed via grep that this is the only place in the non-vendored codebase that mutatesconstant_tags.Additional Notes
None.