Add llm cost_tags field to share tests by XG-xin · Pull Request #6829 · DataDog/system-tests

XG-xin · 2026-04-29T19:47:13Z

Motivation

Add system tests for parameter cost_tags.

dd-trace-py: PR is merged, likely included in v4.9
dd-trace-js: PR is open, need to merge this first to pass the tests.

Changes

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-04-29T19:47:44Z

CODEOWNERS have been resolved as:

manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
tests/parametric/test_llm_observability/test_llm_observability.py       @DataDog/ml-observability @DataDog/system-tests-core
utils/_features.py                                                      @DataDog/system-tests-core
utils/build/docker/java/parametric/src/main/java/com/datadoghq/trace/controller/LlmObsController.java  @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/nodejs/parametric/llmobs.js                          @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/python/parametric/apm_test_client/llmobs.py          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/docker_fixtures/spec/llm_observability.py                         @DataDog/ml-observability @DataDog/system-tests-core

datadog-prod-us1-3 · 2026-04-29T20:03:20Z

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: b0d3ab8 | Docs | Datadog PR Page | Give us feedback!}

Dev-mode check rejects manifest versions ahead of the current tested tracer (4.9.0-rc1). The cost_tags PR is merged but not yet in any released version. Use missing_feature for now; bump to a concrete version once released. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b2177afdf9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

sabrenner

looks great, thanks for adding these tests! just a couple actionable things and one note for when (if?) we add Java cost tracking tags too

sabrenner · 2026-04-30T20:32:13Z

+    return span_event.get("meta", {}).get("metadata", {}).get("_dd", {}).get("cost_tags")
+
+
+@features.llm_observability_sdk_enablement


small request to make a different feature for this! let me know if you need help with that, i think i made a confluence page on it but i can follow up with you on slack!

sabrenner · 2026-04-30T20:35:31Z

+        llmobs_request = LlmObsAnnotationContextRequest(
+            tags={"team": "ml", "feature": "chatbot"},
+            cost_tags=["team", "feature"],
+            children=[LlmObsSpanRequest(kind="llm")],


i would say actually maybe having two children here and making sure each one gets the tags to make sure they are getting properly added would be a slightly better test, so like

children=[LlmObsSpanRequest(kind="llm"), LlmObsSpanRequest(kind="llm")], # ... assert _get_cost_tags(span_events[0]) == ["team", "feature"] assert _get_cost_tags(span_events[1]) == ["team", "feature"]

just to be extra sure!

XG-xin · 2026-05-01T14:28:54Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 129254e0e3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-05-01T14:33:46Z

  tests/parametric/test_library_tracestats.py::Test_Library_Tracestats::test_sample_rate_0_TS007: missing_feature (failure 2.8.0)  # Created by easy win activation script
  tests/parametric/test_library_tracestats.py::Test_Library_Tracestats::test_successes_errors_recorded_separately_TS006: missing_feature (failure 2.8.0)  # Created by easy win activation script
  tests/parametric/test_library_tracestats.py::Test_Library_Tracestats::test_top_level_TS005: missing_feature (failure 2.8.0)  # Created by easy win activation script
+  tests/parametric/test_llm_observability/test_llm_observability.py::Test_CostTags: missing_feature (cost_tags merged in dd-trace-py#17628, not yet in a released version — bump to the version once released)


Quote manifest values containing #

.cursor/rules/pr-review.mdc requires manifest values with special YAML characters, including #, to be quoted. This new Python manifest declaration includes dd-trace-py#17628 as an unquoted scalar, so it violates the repository manifest YAML syntax rule; wrap the whole value in quotes to keep the manifest compliant and avoid tooling/parsing surprises if the reason text is edited.

Useful? React with 👍 / 👎.

sabrenner

two non-blocking comments, feel free to address! otherwise good to go, just looks like the linter is failing & blocking tests from running

sabrenner · 2026-05-01T14:56:31Z

        }
+        # cost_tags was added later — only forward when set so older SDK versions
+        # without the kwarg don't break unrelated tests in the same weblog.
+        if options.get("cost_tags") is None:


i think if we gate the tests that run cost tracking behind the version that it's released in, we don't need this line here, but not blocking

Co-authored-by: Sam Brenner <106700075+sabrenner@users.noreply.github.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The prod tracer image runs python@4.8.0 (pre-#17628), which raises TypeError on LLMObs.annotate(cost_tags=...). Restoring the defensive pop so unrelated annotate-using tests (Test_Enablement, Test_Prompts) keep passing on prod. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

add llm cost_tags field to share tests

14d98d2

XG-xin added the build-python-base-images label Apr 30, 2026

XG-xin marked this pull request as ready for review April 30, 2026 20:10

XG-xin requested review from a team as code owners April 30, 2026 20:10

XG-xin requested review from Kyle-Verhoog, avara1986, jandro996 and manuel-alvarez-alvarez and removed request for a team April 30, 2026 20:10

chatgpt-codex-connector Bot reviewed Apr 30, 2026

View reviewed changes

Comment thread utils/docker_fixtures/spec/llm_observability.py

update comment for known limitation

d4188be

sabrenner reviewed Apr 30, 2026

View reviewed changes

add cost tags in js, and a todo for java

129254e

chatgpt-codex-connector Bot reviewed May 1, 2026

View reviewed changes

XG-xin added the build-nodejs-base-images label May 1, 2026

sabrenner approved these changes May 1, 2026

View reviewed changes

XG-xin and others added 2 commits May 1, 2026 11:23

Apply suggestion from @sabrenner

cea8d79

Co-authored-by: Sam Brenner <106700075+sabrenner@users.noreply.github.com>

remove cost_tags is None check

404e664

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting and removed mergequeue-status: waiting labels May 1, 2026

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: removed label May 1, 2026

style: add blank line before features = _Features() to satisfy ruff

404a28b

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting mergequeue-status: removed and removed mergequeue-status: removed mergequeue-status: waiting labels May 1, 2026

cbeauchesne approved these changes May 4, 2026

View reviewed changes

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: queued mergequeue-status: in_progress mergequeue-status: retrying mergequeue-status: removed and removed mergequeue-status: removed mergequeue-status: queued mergequeue-status: in_progress mergequeue-status: retrying labels May 4, 2026

XG-xin merged commit 29534f6 into main May 4, 2026
920 of 926 checks passed

XG-xin deleted the xinyuan/llm-cost-tags branch May 4, 2026 18:38

		return span_event.get("meta", {}).get("metadata", {}).get("_dd", {}).get("cost_tags")


		@features.llm_observability_sdk_enablement

Conversation

XG-xin commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-prod-us1-3 Bot commented Apr 29, 2026 • edited by datadog-prod-us1-6 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

sabrenner left a comment

Choose a reason for hiding this comment

Uh oh!

sabrenner Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sabrenner Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

XG-xin commented May 1, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 1, 2026

Choose a reason for hiding this comment

Uh oh!

sabrenner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sabrenner May 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

XG-xin commented Apr 29, 2026 •

edited

Loading

github-actions Bot commented Apr 29, 2026 •

edited

Loading

datadog-prod-us1-3 Bot commented Apr 29, 2026 •

edited by datadog-prod-us1-6 Bot

Loading