Skip to content

Fix experiment-dataset linking when running evals with a dataset#103

Merged
delner merged 1 commit intobraintrustdata:mainfrom
alfakini:fix/experiment-dataset-linking
Feb 17, 2026
Merged

Fix experiment-dataset linking when running evals with a dataset#103
delner merged 1 commit intobraintrustdata:mainfrom
alfakini:fix/experiment-dataset-linking

Conversation

@alfakini
Copy link
Contributor

@alfakini alfakini commented Feb 14, 2026

The issue

When running evals with a dataset via Braintrust::Eval.run, the resulting experiment is not linked to the dataset in the Braintrust UI. The UI shows "Rows not attached to a dataset" because dataset_id and dataset_version are never included in the experiment creation request:

Screenshot 2026-02-14 at 4 40 37 PM

The issue is that Eval.resolve_dataset resolved a dataset to an array of cases but discarded the Dataset object, so dataset_obj.id was never captured. Experiments#create did not accept or send dataset_id and dataset_version in the POST /v1/experiment payload, even though the API supports both fields.

Additionally, Dataset#version returns nil when the dataset is not explicitly pinned to a version. The Python SDK handles this by computing the version as max(_xact_id) across all records in the fetched dataset, but the Ruby SDK does not.

Fix

Eval.resolve_dataset now returns a hash with :cases, :dataset_id, and :dataset_version instead of a plain array. When no pinned version is available, it computes dataset_version from max(_xact_id) across fetched records (matching the Python SDK behavior).

Eval.run retrieves dataset_id and dataset_version from the resolved dataset and forwards them to the experiment-creation process. Experiments#create accepts optional dataset_id and dataset_version keyword arguments and includes them in the API payload when present.

After the fix got applied:

Screenshot 2026-02-14 at 4 44 46 PM

Tests

Added assertion to the existing dataset eval test verifying that dataset_id and dataset_version are sent in the POST /v1/experiment request body.

Added test_eval_run_without_dataset_does_not_send_dataset_fields to verify that dataset_id and dataset_version are nil when no dataset is provided.

Copy link
Collaborator

@delner delner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from what I can see! The version lookup is a bit quirky but I think we'll need to rework how datasets are constructed soon anyways (outside the scope of this PR.)

I'll enable the workflow, and if everything is good in CI, we'll merge.

@delner
Copy link
Collaborator

delner commented Feb 17, 2026

I think tests are failing because this PR is from a fork and secrets are not available. I've made #105 to help unblock this. @alfakini you'll want to rebase on this (or on main once it's merged.)

@delner delner force-pushed the fix/experiment-dataset-linking branch from de43aec to 4254b82 Compare February 17, 2026 04:41
@delner delner merged commit c57b14b into braintrustdata:main Feb 17, 2026
7 checks passed
@delner
Copy link
Collaborator

delner commented Feb 18, 2026

This was released in v0.1.4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants