Bug: using `dataset` on `Braintrust::Eval.run` raises `Braintrust::Error: HTTP 303` error

[alfakini]

Using dataset on Braintrust::Eval.run raises Braintrust::Error: HTTP 303 error.

Gem version: tested on 0.1.2 and 0.1.1

Is it because I'm passing the state to Braintrust::Eval.run? I have multiple projects running in the same codebase, that's why I don't rely on the global state.

Code snippet that caused the issue:

Braintrust::Eval.run(
  project: project_name,
  experiment: experiment_name,
  dataset: { name: dataset_name, project: project_name }
  state: braintrust_state,
  scorers: [Braintrust::Eval::Functions.scorer(project: project_name, slug: slug, state: braintrust_state)
  task: ->(input) { ... },
  parallelism: 10
)

Even using the direct API access resulted in the same error:

api = Braintrust::API.new(state: state)
metadata = api.datasets.get(project_name: project_name, name: dataset_name)

Full error message:

Braintrust::Error:
       HTTP 303 for POST https://api.braintrust.dev/btql: See Other. Redirecting to https://bt3subnets-3-lambdaresponsesbucket-[REDACTED].s3.us-east-1.amazonaws.com/responses/2026-02-12/[REDACTED].jsonl.gz?[REDACTED]&x-id=GetObject

[alfakini]

The monkey-patch I'm using. I will not submit a PR because I'm not sure whether this is an issue with my setup or a general issue. Please let me know what you think.

module Braintrust
  class API
    class Datasets
      private
        MAX_REDIRECTS = 5

        alias_method :original_http_request, :http_request

        def http_request(method, path, params: {}, payload: nil, base_url: nil, parse_json: true)
          # Build URI
          base = base_url || @state.api_url
          uri = URI("#{base}#{path}")
          uri.query = URI.encode_www_form(params) unless params.empty?

          # Create request
          request = case method
                    when :get
                      Net::HTTP::Get.new(uri)
                    when :post
                      req = Net::HTTP::Post.new(uri)
                      req["Content-Type"] = "application/json"
                      req.body = JSON.dump(payload) if payload
                      req
                    else
                      raise ArgumentError, "Unsupported HTTP method: #{method}"
          end

          request["Authorization"] = "Bearer #{@state.api_key}"

          # Execute request with timing
          start_time = Time.now
          Log.debug("[API] #{method.upcase} #{uri}")

          http = Net::HTTP.new(uri.host, uri.port)
          http.use_ssl = (uri.scheme == "https")
          response = http.request(request)

          #########################################################
          # NOTE: Monkey patch to support follow redirects
          # (e.g. 303 See Other from /btql -> S3)
          redirect_count = 0
          previous_uri = uri
          while response.is_a?(Net::HTTPRedirection) && redirect_count < MAX_REDIRECTS
            redirect_count += 1
            redirect_uri = URI.join(previous_uri, response["location"])
            Log.debug("[API] Following redirect ##{redirect_count}: #{redirect_uri}")

            # 303 requires switching to GET per HTTP spec
            redirect_request = Net::HTTP::Get.new(redirect_uri)

            # Do NOT forward Authorization header to redirect target (e.g. S3 uses its own auth)

            redirect_http = Net::HTTP.new(redirect_uri.host, redirect_uri.port)
            redirect_http.use_ssl = (redirect_uri.scheme == "https")
            response = redirect_http.request(redirect_request)
            previous_uri = redirect_uri
          end

          if response.is_a?(Net::HTTPRedirection)
            raise Error, "Too many redirects (#{MAX_REDIRECTS}) for #{method.upcase} #{uri}"
          end
          #########################################################

          duration_ms = ((Time.now - start_time) * 1000).round(2)
          Log.debug("[API] #{method.upcase} #{uri} -> #{response.code} (#{duration_ms}ms, #{response.body.bytesize} bytes)")

          # Handle response
          unless response.is_a?(Net::HTTPSuccess)
            Log.debug("[API] Error response body: #{response.body}")
            raise Error, "HTTP #{response.code} for #{method.upcase} #{uri}: #{response.body}"
          end

          parse_json ? JSON.parse(response.body) : response
        end
    end
  end
end

[delner]

Hi @alfakini ! Thanks for reporting.

From my first pass, it looks like it's a large payload and we're not handling the 303 properly. So we'll need to do something like what's in your monkeypatch.

Unrelated to this issue I just released 0.1.3 which now has Eval take api instead of state but I don't think this will change things for your case (made this change before I saw this issue...) I want to examine your session management needs and consider whether there's a simpler, more reliable UX we can offer for this.

Can you tell me more about the use of projects? Eval.run already accepts project (which you configured), so I would imagine that should work. state stores data like where to send data to... its more akin to a session which shouldn't be necessary to change unless you're working across orgs?

[alfakini]

Thanks for looking into this @delner!

Re: use of projects and state/session management:

We're within a single org, but we need session isolation because different features have different API keys and project configs running concurrently in the same process. The core need is: pass an explicit API/session to Eval.run and its scorers so they share the same authenticated session, rather than relying on a global singleton that would conflict across features.

Why set_global: false: We run multiple Braintrust projects in the same Ruby process (a Rails app), each with its own API key and project name:

# Each feature gets its own isolated Braintrust session and tracer provider
tracer_provider = OpenTelemetry::SDK::Trace::TracerProvider.new

instance = Braintrust.init(
  api_key: api_key,
  default_project: project_name,
  set_global: false,
  auto_instrument: false,
  tracer_provider: tracer_provider
)

client = OpenAI::Client.new(access_token: access_token)
Braintrust.instrument!(:ruby_openai, target: client, tracer_provider: tracer_provider)

tracer = tracer_provider.tracer(project_name)
tracer.in_span("Custom task span") do |span|
  span.set_attribute("custom attribute", 123)

  client.chat("Hello")
end

This ensures the full span tree for feature 1 (including custom task/tool spans and auto-instrumented OpenAI calls) routes exclusively to the feature 1 Braintrust project, fully isolated from feature 2 spans, which route to their own project.

Why do we pass state, api, and tracer_provider explicitly:

Without passing state to Functions.scorer and api to Eval.run, we get Braintrust::Error: No state available. And without passing tracer_provider, no rows are reported in the Experiment UI.

Why we manage OpenTelemetry context manually:

We also had to set up a custom OpenTelemetry configuration to avoid mixing regular application OTel spans with AI spans. We manage OpenTelemetry::Context ourselves between transactions to ensure sub-spans attach correctly under the root span rather than creating new, disconnected ones.

I don't know if I made myself clear with these snippets. I'm open to connecting for a meeting to screen share our setup with you if that's helpful.

Thank you!

[delner]

Opened #106 which should resolve the 303 bug. As soon as it's merged I'll release this and close this issue. I think it's still worth following up on the other things discussed here (UX for using evals for multiple projects), but perhaps we should do so in another issue/thread.

[delner]

This is now released in v0.1.4! Check it out and let us know if it needs anything else.