Fix: Pass metadata from dataset to prompts in Python SDK init_function

[Alex Jean-Baptiste (justcodebruh)]

Summary

Fixes a bug where metadata parameters from dataset records were not being passed through to prompt templates when running experiments via the Python SDK, even though they worked correctly in the UI.

Problem

When using init_function() to create a task for Eval(), dataset records containing metadata fields were not being propagated to the prompt templates. This caused mustache template variables like {{metadata.fieldName}} to render as empty strings instead of their actual values.

Example of the issue:

# Dataset record with metadata
{
  "input": {"name": "John", "topic": "AI"},
  "metadata": {"test0": 99, "test1": 42},
  "expected": "Expected output"
}

# Prompt template
"Tell me about {{metadata.test0}}. And end with a salutation to {{metadata.test1}}"

# Result BEFORE fix: "Tell me about . And end with a salutation to "
# Result AFTER fix: "Tell me about 99. And end with a salutation to 42"

Root Cause

The init_function() wrapper was not extracting metadata from the hooks parameter that the evaluation framework provides when calling task functions. The framework passes a DictEvalHooks object containing the metadata, but init_function() was only forwarding the input data to invoke().

Solution

Modified init_function() to:

1. Accept exactly 2 parameters (input, hooks) to match framework requirements for passing hooks
2. Extract metadata from hooks.metadata when available
3. Pass the metadata to the invoke() call for proper template substitution
4. Maintain backward compatibility for both task and scorer modes

Changes

• Modified function signature from f(*args, **kwargs) to f(input, hooks=None)
• Added logic to detect and handle the hooks object vs other parameter types
• Extracts metadata and passes it to invoke() for both task and scorer modes

Test Plan

1. Created test script that reproduces the issue:

# Test with a prompt that uses metadata variables
prompt_ref = init_function("project", "prompt-with-metadata")
dataset_ref = init_dataset(project="project", name="dataset-with-metadata")

# Run eval
await braintrust.EvalAsync(
    name="test",
    task=prompt_ref,
    data=dataset_ref,
    scores=[scorer_ref]
)

2. Verified the fix with direct invocation:

class FakeHooks:
    def __init__(self):
        self.metadata = {"test0": 99, "test1": 42}

hooks = FakeHooks()
result = prompt_ref({"name": "User", "topic": "Topic"}, hooks)
# Result now correctly shows "99" and "42" in output

3. Confirmed backward compatibility:

• Tasks without hooks still work
• Scorer functions still receive metadata via input dict
• Existing code without metadata continues to function

Impact

This fix ensures that SDK users can properly use metadata parameters in their prompt templates, achieving feature parity with the UI. This is especially important for users who rely on metadata for dynamic prompt generation or context injection.

🤖 Generated with Claude Code

[Matt Perpick (clutchski)]

i'm not sure if input / hooks are the only possible args, but if other things were passed in e.g.f(foo=123) this will be backwards incompatible.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

[github-actions]

This pull request was closed because it has been inactive for 21 days (14 days of inactivity before being marked stale, plus 7 additional days). If this PR is still relevant, please feel free to reopen it. Thank you!