investigate OTel context-leak under parallel subworkflows

[cchinchilla-dev]

Description

ValueError: Token was created in a different Context surfaces when a workflow with parallel subworkflows runs under an active OTel observer. Observed once during the 0.4.0 audit while running examples/10_content_localization.yaml (which fans out into three parallel subworkflows: localize_es, localize_de, localize_ja). The workflow finishes (with [FAIL] because the upstream fetch_content tool failed for unrelated reasons), but the contextvars error leaks through the engine's task group and lands on stderr.

The classic culprit for this error in async-Python is a contextvars.Token being reset from a different task than the one that set it — typically when ContextVar.set() is called inside one async def and ContextVar.reset(token) is called inside a sibling or descendant task. OTel uses current_context as a ContextVar and may attach/detach it at span boundaries. AgentLoom's observability/tracing.py and metrics.py both manipulate OTel context around step spans.

What makes this hard: the observation is from one audit run that ALSO had other things failing. We don't have a deterministic reproducer that isolates the contextvars leak from the unrelated upstream failure. Filing as an investigation issue so the next step is "build a reproducer", not "ship a fix".

Repro (non-deterministic, audit observation)

$ set -a; source .env.audit; set +a
$ agentloom run examples/10_content_localization.yaml
…
ValueError: <Token var=<ContextVar name='current_context' default={} at 0x10b9210d0> at 0x10bfd1280> was created in a different Context
…
Status: failed

The fetch_content step also fails in this run (unrelated). Need to confirm whether the ValueError happens with fetch_content succeeding.

Expected vs actual

• Expected: no contextvars Token leak across task boundaries; OTel span lifecycle is correct under parallel subworkflows.
• Actual: ValueError: Token was created in a different Context surfaces from inside the engine's anyio.create_task_group().

Investigation plan

1. Build a minimal reproducer. Workflow with three parallel subworkflow steps + a noop observer that exercises the OTel context lifecycle. No external dependencies, no network calls. Goal: trigger the ValueError deterministically with pytest.
2. Audit OTel Token lifecycle. Walk src/agentloom/observability/tracing.py and metrics.py. Identify every attach() / detach() pair. Confirm each pair runs in the same task (no awaits between attach and detach).
3. Inspect the engine boundary. In core/engine.py, when a subworkflow step launches a child engine inside anyio.create_task_group(), does the child engine inherit or copy the parent's context? contextvars.copy_context() semantics matter here.
4. Fix surface (once reproducer is solid): typically contextvars.copy_context().run(...) per child task, or moving Token.set/reset inside the same async def. Sometimes the fix is to use otel.trace.use_span(span, end_on_exit=True) instead of manual attach/detach.

Scope (preliminary)

• tests/observability/test_tracing.py (new test, parallel subworkflows + OTel observer)
• src/agentloom/observability/tracing.py
• src/agentloom/observability/metrics.py
• src/agentloom/steps/subworkflow.py (context propagation across the boundary)

Notes

• Surfaced in #146 deep-pass section as F10. Severity at the time: HIGH (real concurrency bug), downgraded here to MEDIUM until the reproducer confirms it triggers without unrelated upstream failures.
• Likely interacts with the subworkflow fixes scoped in 0.5.0's fix(core) PR (issue 057 in the audit-followup plan). If the cause turns out to be subworkflow context propagation rather than OTel lifecycle, fold the fix into that PR.
• Not a blocker for 0.5.0 — observed only with parallel subworkflows under OTel, an uncommon combination. Default workflows are unaffected.