Repository navigation
[Go] Support context.Context in register.CombinerN methods - #40991
jonathansudhakar1 wants to merge 1 commit into
Conversation
register.Combiner1/2/3 panicked with "Failed to optimize ... Failed to infer types" for any CombineFn whose CreateAccumulator, AddInput, MergeAccumulators or ExtractOutput takes a leading context.Context, even though CombineFn validation accepts that parameter (it is how combiners use metrics and logging). Add ctx-first variants of the combiner extraction interfaces to register.tmpl, plus a branch for each in CombinerN that registers the caller for the func(context.Context, ...) type and builds the method wrapper, the same way the existing branches do. Regenerate register.go. Combiners without a context behave as before. Fixes apache#23550 Generated-by: Claude Code
There was a problem hiding this comment.
📋 Review Summary
This PR adds support for optional leading context.Context parameters on CombineFn lifecycle methods (CreateAccumulator, AddInput, MergeAccumulators, and ExtractOutput) in register.Combiner1, Combiner2, and Combiner3, fixing #23550. The template changes in register.tmpl, regenerated code in register.go, and unit tests in register_test.go are clean, well-structured, and thoroughly tested.
🔍 General Feedback
- The new extraction interfaces and type-switch branches in
register.tmplaccurately mirror the existing non-context patterns and align withgraph.AsCombineFnsignature validation. - Unit test coverage in
TestCombiner_ContextCombinersis comprehensive, verifying caller registration (avoiding reflection fallback), context propagation, error returns, andgraph.NewCombineFnintegration across allCombinerNvariants.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
Assigning reviewers: R: @lostluck for label go. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Fixes #23550.
register.Combiner1/2/3panicked withFailed to optimize MergeAccumulators for combiner &{}. Failed to infer types(and likewise for the other methods) when a CombineFn method takes a leadingcontext.Context. CombineFn validation ingraphalready accepts that parameter, and it's how combiners use metrics and logging. Onlyregisterrejected it.This follows the plan @lostluck agreed to on the issue (thank you!):
register.tmpl: adds ctx-first variants of the combiner extraction interfaces (createAccumulator1x1/1x2,addInput3x1/3x2,mergeAccumulators3x1/3x2,extractOutput2x1/2x2). In eachCombinerN, adds a branch per variant, next to its non-ctx counterpart. Each branch registers thecallerfor thefunc(context.Context, ...)type and builds the method wrapper, the same way the existing branches do.register.go: regenerated from the template. Only the combiner section changes.Testing
All commands run from
sdks/, Go 1.26.cmd/specializeand ran it the waydoc.go'sgo:generatelines do (specialize --package=register --input=register.tmpl --x=data,universals --imports=typex, then gofmt). As a check, I first regenerated from the unchanged template: the output was byte-for-byte identical to the checked-inregister.go. Then I regenerated from the edited template.TestCombiner_ContextCombinersinregister_test.go, table-driven, with 7 subtests:errorreturns, forCombiner1,Combiner2andCombiner3. Together these cover all eight new interfaces, for each type pairing[T0, T0/T1/T2]that has an error-return variant. A seventh combiner, with onlyMergeAccumulators(ctx, a, b), is the exact repro from the issue.reflectx.WrapMethods:context.Contextparameter;graph.NewCombineFn: it accepts the wrapped methods, andfuncxdetects their context parameter.go test ./go/pkg/beam/register/ -run TestCombiner_ContextCombiners -v: all 7 subtests PASS.With the pre-change
register.gofromupstream/master, the same test fails with the issue's panic:panic: Failed to optimize MergeAccumulators for combiner &{}. Failed to infer types.go test ./go/pkg/beam/register/...:ok.go vet ./go/pkg/beam/register/: clean.gofmt -l go/pkg/beam/register/: clean.go test ./go/pkg/beam/transforms/stats/... ./go/pkg/beam/transforms/top/... ./go/examples/snippets/...(other users ofregister.CombinerN): allok.beam.Combinewith a registered combiner that takes a ctx in every method. ItsAddInputincrements a counter through that ctx."55", and the counter is committed as 10."56"fails the passert, and expecting 11 for the counter fails the check.register.goit panics atinit.Pipeline test source
Open questions
registerpackage's test binary, running any pipeline callsbeam.Init. After that, the existing unit tests that callruntime.RegisterTypeinside test functions panic with "Init hooks have already run" (I tried it:TestEmitter1fails). If you'd like an end-to-end test in the repo, I'm happy to add it somewhere better suited, for examplesdks/go/test/integration/primitives. Let me know where you'd prefer it.🤖 Generated with Claude Code