Summary
In arch-v2, destination-scoped processors run AFTER the destination write, not before it. v1 does the opposite. There is zero test coverage of destination-scoped processors in arch-v2, which is why this has gone unnoticed.
Found while planning slice 3c. Pre-existing since #1913, unrelated to the multi-connector work.
The code
pkg/lifecycle-poc/service.go, buildDestinationTasks:
destTasks := make([]funnel.Task, 0)
destTasks = append(destTasks, destTask) // destination FIRST
destTasks = append(destTasks, procTasks...) // its processors AFTER
buildSharedTail chains them in that order, so the runtime graph is DestinationTask → proc1 → proc2. The destination write happens first; the processors then transform records whose output goes only to acker.Ack and nowhere else.
v1 chains them the other way — pkg/lifecycle/service.go builds the destination's processor nodes between prev and the destination node, so processors run before the write.
Impact
A destination-scoped processor is silently a no-op with respect to what the destination actually receives. Configure field.exclude to strip a secret before writing to a sink, or field.set to reshape a payload for a specific destination, and arch-v2 writes the untransformed record and then transforms a copy that goes nowhere.
That is a correctness and potentially a data-exposure problem (a redaction processor scoped to one destination does not redact), and it is silent — no error, no warning.
Scope today: Preview.PipelineArchV2 only. That engine is scheduled to become the default, so this is an arch-v2 graduation blocker.
Why it was never caught
grep for ProcessorIDs / destTasks across pkg/lifecycle-poc/*_test.go and pkg/lifecycle-poc/funnel/*_test.go returns nothing. Destination-scoped processors are entirely untested in arch-v2 — the feature exists in the builder and has no test that would notice it running in the wrong order.
Fix
Chain the destination's processors before the destination task, matching v1. Ships with the regression test that fails without the fix: a destination-scoped processor that mutates the payload, asserting the destination receives the mutated record.
Worth also adding a test for the source-scoped equivalent, to confirm that ordering is right (it appears to be, but it is equally untested).
Summary
In arch-v2, destination-scoped processors run AFTER the destination write, not before it. v1 does the opposite. There is zero test coverage of destination-scoped processors in arch-v2, which is why this has gone unnoticed.
Found while planning slice 3c. Pre-existing since #1913, unrelated to the multi-connector work.
The code
pkg/lifecycle-poc/service.go,buildDestinationTasks:buildSharedTailchains them in that order, so the runtime graph isDestinationTask → proc1 → proc2. The destination write happens first; the processors then transform records whose output goes only toacker.Ackand nowhere else.v1 chains them the other way —
pkg/lifecycle/service.gobuilds the destination's processor nodes betweenprevand the destination node, so processors run before the write.Impact
A destination-scoped processor is silently a no-op with respect to what the destination actually receives. Configure
field.excludeto strip a secret before writing to a sink, orfield.setto reshape a payload for a specific destination, and arch-v2 writes the untransformed record and then transforms a copy that goes nowhere.That is a correctness and potentially a data-exposure problem (a redaction processor scoped to one destination does not redact), and it is silent — no error, no warning.
Scope today:
Preview.PipelineArchV2only. That engine is scheduled to become the default, so this is an arch-v2 graduation blocker.Why it was never caught
grepforProcessorIDs/destTasksacrosspkg/lifecycle-poc/*_test.goandpkg/lifecycle-poc/funnel/*_test.goreturns nothing. Destination-scoped processors are entirely untested in arch-v2 — the feature exists in the builder and has no test that would notice it running in the wrong order.Fix
Chain the destination's processors before the destination task, matching v1. Ships with the regression test that fails without the fix: a destination-scoped processor that mutates the payload, asserting the destination receives the mutated record.
Worth also adding a test for the source-scoped equivalent, to confirm that ordering is right (it appears to be, but it is equally untested).