fix(sink): batch transformed raw data - #4093
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #4093 +/- ##
==========================================
+ Coverage 70.79% 70.84% +0.05%
==========================================
Files 463 464 +1
Lines 54339 54443 +104
==========================================
+ Hits 38466 38568 +102
- Misses 12887 12888 +1
- Partials 2986 2987 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ngjaying
marked this pull request as ready for review
August 4, 2026 00:20
Signed-off-by: Jiyong Huang <huangjy@emqx.io>
Yisaer
approved these changes
Aug 4, 2026
ngjaying
added a commit
that referenced
this pull request
Aug 5, 2026
- Support batching sink data produced as pre-encoded `RawTuple` values by `dataTemplate`. - Preserve format-specific batch framing for JSON, delimited, and URL-encoded output. - Document how `sendSingle`, batching, `dataTemplate`, and `format` interact. - A sink using both `batchSize` and `dataTemplate` could fail at runtime with `unknown data type: *xsql.RawTuple`. - The streaming batch pipeline transforms each input before the batch writer flushes it, but the batch writer previously accepted only structured rows and tuple lists. - Planning raw versus structured output up front keeps the runtime path deterministic and lets each format writer frame pre-encoded template results without encoding them again. - Add the optional `RawConvertWriter` capability and implement it for JSON, delimited, URL-encoded, and fallback stack writers. - Let `TransformOp` report whether it produces raw output and select raw/common batch writer mode during sink planning, including the slice-tuple exception. - Fall back to `StackWriter` during planning when a selected writer does not support raw output. - Transfer flushed buffer ownership so later writer reuse cannot mutate emitted batches. - Add deterministic ownership regression tests across writer reuse cycles. - Add unit, race, and FVT coverage for raw batching, planner output mode selection, legacy writer fallback, and REST sink output. - Add English and Chinese documentation with processing-order and format-framing guidance. - Existing `ConvertWriter` implementations remain source compatible because `RawConvertWriter` is optional. - In batch mode, `dataTemplate` describes one batch element rather than the completed batch. Template authors remain responsible for producing output compatible with the configured format. - JSON adds commas and an outer array, delimited output adds newlines, and URL-encoded output adds `&` between transformed elements. - Validation performed: - `go test ./internal/topo/node ./internal/topo/planner ./internal/converter ./internal/converter/json ./internal/converter/delimited ./internal/converter/urlencoded ./pkg/message ./pkg/modules` - `go test -count=1 -race ./internal/topo/node ./internal/topo/planner ./internal/converter ./internal/converter/json ./internal/converter/delimited ./internal/converter/urlencoded` - `go test ./fvt -run '^TestRuleSuite$/^TestBatchDataTemplateJSON$' -count=1 -v` - Markdown lint for the modified English and Chinese sink documentation Signed-off-by: Jiyong Huang <huangjy@emqx.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RawTuplevalues bydataTemplate.sendSingle, batching,dataTemplate, andformatinteract.Why
batchSizeanddataTemplatecould fail at runtime withunknown data type: *xsql.RawTuple.Changes In This PR
RawConvertWritercapability and implement it for JSON, delimited, URL-encoded, and fallback stack writers.TransformOpreport whether it produces raw output and select raw/common batch writer mode during sink planning, including the slice-tuple exception.StackWriterduring planning when a selected writer does not support raw output.Notes
ConvertWriterimplementations remain source compatible becauseRawConvertWriteris optional.dataTemplatedescribes one batch element rather than the completed batch. Template authors remain responsible for producing output compatible with the configured format.&between transformed elements.go test ./internal/topo/node ./internal/topo/planner ./internal/converter ./internal/converter/json ./internal/converter/delimited ./internal/converter/urlencoded ./pkg/message ./pkg/modulesgo test -count=1 -race ./internal/topo/node ./internal/topo/planner ./internal/converter ./internal/converter/json ./internal/converter/delimited ./internal/converter/urlencodedgo test ./fvt -run '^TestRuleSuite$/^TestBatchDataTemplateJSON$' -count=1 -v