Skip to content

fix(sink): batch transformed raw data - #4093

Merged
ngjaying merged 1 commit into
lf-edge:masterfrom
ngjaying:template
Aug 4, 2026
Merged

ngjaying merged 1 commit into
lf-edge:masterfrom
ngjaying:template

Conversation

@ngjaying

@ngjaying ngjaying commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Support batching sink data produced as pre-encoded RawTuple values by dataTemplate.
  • Preserve format-specific batch framing for JSON, delimited, and URL-encoded output.
  • Document how sendSingle, batching, dataTemplate, and format interact.

Why

  • A sink using both batchSize and dataTemplate could fail at runtime with unknown data type: *xsql.RawTuple.
  • The streaming batch pipeline transforms each input before the batch writer flushes it, but the batch writer previously accepted only structured rows and tuple lists.
  • Planning raw versus structured output up front keeps the runtime path deterministic and lets each format writer frame pre-encoded template results without encoding them again.

Changes In This PR

  • Add the optional RawConvertWriter capability and implement it for JSON, delimited, URL-encoded, and fallback stack writers.
  • Let TransformOp report whether it produces raw output and select raw/common batch writer mode during sink planning, including the slice-tuple exception.
  • Fall back to StackWriter during planning when a selected writer does not support raw output.
  • Transfer flushed buffer ownership so later writer reuse cannot mutate emitted batches.
  • Add deterministic ownership regression tests across writer reuse cycles.
  • Add unit, race, and FVT coverage for raw batching, planner output mode selection, legacy writer fallback, and REST sink output.
  • Add English and Chinese documentation with processing-order and format-framing guidance.

Notes

  • Existing ConvertWriter implementations remain source compatible because RawConvertWriter is optional.
  • In batch mode, dataTemplate describes one batch element rather than the completed batch. Template authors remain responsible for producing output compatible with the configured format.
  • JSON adds commas and an outer array, delimited output adds newlines, and URL-encoded output adds & between transformed elements.
  • Validation performed:
    • go test ./internal/topo/node ./internal/topo/planner ./internal/converter ./internal/converter/json ./internal/converter/delimited ./internal/converter/urlencoded ./pkg/message ./pkg/modules
    • go test -count=1 -race ./internal/topo/node ./internal/topo/planner ./internal/converter ./internal/converter/json ./internal/converter/delimited ./internal/converter/urlencoded
    • go test ./fvt -run '^TestRuleSuite$/^TestBatchDataTemplateJSON$' -count=1 -v
    • Markdown lint for the modified English and Chinese sink documentation

@codecov

codecov Bot commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 79.27928% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.84%. Comparing base (e0ad968) to head (d0fc448).

Files with missing lines Patch % Lines
internal/topo/node/batch_writer_op.go 54.84% 10 Missing and 4 partials ⚠️
internal/converter/urlencoded/writer.go 83.78% 4 Missing and 2 partials ⚠️
internal/converter/delimited/writer.go 82.35% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4093      +/-   ##
==========================================
+ Coverage   70.79%   70.84%   +0.05%     
==========================================
  Files         463      464       +1     
  Lines       54339    54443     +104     
==========================================
+ Hits        38466    38568     +102     
- Misses      12887    12888       +1     
- Partials     2986     2987       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ngjaying
ngjaying marked this pull request as ready for review August 4, 2026 00:20
Signed-off-by: Jiyong Huang <huangjy@emqx.io>
@ngjaying
ngjaying merged commit 4cb929b into lf-edge:master Aug 4, 2026
70 of 73 checks passed
ngjaying added a commit that referenced this pull request Aug 5, 2026
- Support batching sink data produced as pre-encoded `RawTuple` values
by `dataTemplate`.
- Preserve format-specific batch framing for JSON, delimited, and
URL-encoded output.
- Document how `sendSingle`, batching, `dataTemplate`, and `format`
interact.

- A sink using both `batchSize` and `dataTemplate` could fail at runtime
with `unknown data type: *xsql.RawTuple`.
- The streaming batch pipeline transforms each input before the batch
writer flushes it, but the batch writer previously accepted only
structured rows and tuple lists.
- Planning raw versus structured output up front keeps the runtime path
deterministic and lets each format writer frame pre-encoded template
results without encoding them again.

- Add the optional `RawConvertWriter` capability and implement it for
JSON, delimited, URL-encoded, and fallback stack writers.
- Let `TransformOp` report whether it produces raw output and select
raw/common batch writer mode during sink planning, including the
slice-tuple exception.
- Fall back to `StackWriter` during planning when a selected writer does
not support raw output.
- Transfer flushed buffer ownership so later writer reuse cannot mutate
emitted batches.
- Add deterministic ownership regression tests across writer reuse
cycles.
- Add unit, race, and FVT coverage for raw batching, planner output mode
selection, legacy writer fallback, and REST sink output.
- Add English and Chinese documentation with processing-order and
format-framing guidance.

- Existing `ConvertWriter` implementations remain source compatible
because `RawConvertWriter` is optional.
- In batch mode, `dataTemplate` describes one batch element rather than
the completed batch. Template authors remain responsible for producing
output compatible with the configured format.
- JSON adds commas and an outer array, delimited output adds newlines,
and URL-encoded output adds `&` between transformed elements.
- Validation performed:
- `go test ./internal/topo/node ./internal/topo/planner
./internal/converter ./internal/converter/json
./internal/converter/delimited ./internal/converter/urlencoded
./pkg/message ./pkg/modules`
- `go test -count=1 -race ./internal/topo/node ./internal/topo/planner
./internal/converter ./internal/converter/json
./internal/converter/delimited ./internal/converter/urlencoded`
- `go test ./fvt -run '^TestRuleSuite$/^TestBatchDataTemplateJSON$'
-count=1 -v`
- Markdown lint for the modified English and Chinese sink documentation

Signed-off-by: Jiyong Huang <huangjy@emqx.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants