Repository navigation
out_opensearch_data_stream: honor id_key - #200
Conversation
The data stream output wrote every record with an empty create header, so the id_key value never reached the _id of the document and duplicate events could not be deduplicated. Set _id from id_key before remove_keys runs, the same way the regular output does. Fixes fluent#124 Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for working on this. Honoring id_key in the data stream output is useful, and the change is small and easy to follow.
I marked each point with a priority. MUST items need to be fixed before merging. SHOULD items are strongly recommended. Details are in the inline comments.
- MUST: An exception while reading a nested
id_keysilently drops the record. - MUST: Some
_idvalues make the whole bulk request fail, so valid records in the same chunk are held back too. - SHOULD: A duplicate
_idreturns 409, and the data stream output reports it as an error. The regular output counts it as a duplicate.
SHOULD: Could you also update the data stream section of the README? It would help to mention that id_key is supported, and that records without the key still get an auto-generated ID. It would also help to note the limits on explicit IDs. They must be non-empty and no longer than 512 UTF-8 bytes, and objects and arrays are not supported.
Reviewed at 4a8beeb.
| record[@tag_key] = tag | ||
| end | ||
| meta = {} | ||
| if @id_key_accessor && (id = @id_key_accessor.call(record)) |
There was a problem hiding this comment.
MUST: An exception while reading a nested id_key silently drops the record.
For example, with id_key nested.request_id, a record containing {"nested" => "plain-string"} causes the record accessor to raise TypeError. The rescue below catches it. But in the output plugin, emit_error_label_event is the reader generated by config_param, so the block passed to it is never called. As a result, the record never reaches the bulk request, and nothing is logged or emitted to @ERROR. Before this PR, the same record was indexed with an auto-generated ID.
The block call is older than this PR, but this PR adds a new way to reach it.
I reproduced it by feeding these two records with id_key nested.request_id and emit_error_label_event true:
d.feed(event_time, {'message' => 'good', 'nested' => {'request_id' => '42'}})
d.feed(event_time, {'message' => 'bad', 'nested' => 'plain-string'})- 4a8beeb: only
goodis sent. There is no error event and no warn or error log. - main (8c46a3c): both records are sent.
Could you replace the block call with if emit_error_label_event?, as in the regular output, and add a regression test for this case?
| end | ||
| meta = {} | ||
| if @id_key_accessor && (id = @id_key_accessor.call(record)) | ||
| meta["_id".freeze] = id |
There was a problem hiding this comment.
MUST: Could you validate _id before adding the record to the bulk request, so an invalid ID cannot prevent valid records in the same chunk from being indexed?
The value is sent as _id as is, and OpenSearch rejects some values. I checked the plugin with stubbed responses and read the OpenSearch 2.11 source. I did not test against a real cluster.
- A Hash, an Array, or a value longer than 512 UTF-8 bytes fails the whole bulk request. Hash and Array values are rejected while parsing the action line (BulkRequestParser). Long values fail request validation (BulkRequest#validate, validateDocIdLength).
- The plugin turns that error into
RecoverableRequestFailure(here). So the whole chunk is retried according to the buffer settings, and valid records in the same chunk are held back with it. - An empty string is truthy in Ruby, so
{"request_id" => ""}sends"_id": "". OpenSearch rejects it as a per-item failure (IndexRequest, TransportBulkAction). Before this PR, that record was indexed with an auto-generated ID.
One option is to treat an empty string like a missing key, and send other invalid values to @ERROR.
Please also add a regression test that puts valid and invalid IDs in the same chunk. It should check that the valid records are still sent.
| @remove_keys.each { |key| record.delete(key) } | ||
| end | ||
| bulk_message = append_record_to_messages(CREATE_OP, {}, headers, record, bulk_message) | ||
| bulk_message = append_record_to_messages(CREATE_OP, meta, headers, record, bulk_message) |
There was a problem hiding this comment.
SHOULD: Could the data stream output count 409 on create as a duplicate, as the regular output does?
With _id, create returns 409 when the document already exists. This happens when a chunk is retried after the first request actually succeeded, for example after a timeout. Avoiding these duplicates is a main reason to set id_key, and it works. The problem is that the result is reported as an error.
The data stream output treats every status >= 300 as a failure (here). Each duplicate is counted as a failed item in the error log. With emit_error_label_event true (the default), it is also emitted to @ERROR as status 409. I confirmed this with a stubbed 409 response.
The regular output counts 409 on create as a duplicate instead (opensearch_error_handler.rb).
…uplicate Send records whose id_key value is an object, an array or longer than 512 bytes to @error instead of the bulk request, since OpenSearch fails the whole request for them. Treat an empty string like a missing key. Emit errors raised while building the bulk request when emit_error_label_event is on. The block passed to the config reader was never called, so these records were dropped silently. Count 409 responses as duplicates, as the regular output does, so a retried chunk does not report already indexed records as failures. Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>
|
Thanks for the detailed review. All points are addressed in d28b8b5:
The data stream tests pass (71 tests), and the full suite passes (458 tests). The 3 new tests fail on 4a8beeb. |
Fixes #124.
opensearch_data_streamacceptsid_keybut ignores it. Itswritemethod sends every record with an empty{"create":{}}header, so the document never gets the_idfrom the record. Because of this, theopensearch_genidplusid_keysetup can't deduplicate retried events.This change reads
id_keythrough the same record accessor as the regularopensearchoutput, so nested keys such asnested.request_idwork too. It sets_idin the create header and does this beforeremove_keysruns, soid_key _hashwithremove_keys _hashworks as it does in the regular output. When the record has no value for the key, the header stays empty, as before.Tests added to
test/plugin/test_out_opensearch_data_stream.rb:test_record_with_id_key:_idis set from a top-level key.test_record_with_nested_id_key_and_remove_keys: nested key, removed from the body after it is read.test_record_without_id_key_value: the header stays{}when the key is missing.(check all that apply)
versionin gemspec are untouched