Skip to content

out_opensearch_data_stream: honor id_key - #200

Merged
Watson1978 merged 2 commits into
fluent:mainfrom
RaphaelFakhri:fix-data-stream-id-key
Nov 4, 2026
Merged

Watson1978 merged 2 commits into
fluent:mainfrom
RaphaelFakhri:fix-data-stream-id-key

Conversation

@RaphaelFakhri

Copy link
Copy Markdown
Contributor

Fixes #124.

opensearch_data_stream accepts id_key but ignores it. Its write method sends every record with an empty {"create":{}} header, so the document never gets the _id from the record. Because of this, the opensearch_genid plus id_key setup can't deduplicate retried events.

This change reads id_key through the same record accessor as the regular opensearch output, so nested keys such as nested.request_id work too. It sets _id in the create header and does this before remove_keys runs, so id_key _hash with remove_keys _hash works as it does in the regular output. When the record has no value for the key, the header stays empty, as before.

Tests added to test/plugin/test_out_opensearch_data_stream.rb:

  • test_record_with_id_key: _id is set from a top-level key.
  • test_record_with_nested_id_key_and_remove_keys: nested key, removed from the body after it is read.
  • test_record_without_id_key_value: the header stays {} when the key is missing.

(check all that apply)

  • tests added
  • tests passing
  • README updated (if needed)
  • README Table of Contents updated (if needed)
  • History.md and version in gemspec are untouched
  • backward compatible

The data stream output wrote every record with an empty create
header, so the id_key value never reached the _id of the document
and duplicate events could not be deduplicated. Set _id from id_key
before remove_keys runs, the same way the regular output does.

Fixes fluent#124

Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this. Honoring id_key in the data stream output is useful, and the change is small and easy to follow.

I marked each point with a priority. MUST items need to be fixed before merging. SHOULD items are strongly recommended. Details are in the inline comments.

  • MUST: An exception while reading a nested id_key silently drops the record.
  • MUST: Some _id values make the whole bulk request fail, so valid records in the same chunk are held back too.
  • SHOULD: A duplicate _id returns 409, and the data stream output reports it as an error. The regular output counts it as a duplicate.

SHOULD: Could you also update the data stream section of the README? It would help to mention that id_key is supported, and that records without the key still get an auto-generated ID. It would also help to note the limits on explicit IDs. They must be non-empty and no longer than 512 UTF-8 bytes, and objects and arrays are not supported.

Reviewed at 4a8beeb.

record[@tag_key] = tag
end
meta = {}
if @id_key_accessor && (id = @id_key_accessor.call(record))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: An exception while reading a nested id_key silently drops the record.

For example, with id_key nested.request_id, a record containing {"nested" => "plain-string"} causes the record accessor to raise TypeError. The rescue below catches it. But in the output plugin, emit_error_label_event is the reader generated by config_param, so the block passed to it is never called. As a result, the record never reaches the bulk request, and nothing is logged or emitted to @ERROR. Before this PR, the same record was indexed with an auto-generated ID.

The block call is older than this PR, but this PR adds a new way to reach it.

I reproduced it by feeding these two records with id_key nested.request_id and emit_error_label_event true:

d.feed(event_time, {'message' => 'good', 'nested' => {'request_id' => '42'}})
d.feed(event_time, {'message' => 'bad', 'nested' => 'plain-string'})
  • 4a8beeb: only good is sent. There is no error event and no warn or error log.
  • main (8c46a3c): both records are sent.

Could you replace the block call with if emit_error_label_event?, as in the regular output, and add a regression test for this case?

end
meta = {}
if @id_key_accessor && (id = @id_key_accessor.call(record))
meta["_id".freeze] = id

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: Could you validate _id before adding the record to the bulk request, so an invalid ID cannot prevent valid records in the same chunk from being indexed?

The value is sent as _id as is, and OpenSearch rejects some values. I checked the plugin with stubbed responses and read the OpenSearch 2.11 source. I did not test against a real cluster.

  • A Hash, an Array, or a value longer than 512 UTF-8 bytes fails the whole bulk request. Hash and Array values are rejected while parsing the action line (BulkRequestParser). Long values fail request validation (BulkRequest#validate, validateDocIdLength).
  • The plugin turns that error into RecoverableRequestFailure (here). So the whole chunk is retried according to the buffer settings, and valid records in the same chunk are held back with it.
  • An empty string is truthy in Ruby, so {"request_id" => ""} sends "_id": "". OpenSearch rejects it as a per-item failure (IndexRequest, TransportBulkAction). Before this PR, that record was indexed with an auto-generated ID.

One option is to treat an empty string like a missing key, and send other invalid values to @ERROR.

Please also add a regression test that puts valid and invalid IDs in the same chunk. It should check that the valid records are still sent.

@remove_keys.each { |key| record.delete(key) }
end
bulk_message = append_record_to_messages(CREATE_OP, {}, headers, record, bulk_message)
bulk_message = append_record_to_messages(CREATE_OP, meta, headers, record, bulk_message)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHOULD: Could the data stream output count 409 on create as a duplicate, as the regular output does?

With _id, create returns 409 when the document already exists. This happens when a chunk is retried after the first request actually succeeded, for example after a timeout. Avoiding these duplicates is a main reason to set id_key, and it works. The problem is that the result is reported as an error.

The data stream output treats every status >= 300 as a failure (here). Each duplicate is counted as a failed item in the error log. With emit_error_label_event true (the default), it is also emitted to @ERROR as status 409. I confirmed this with a stubbed 409 response.

The regular output counts 409 on create as a duplicate instead (opensearch_error_handler.rb).

…uplicate

Send records whose id_key value is an object, an array or longer than 512
bytes to @error instead of the bulk request, since OpenSearch fails the
whole request for them. Treat an empty string like a missing key.

Emit errors raised while building the bulk request when
emit_error_label_event is on. The block passed to the config reader was
never called, so these records were dropped silently.

Count 409 responses as duplicates, as the regular output does, so a
retried chunk does not report already indexed records as failures.

Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>
@RaphaelFakhri

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review. All points are addressed in d28b8b5:

  • Nested id_key errors: the rescue now uses if emit_error_label_event?, so the record goes to @ERROR instead of being dropped. test_unreadable_nested_id_key_is_emitted_as_error_event feeds your two records and checks that good is sent and bad is emitted.
  • Invalid _id values: objects, arrays and values longer than 512 UTF-8 bytes are sent to @ERROR before the bulk request is built. An empty string is treated like a missing key, so it gets an auto-generated ID as before. test_invalid_id_values_do_not_block_valid_records puts valid and invalid IDs in one chunk, including a 512-byte multibyte ID at the limit, and checks that the valid records are still sent.
  • 409 on create: counted as a duplicate (logged at debug), not as a failed item or an @ERROR event. The "Could not bulk insert" error is only logged when real failures remain.
  • README: the data stream section now covers id_key, the auto-generated fallback, the ID limits and the 409 behavior.

The data stream tests pass (71 tests), and the full suite passes (458 tests). The 3 new tests fail on 4a8beeb.

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@Watson1978
Watson1978 merged commit 3fda76a into fluent:main Nov 4, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenSearch Output Data Stream - id_key ignored

2 participants