Repository navigation
out_opensearch: expand logstash_prefix placeholders in the target_index_affinity search - #204
Conversation
…ex_affinity search
get_affinity_target_indices built the search index from the
logstash_prefix reader, which returns the raw configuration value, so
with `logstash_prefix ${tag}` it searched `${tag}-*` and found nothing.
The update then went to the index of the current time instead of the
one that already holds the document. Fill in the placeholders from the
chunk, as process_message does for the index it writes to.
Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the fix. I checked it with OpenSearch 2.19.1. For an ordinary tag or record value, the search now finds the existing document and the update goes to its index.
One case needs a change before merge. Please see the inline comment.
With the logstash_prefix placeholders expanded in the affinity search, a tag or chunk key value with "," or "*" widened the search beyond the prefix, so an update could go to another tenant's index, and a value that is not a valid index name made the search, and the chunk, fail. Skip the search when a value used in logstash_prefix has such a character, or makes the expression start with _, - or +, so the record goes where it went before. "*" or "," written in logstash_prefix itself is kept. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the update. The three cases from my previous comment now keep the main result. I checked them with OpenSearch 2.19.1.
The previous comment is not fully resolved yet, though: #204 (comment). Some values still widen the search or make it fail. Each case below uses the earlier configuration with only the listed changes. So target_index_key destination is set, and the record has "destination": "logs-safe-2021.04.29". main writes the record to logs-safe-2021.04.29 in all of them.
logstash_prefix ${tenant}andlogstash_prefix_separator _, and the record has notenantor an empty one. The search goes to_*and returns 400 (must not start with '_'), so the chunk fails.logstash_prefix ${tag[1]}with the tagappfails the same way.- The same as 1, but with
logstash_prefix_separator "". The search goes to*, and the update goes to the document inlogs-blue-2021.04.28. logstash_prefix logs-${tag}, the tag is literally${tenant}, andtenantis*,private. Fluentd fills in${tenant}after it puts in the tag. So the search goes tologs-*,private-*and updateslogs-blue-2021.04.28. The check does not look attenant, because${tenant}is not in the configured prefix.
Cases 1 and 2 happen because the check looks at the prefix without logstash_prefix_separator, and an empty result passes it. Case 3 happens because the check collects the values, and a value can also come in through another placeholder.
One way to cover all three is to compare the prefix before and after the expansion. Compare @logstash_prefix plus @logstash_prefix_separator with the expanded prefix plus the separator. Leave out the trailing * that the search adds.
- Skip the search when any character of
UNSAFE_INDEX_EXPRESSION_CHARACTERSappears more times after the expansion than before. Count each character separately. A strftime format like%#pdrops its#, so a total count can hide an added*. - Also skip it when a comma-separated target becomes empty, or starts with
_,-or+, while the configured target does not."".split(",", -1)returns[], so an empty result needs its own check.
I tried this on top of 36ad95e. Your tests pass, and the cases above give the main result.
Could you add tests for these three cases?
Checking the values used by the configured placeholders missed an empty value, the separator, and a value that comes in through another placeholder. Compare the configured and the expanded prefix with the separator instead: skip the affinity search when an unsafe character is added, or when a target becomes empty or starts with _, - or +. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
|
Thanks for the cases and the approach. Pushed 9f19c8b, which compares the prefix before and after the expansion as you suggested:
New data-driven test
Each asserts no |
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the update. The three cases from my previous review now keep the main result. So do the other values I tried with OpenSearch 2.19.1, including values with , or *, empty values, values that come in through the tag, and strftime formats such as %#p.
I found one more value that makes the search fail. Please see the inline comment.
…255 bytes A long value from a tag or a record made the search exceed http.max_initial_line_length, and OpenSearch rejected it with 400, so the chunk failed. A target longer than 255 bytes, the limit of an index name, cannot match any index. Only the expanded side is checked. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
|
Thanks. Pushed 03e5b64 with your suggestion. New tests with
|
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the update. I checked 03e5b64 with OpenSearch 2.19.1. The long values from my previous review now skip the search and keep the main result. An expanded target of up to 255 bytes is still searched, and a * written in logstash_prefix is not counted. The three new cases for long values fail on 9f19c8b and pass now. The cases from my earlier reviews also still keep the main result.
LGTM. Thank you for working through all the rounds.
With
target_index_affinity true,get_affinity_target_indicessearches for existing documents in"#{logstash_prefix}#{@logstash_prefix_separator}*".logstash_prefixthere is the reader generated byconfig_param, so it holds the raw configuration value, placeholders included. Withlogstash_prefix ${tag}the search goes toPOST /${tag}-*/_searchand never finds anything. The update then goes to the index for the current date, which brings back the duplicate document this option is meant to avoid at index rotation.process_messagealready writes to the index built from the prefix expanded for the chunk (expand_placeholders). This change usesextract_placeholders(@logstash_prefix, chunk)for the search too, so the affinity search expands the samelogstash_prefixplaceholders as the write path. Configurations without placeholders search the same pattern as before.When the expansion adds
*,,or a character that an index name cannot have, turns a target empty or into one starting with_,-or+, or leaves a target longer than 255 bytes (not counting*), the affinity search is skipped and the record follows the normal write path. A*or,written inlogstash_prefixitself is kept.Tests added to
test/plugin/test_out_opensearch.rb:test_writes_to_affinity_target_index_with_logstash_prefix_placeholder:logstash_prefix ${tag}with tagtestsearchestest-*and updates the document in the index that search returns. Onmainit fails withUnregistered request: POST http://localhost:9200/$%7Btag%7D-*/_search.test_skips_affinity_search_for_unsafe_logstash_prefix_valueandtest_skips_affinity_search_when_the_expanded_prefix_is_unsafe: values that add*or,, also through another placeholder in the tag, and values that make a target empty or start it with_. Each skips the search and updates the index from the normal write path.test_skips_affinity_search_when_the_expanded_prefix_is_too_long: values that make the target longer than 255 bytes ("a" * 250,"a" * 4096,"あ" * 460) skip the search.test_affinity_search_for_an_expanded_prefix_up_to_255_bytes: a target of 255 bytes is still searched, withlogs-${tenant}and withlogs*-${tenant}.test_affinity_search_keeps_wildcard_written_in_logstash_prefix: a*written inlogstash_prefixis kept in the search.TZ=UTC bundle exec rake test: 475 tests, 0 failures, 0 errors on this branch, and 481 tests, 0 failures, 0 errors when merged withmainat 983b052. WithoutTZ=UTC,test_out_opensearch_data_stream.rbtest_time_placeholdercan fail near midnight, onmaintoo, because its stub uses the local date and the request the UTC date.(check all that apply)
versionin gemspec are untouched