Requirement tests can match the wrong event when samples share a sourcetype

[mkolasinski-splunk]

Summary

Event-backed requirement tests can return false-positive results when multiple XML samples share a sourcetype. PSA knows each sample's explicitly declared transport host, but currently carries that value only in the pytest parameter ID. The requirement-field search receives only the sourcetype, and the requirement-datamodel search receives no structured metadata constraint.

As a result, the test generated for sample B can retrieve sample A's event and validate the wrong event successfully.

Observed failure mode

Sample A: sourcetype=vendor:product, host=transport-a, expected fields are correct
Sample B: sourcetype=vendor:product, host=transport-b, one expected extraction is broken

Current result:
sample B's search can return sample A's event -> sample B passes incorrectly

This was reproduced in splunk-add-on-for-carbon-black PR #381. That PR implements a narrow consumer-side mitigation by parsing PSA's pytest parameter ID, resolving pytest's numeric duplicate-ID suffixes, and writing a quoted host into modinput_params.

The workaround is intentionally temporary because pytest IDs are display labels, not a stable data contract.

Root cause

In FieldTestGenerator.generate_requirements_tests, PSA builds:

modinput_params = {
    "sourcetype": event.metadata.get("sourcetype_to_search"),
}

The effective sample host is included only in an ID such as:

sample_name::sample.xml::host::sample-host

Pytest may append numeric suffixes to duplicate explicit IDs, so recovering the host from that string is ambiguous.

Historically, PSA included host and source in requirement search parameters. They were removed in PR #658 while requirement tests were refactored from one test per field to one test per event; the PR does not document an intentional change to host scoping.

Important compatibility finding

A naïve fix that always adds event.metadata["host"] to modinput_params is unsafe:

• hosts containing spaces, quotes, or backslashes need correct SPL quoting;
• metadata["host"] is not always the indexed host for host_type=event, SC4S, UF monitoring, or index-time host rewrites;
• --ingest-events=false can validate externally supplied events whose host is not controlled by PSA;
• Edge Processor tests already use a stronger UUID selector;
• Carbon Black's temporary hook pre-quotes the legacy host, which could be double-quoted by a new formatter.

Proposed design

1. Preserve explicit transport-host provenance

When parsing a requirement XML event, preserve the raw value of an explicitly declared transport host separately from general event metadata.

Only explicit XML transport hosts should become new search constraints. Generated or stanza-default metadata hosts must not be promoted automatically.

This keeps the change narrow and avoids guessing the final indexed host for ingestion paths PSA does not fully control.

2. Add structured search constraints

Add an additive raw search_constraints mapping to both event-backed parameter types:

• splunk_searchtime_fields_requirements
• splunk_searchtime_fields_datamodels

Example:

{
    "search_constraints": {
        "host": "Carbon Black v761",
    },
    # existing fixture data remains present
}

Keep modinput_params for backward compatibility.

3. Apply structured-over-legacy precedence

At query construction:

1. preserve today's verbatim behavior for legacy modinput_params keys;
2. render structured constraint values through the new formatter;
3. if both mappings contain the same key, use only the structured value.

This makes the Carbon Black compatibility hook harmless after upgrading: its legacy pre-quoted host is ignored when PSA supplies the structured host, so the value is formatted exactly once.

Legacy/custom fixtures that lack search_constraints continue to execute unchanged.

4. Format structured SPL values at the query boundary

Store raw semantic values in generated parameters. Render new structured strings as balanced double-quoted SPL literals, escaping embedded backslashes and double quotes.

Do not use JSON serialization as the SPL contract.

5. Preserve existing selectors

• When an Edge Processor UUID is available, use the UUID selector without requiring host scoping.
• When no explicit XML transport host exists, retain legacy event selection.
• Keep pytest parameter IDs unchanged for readable output, but never parse them for behavior.

Proposed behavior

Case	Expected behavior
Explicit XML transport host	Both event-backed requirement searches include safely quoted host
No explicit transport host	Legacy unscoped behavior remains
Host contains spaces/quotes/backslashes	One valid quoted SPL predicate
Duplicate pytest parameter ID	Numeric suffix has no effect on selection
Edge Processor UUID	UUID remains the selector; host is not required
Legacy/custom fixture	Existing modinput_params behavior remains
Consumer still has the Carbon Black hook	Structured host wins; no double quoting
xdist/parser cache/pregenerated events	Raw structured constraint survives serialization unchanged

Acceptance criteria

• Explicit XML transport host provenance survives tokenization, copying, pickle-based pregeneration, parser caching, and xdist distribution.
• Requirement-field and requirement-datamodel parameters receive the same additive structured host contract.
• Samples sharing a sourcetype but declaring different hosts cannot satisfy one another's event-backed requirement searches.
• A sample with a broken expected extraction fails even if another same-sourcetype sample contains the expected field.
• Hosts containing whitespace, double quotes, and backslashes produce valid SPL string literals.
• New structured values take precedence over same-key legacy values.
• Legacy-only/custom fixtures retain existing query construction.
• Edge Processor UUID tests retain existing behavior.
• Pytest IDs and duplicate-ID suffixes are never parsed for search behavior.
• Targeted Docker e2e coverage proves the false-positive regression is closed.
• Unit, xdist, and pre-commit checks pass.
• Requirement-test documentation describes the explicit-host contract and the consumer-shim removal path.

Suggested implementation areas

• pytest_splunk_addon/sample_generation/sample_stanza.py
• pytest_splunk_addon/sample_generation/sample_event.py and copy/serialization paths as needed
• pytest_splunk_addon/fields_tests/test_generator.py
• pytest_splunk_addon/fields_tests/test_templates.py
• focused unit tests under tests/unit/tests_standard_lib/
• targeted requirement e2e fixtures and constants under tests/e2e/
• docs/requirement_tests.md

Non-goals

• inferring a final host for every ingestion mode;
• changing ingestion behavior;
• expanding UUID support to unsupported ingestors;
• changing pytest IDs or test counts;
• adding source scoping;
• changing CIM data or package dependencies.