[first_epss] Add latest transform for ES|QL LOOKUP JOIN enrichment by clement-fouque · Pull Request #21843 · elastic/integrations

clement-fouque · 2026-10-02T13:49:02Z

Proposed commit message

Adds a latest Elasticsearch transform (latest_vulnerability) to the first_epss integration that deduplicates the re-ingested EPSS catalog into one row per CVE, writing the result to a lookup-mode destination index aliased as logs-first_epss_latest.vulnerability.

The FIRST.org feed re-ingests its full catalog on every poll cycle, so the same CVE repeats with updated scores in logs-first_epss.vulnerability-*. Collapsing those snapshots into a lookup index lets users enrich vulnerability findings at query time with the ES|QL LOOKUP JOIN command on vulnerability.id.

The transform dedups on vulnerability.id, sorts by event.ingested to keep the latest score, and excludes cold/frozen tier copies since the full catalog re-ingests each cycle anyway. It mirrors the existing cisa_kevs, hackerone, and qualys_gav lookup-transform pattern.

The package format_version is bumped to 3.6.6 to support index.mode: lookup, and the package version is bumped to 1.6.0.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

From packages/first_epss/:

elastic-package format
elastic-package lint
elastic-package check
elastic-package build
elastic-package test static

# With a running stack
elastic-package stack up -d
elastic-package test asset
elastic-package test system --data-streams vulnerability

After install, confirm the logs-first_epss_latest.vulnerability transform runs and smoke-test a join:

FROM logs-endpoint.vulnerability-*
| LOOKUP JOIN logs-first_epss_latest.vulnerability ON vulnerability.id
| KEEP vulnerability.id, first_epss.vulnerability.epss, first_epss.vulnerability.percentile, first_epss.vulnerability.date
| WHERE first_epss.vulnerability.epss IS NOT NULL

LOOKUP JOIN against a lookup-mode index requires Elasticsearch 9.1+. On older stacks, target the concrete destination index logs-first_epss_latest.dest_vulnerability-1.

…ansform that supports deduplicated EPSS score lookups per CVE.

…by adding external references and creating new YAML files for latest vulnerability fields and input types.

…ta processing, enabling deduplicated EPSS score lookups per CVE.

…OKUP JOIN, explaining the latest transform for deduplicated EPSS score lookups per CVE.

Update the changelog link from the fork issue to elastic#21843. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-10-02T13:50:43Z

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

Address Vale Elastic.Semicolons suggestion on the query-time enrichment section. Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot

Copilot review overview

🟡 Changes recommended

The documented alias query fails on supported Elasticsearch 9.0 deployments without the concrete-index fallback.

Review effort: Balanced
Findings: 1

Open (1)

Document 9.1+ requirement for lookup alias example · New

What changed in this PR

Adds a latest-state lookup transform for query-time EPSS enrichment by CVE.

Changes:

Creates and maps the lookup-mode transform destination.
Documents ES|QL enrichment usage.
Bumps the package version and changelog.

File	Description
`packages/first_epss/manifest.yml`	Bumps package and format versions.
`packages/first_epss/elasticsearch/transform/latest_vulnerability/transform.yml`	Defines latest-CVE transform behavior.
`packages/first_epss/elasticsearch/transform/latest_vulnerability/manifest.yml`	Configures the lookup index template.
`packages/first_epss/elasticsearch/transform/latest_vulnerability/fields/is-transform-source-false.yml`	Marks transform output documents.
`packages/first_epss/elasticsearch/transform/latest_vulnerability/fields/fields.yml`	Maps EPSS output fields.
`packages/first_epss/elasticsearch/transform/latest_vulnerability/fields/beats.yml`	Maps the input type field.
`packages/first_epss/elasticsearch/transform/latest_vulnerability/fields/base-fields.yml`	Defines base transform fields.
`packages/first_epss/docs/README.md`	Adds generated enrichment documentation.
`packages/first_epss/data_stream/vulnerability/fields/base-fields.yml`	Uses ECS base-field definitions.
`packages/first_epss/changelog.yml`	Records the 1.6.0 enhancement.
`packages/first_epss/_dev/build/docs/README.md`	Adds source enrichment documentation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Copilot review overview

🔵 Needs a closer look

The documented alias query fails on supported pre-9.1 Elasticsearch versions without the concrete-index fallback.

Review effort: Balanced
Findings: 1

Open (1)

Document 9.1+ requirement for lookup alias example

infra-vault-gh-plugin-prod · 2026-10-03T13:22:59Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

… files

vera-review-bot · 2026-10-05T07:55:49Z

    enable_request_tracer: true
 assert:
-  hit_count: 3
+  hit_count: 3


Severity: 🔵 Low confidence: medium path: packages/cisa_kevs/data_stream/vulnerability/_dev/test/system/test-default-config.yml:9

This PR touches an unrelated package (cisa_kevs) with a trailing-newline-only change; drop it from this PR so the first_epss change stays self-contained and does not pull cisa_kevs into this PR's CI run.

Details

The only change to packages/cisa_kevs/...test-default-config.yml is adding a trailing newline. It has no functional effect, is not mentioned in the PR description or any changelog, and causes CI to treat cisa_kevs as an affected package for this PR (see is_pr_affected in .buildkite/scripts/common.sh). It looks like an accidental artifact of running a repo-wide formatter.

Recommendation:

Revert the file to its main state in this PR, or if the formatter fix is wanted, land it in a separate trivial PR:

git checkout origin/main -- packages/cisa_kevs/data_stream/vulnerability/_dev/test/system/test-default-config.yml

🤖 AI-Generated Review | Vera Review Bot - v0.4.2 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Copilot

Copilot review overview

🟡 Changes recommended

The generated README is missing documentation generated from the new transform and field assets.

Review effort: Balanced
Findings: 1

Open (1)

Regenerated README omits transform assets and exported field · New

Resolved since last review (1)

Document 9.1+ requirement for lookup alias example

elastic-vault-github-plugin-prod · 2026-10-05T08:14:43Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

vera-review-bot · 2026-10-05T22:45:59Z

+        "name": "elastic-agent-99127",
        "type": "filebeat",
-        "version": "8.13.0"
+        "version": "9.5.4"


Severity: 🟡 Medium confidence: high path: packages/cisa_kevs/docs/README.md:65

Regenerated cisa_kevs artifacts (docs/README.md sample event, sample_event.json, pipeline expected output, test config newline) are included without a cisa_kevs version bump or changelog entry; revert all packages/cisa_kevs changes from this PR.

Details

The PR modifies packages/cisa_kevs/docs/README.md (sample event: agent ids, agent version 8.13.0 -> 9.5.4, namespace, event.ingested, date rendering), sample_event.json, the pipeline expected output, and _dev/test/system/test-default-config.yml (trailing newline only). None of these relate to the first_epss transform; they are byproducts of running the cisa_kevs test suite locally on a newer stack. The cisa_kevs manifest.yml and changelog.yml are untouched, so package content changes without a version bump, contrary to the repository convention that every package content change ships with a bump and changelog entry, and the change pulls cisa_kevs owners into an unrelated review. Prior bot finding 6131f1ef2ab5d30f raised the newline-only change; the README and sample event regeneration are new since that review (they appear as added lines in the re-review delta), so this extends that finding rather than repeating it. Also note the PR description says the transform mirrors an existing cisa_kevs lookup transform, but packages/cisa_kevs has no elasticsearch/transform/ directory in this checkout; hackerone and qualys_gav are the actual precedents.

Recommendation:

Revert the cisa_kevs files to their main state:

git checkout origin/main -- packages/cisa_kevs

If the cisa_kevs sample refresh is wanted, land it separately with its own version bump and changelog entry.

🤖 AI-Generated Review | Vera Review Bot - v0.4.2 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

efd6

This needs an elastic-package build. Also, please amend the proposed commit message to remove markdown and have the commit message body be prose rather than a WHAT/WHY-bulleted list.

Copilot

Copilot review overview

🟡 Changes recommended

The unrelated cisa_kevs changes should be reverted or released separately with appropriate metadata.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Revert unrelated cisa_kevs artifact changes · New
Regenerated README omits transform assets and exported field

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot review overview

🟢 Approval recommended

The transform, mappings, package metadata, and documentation consistently implement the established lookup-index pattern.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Revert unrelated cisa_kevs artifact changes
Regenerated README omits transform assets and exported field

elastic-vault-github-plugin-prod · 2026-10-06T20:47:36Z

✅ All changelog entries have the correct PR link.

vera-review-bot · 2026-10-06T20:59:09Z

🟢 Reviewed the latest commits 3aa889a (history rewritten since 591a150) — nothing new beyond already posted comments.

⚠️ 2 comments still unresolved from earlier commits — 1 medium, 1 low

🟡 Regenerated cisa_kevs artifacts (docs/README.md sample event, sample_event.json, pipeline expected output, test config newline) are included without a cisa_kevs version bump or changelog entry (link)
🔵 This PR touches an unrelated package (cisa_kevs) with a trailing-newline-only change (link)

Review summary

Issues found across earlier commits f046209 — 1 medium

🟡 Regenerated cisa_kevs artifacts (docs/README.md sample event, sample_event.json, pipeline expected output, test config newline) are included without a cisa_kevs version bump or changelog entry (link) (Outdated)

Issues found across earlier commits c009ba3…4cb7418 (3 commits) — 1 low

🔵 This PR touches an unrelated package (cisa_kevs) with a trailing-newline-only change (link) (Outdated)

Issues found across earlier commits 86fe356 — 1 medium, 2 low

🟡 The lookup index declares no mapping for vulnerability.id (the LOOKUP JOIN key) or the other ECS fields the latest transform copies from source (link)
🔵 The destination index is labelled labels.is_transform_source: "false" but the source data stream never sets the label to "true" (link)

Package-level:

🔵 Proposed commit message

first_epss: add latest transform for ES|QL LOOKUP JOIN enrichment

Add a latest transform (latest_vulnerability) that collapses the
re-ingested EPSS catalog into one document per CVE and writes it to a
lookup-mode destination index, logs-first_epss_latest.dest_vulnerability-1,
aliased as logs-first_epss_latest.vulnerability.

The FIRST.org EPSS feed re-ingests the full catalog on every poll cycle,
so the same CVE appears repeatedly in logs-first_epss.vulnerability-* with
updated scores. Keeping only the latest row per CVE lets users enrich
vulnerability findings at query time with the ES|QL LOOKUP JOIN command on
vulnerability.id, which the ingest pipeline already sets from
first_epss.vulnerability.cve.

The transform deduplicates on vulnerability.id, sorts by event.ingested,
excludes error documents and the cold and frozen tiers, and runs unattended
with deduce_mappings disabled. The package format_version is raised to
3.6.6 and the package version to 1.6.0.

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.4.2 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

infra-vault-gh-plugin-prod · 2026-10-06T21:01:32Z

💚 Build Succeeded

Buildkite Build
Commit: 3aa889a

History

💔 Build #51057 failed f046209
💔 Build #51012 failed 4cb7418
💚 Build #50984 succeeded 86fe356
💚 Build #50982 succeeded 467c9d3

clement-fouque · 2026-10-06T21:15:20Z

@efd6 it should be fine now. Can you please review again and merge as well if it's fine. Thank you.

github-actions · 2026-10-07T02:02:12Z

elastic-vault-github-plugin-prod · 2026-10-07T02:18:08Z

Package first_epss - 1.6.0 containing this change is available at https://epr.elastic.co/package/first_epss/1.6.0/

clement-fouque and others added 6 commits October 2, 2026 15:43

Update manifest.yml to version 1.6.0 and change format_version to 3.6.6

37fc86a

Add version 1.6.0 to changelog with enhancement for a new latest tr…

cf1222c

…ansform that supports deduplicated EPSS score lookups per CVE.

Refactor vulnerability field definitions to align with ECS standards …

7f2eca5

…by adding external references and creating new YAML files for latest vulnerability fields and input types.

Add new manifest and transform YAML files for latest vulnerability da…

479d2e5

…ta processing, enabling deduplicated EPSS score lookups per CVE.

Enhance README.md with details on query-time EPSS enrichment using LO…

01c48ef

…OKUP JOIN, explaining the latest transform for deduplicated EPSS score lookups per CVE.

first_epss: point 1.6.0 changelog entry to upstream PR

467c9d3

Update the changelog link from the fork issue to elastic#21843. Co-authored-by: Cursor <cursoragent@cursor.com>

clement-fouque marked this pull request as ready for review October 2, 2026 14:09

clement-fouque requested review from a team as code owners October 2, 2026 14:09

Copilot AI balanced review requested due to automatic review settings October 2, 2026 14:09

Copilot started reviewing on behalf of clement-fouque October 2, 2026 14:11 View session

first_epss: split semicolon sentence in LOOKUP JOIN docs

86fe356

Address Vale Elastic.Semicolons suggestion on the query-time enrichment section. Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot AI reviewed Oct 2, 2026

View reviewed changes

Comment thread packages/first_epss/_dev/build/docs/README.md

Copilot AI balanced review requested due to automatic review settings October 2, 2026 14:15

Copilot started reviewing on behalf of clement-fouque October 2, 2026 14:15 View session

Copilot AI reviewed Oct 2, 2026

View reviewed changes

vera-review-bot Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread packages/first_epss/elasticsearch/transform/latest_vulnerability/fields/fields.yml

Comment thread ...first_epss/elasticsearch/transform/latest_vulnerability/fields/is-transform-source-false.yml

This was referenced Oct 4, 2026

Wayfinder: VulnExploit v2 redesign clement-fouque/vulnexploit#6

Open

T1: GitHub Advisories lookup-mode index availability clement-fouque/vulnexploit#7

Closed

clement-fouque added 2 commits October 5, 2026 09:51

Add new YAML file for is_transform_source field set to true.

c009ba3

Add new ECS field definitions for vulnerability data in separate YAML…

7a32fef

… files

Copilot AI balanced review requested due to automatic review settings October 5, 2026 07:52

Copilot started reviewing on behalf of clement-fouque October 5, 2026 07:52 View session

vera-review-bot Bot reviewed Oct 5, 2026

View reviewed changes

Copilot AI reviewed Oct 5, 2026

View reviewed changes

Comment thread packages/first_epss/docs/README.md

qcorporation added the Integration:cisa_kevs CISA Known Exploited Vulnerabilities (Community supported) label Oct 5, 2026

Copilot AI balanced review requested due to automatic review settings October 5, 2026 22:36

Copilot AI reviewed Oct 5, 2026

vera-review-bot Bot reviewed Oct 5, 2026

View reviewed changes

efd6 reviewed Oct 6, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 6, 2026 20:32

Copilot started reviewing on behalf of clement-fouque October 6, 2026 20:33 View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

Comment thread packages/cisa_kevs/data_stream/vulnerability/sample_event.json Outdated

Regenerate package

3aa889a

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings October 6, 2026 20:42

clement-fouque force-pushed the feat/first_epss-latest-transform branch from 591a150 to 3aa889a Compare October 6, 2026 20:42

Copilot started reviewing on behalf of clement-fouque October 6, 2026 20:42 View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

efd6 approved these changes Oct 7, 2026

View reviewed changes

efd6 merged commit c2bd350 into elastic:main Oct 7, 2026
16 checks passed

Conversation

clement-fouque commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Elastic Docs Style Checker (Vale)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

infra-vault-gh-plugin-prod Bot commented Oct 3, 2026

Uh oh!

Uh oh!

Uh oh!

vera-review-bot Bot Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Oct 5, 2026

🚀 Benchmarks report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

vera-review-bot Bot Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

elastic-vault-github-plugin-prod Bot commented Oct 6, 2026

Uh oh!

vera-review-bot Bot commented Oct 6, 2026

Review summary

Uh oh!

infra-vault-gh-plugin-prod Bot commented Oct 6, 2026

💚 Build Succeeded

History

Uh oh!

clement-fouque commented Oct 6, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 7, 2026

Backport branches

Uh oh!

elastic-vault-github-plugin-prod Bot commented Oct 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

clement-fouque commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading