[9.5](backport #7593) Add option to enable graceful unenroll to invalid API key agents.

[mergify]

What is the problem this PR solves?

Once an Elastic Agent is force unenrolled the Elastic Agents will continue to communicate to Fleet Server. There are cases where it would be best to just have the Elastic Agent stop all of its components, unenroll if it can (those with tamper protection on will not be able to), and those that cannot will continue to receive 401's for a full hour, until the cycle starts again.

How does this PR solve the problem?

This changes the behavior of invalid API keys from being a 401 error that just gets retried non-stop to a 200 with a policy change action that is an empty policy, then a unenroll action, and then back to the 401 error.

1. The empty policy will cause the Elastic Agent to stop all of its running components.
2. The unenroll action will cause the Elastic Agent to stop communicating with Fleet (as long as there is not tamper protection enabled, if enabled this will not stop the Elastic Agent).
3. 401 original error returns for a full hour.

How to test this PR locally

1. Enroll an Elastic Agent.
2. Perform force unenroll in Fleet.
3. Wait for next check-in and see that it stops all of its running components.
4. Wait for follow-up check-in and see that it unenrolls.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• [ ] I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• [ ] I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

---

This is an automatic backport of pull request #7593 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 34a8c89 has failed:

On branch mergify/bp/9.5/pr-7593
Your branch is up to date with 'origin/9.5'.

You are currently cherry-picking commit 34a8c89.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1786137338-unenroll-on-invalid-api-key.yaml
	modified:   fleet-server.reference.yml
	modified:   internal/pkg/api/handleCheckin_test.go
	modified:   internal/pkg/config/input.go
	modified:   internal/pkg/server/fleet.go
	modified:   internal/pkg/server/fleet_integration_test.go
	modified:   testing/e2e/stand_alone_test.go
	new file:   testing/e2e/testdata/stand-alone-https-unenroll.tpl

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/api/handleCheckin.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-actions]

TL;DR

The Buildkite failures are caused by unresolved merge conflict markers committed in internal/pkg/api/handleCheckin.go, which breaks Go parsing during check-ci and both packaging jobs. Resolve the conflict in that file and rerun CI.

Remediation

• Remove <<<<<<<, =======, and >>>>>>> conflict markers in internal/pkg/api/handleCheckin.go, then keep the intended combined CheckinT struct/options implementation for the backport.
• Re-run mage check:all (or at minimum goimports/build) to confirm the file parses and formatting checks pass before re-triggering Buildkite.

Investigation details

Root Cause

This is a merge-conflict carryover in the backport branch (mergify/bp/9.5/pr-7593). The PR diff itself for internal/pkg/api/handleCheckin.go contains conflict markers in the CheckinT struct block (around the gwPool/bulker fields), which makes the file invalid Go source.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16338
• Jobs/steps:
  • Run check-ci
  • Package x86_64
  • Package x86_64 FIPS
• Key log excerpts:
  • internal/pkg/api/handleCheckin.go:104:1: expected '}', found '<<'
  • internal/pkg/api/handleCheckin.go:127:1: expected statement, found '>>'
  • internal/pkg/api/handleCheckin.go:127:85: illegal character U+0023 '#'
  • packaging jobs also fail with equivalent parser errors (syntax error: unexpected <<, unexpected ==, unexpected >>)
• Prior PR context also matches: Mergify reported both modified: internal/pkg/api/handleCheckin.go during backport cherry-pick conflict resolution.

Verification

• Not run in this environment (read-only detective workflow); conclusions are based on Buildkite logs + PR patch content.

Follow-up

• After resolving the conflict markers, rerun the same Buildkite pipeline; no separate flaky-test indicator was found for this failure pattern.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.