fix: Receiving TargetDown after upgrading GitOps

[akhilnittala]

What type of PR is this?

Uncomment only one /kind line, and delete the rest.
For example, > /kind bug would simply become: /kind bug

/kind bug

/kind cleanup
/kind failing-test
/kind enhancement
/kind documentation
/kind code-refactoring

What does this PR do / why we need it:
Issue Description / Context:
During the upgrade from an older version (1.17.2) to a newer version (1.17.3) — noting that the specific versions mentioned are for testing and validation purposes — we observed that the TargetDown alert was being triggered. Investigation showed that the metrics endpoint was returning a 403 Forbidden response.

Root Cause Analysis:
The 403 error indicates an authorization failure. One identified scenario is that the required RBAC permissions were missing for accessing the /metrics endpoint. As a result, Prometheus was unable to scrape metrics successfully, leading to the TargetDown alert.

Resolution / Fix:
To address this issue, we updated the RBAC configuration by explicitly adding the /metrics non-resource URL to the relevant ClusterRole/ClusterRoleBinding. With this change in place, Prometheus is able to access the metrics endpoint successfully, and the TargetDown alert is resolved.

Outcome:
Post-change verification confirms that metrics scraping works as expected and no further 403 errors are observed.
Have you updated the necessary documentation?

• Documentation update is required by this PR.
• Documentation has been updated.

Which issue(s) this PR fixes:
https://issues.redhat.com/browse/GITOPS-8591
Fixes #?
https://issues.redhat.com/browse/GITOPS-8591
Test acceptance criteria:

• Unit Test
• E2E Test

How to test changes / Special notes to the reviewer:
Install the GitOps Operator using a version that includes the relevant changes.

Navigate to the OLM UI and verify the metrics targets.

Confirm that all targets are in an UP state and correctly labeled with the gitops identifier.

Ensure that no alerts are triggered for TargetDown.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[svghadi]

Can you run make generate manifests bundle to fix the CI?

[svghadi]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: svghadi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [svghadi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[svghadi]

/cherry-pick v1.19
/cherry-pick v1.18
/cherry-pick v1.17

[openshift-cherrypick-robot]

@svghadi: once the present PR merges, I will cherry-pick it on top of v1.17, v1.18, v1.19 in new PRs and assign them to you.

Details

In response to this:

/cherry-pick v1.19
/cherry-pick v1.18
/cherry-pick v1.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-cherrypick-robot]

@svghadi: #1044 failed to apply on top of branch "v1.19":

Applying: fix: Reciving TargetDown after upgrading GitOps
Applying: fix: Reciving TargetDown after upgrading GitOps
Using index info to reconstruct a base tree...
M	bundle/manifests/gitops-operator.clusterserviceversion.yaml
Falling back to patching base and 3-way merge...
Auto-merging bundle/manifests/gitops-operator.clusterserviceversion.yaml
CONFLICT (content): Merge conflict in bundle/manifests/gitops-operator.clusterserviceversion.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0002 fix: Reciving TargetDown after upgrading GitOps

Details

In response to this:

/cherry-pick v1.19
/cherry-pick v1.18
/cherry-pick v1.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-cherrypick-robot]

@svghadi: #1044 failed to apply on top of branch "v1.18":

Applying: fix: Reciving TargetDown after upgrading GitOps
Applying: fix: Reciving TargetDown after upgrading GitOps
Using index info to reconstruct a base tree...
M	bundle/manifests/gitops-operator.clusterserviceversion.yaml
Falling back to patching base and 3-way merge...
Auto-merging bundle/manifests/gitops-operator.clusterserviceversion.yaml
CONFLICT (content): Merge conflict in bundle/manifests/gitops-operator.clusterserviceversion.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0002 fix: Reciving TargetDown after upgrading GitOps

Details

In response to this:

/cherry-pick v1.19
/cherry-pick v1.18
/cherry-pick v1.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-cherrypick-robot]

@svghadi: #1044 failed to apply on top of branch "v1.17":

Applying: fix: Reciving TargetDown after upgrading GitOps
Applying: fix: Reciving TargetDown after upgrading GitOps
Using index info to reconstruct a base tree...
M	bundle/manifests/gitops-operator.clusterserviceversion.yaml
Falling back to patching base and 3-way merge...
Auto-merging bundle/manifests/gitops-operator.clusterserviceversion.yaml
CONFLICT (content): Merge conflict in bundle/manifests/gitops-operator.clusterserviceversion.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0002 fix: Reciving TargetDown after upgrading GitOps

Details

In response to this:

/cherry-pick v1.19
/cherry-pick v1.18
/cherry-pick v1.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.