[ci] Use mock LLM for Lightspeed CRC deployment by malingatembo · Pull Request #110 · openstack-k8s-operators/lightspeed-operator

malingatembo · 2026-09-30T20:21:13Z

[ci] Deploy Lightspeed with a mock provider and run integration tests

Deploy Lightspeed with the mock LLM, prepare the lightspeed-tests checkout,
and run its integration suite after Lightspeed reaches Ready. Retain JUnit
reports and runner logs, and fail the deployment job when tests fail.

The deployment-only job has passed upstream. The controller integration
passed five tests locally on CRC; an invalid-token run verified failure
propagation and report retention. Combined upstream validation of these
integration changes remains pending.

Related-Issue: OSPRH-33344
Depends-On: openstack-k8s-operators/ci-framework#4222

coderabbitai · 2026-09-30T20:21:21Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

centosinfra-prod-github-app · 2026-10-01T00:07:12Z

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/46a50010bc0348f28d474e0632bd3260

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 28m 09s
❌ lightspeed-operator-deployment-crc FAILURE in 1h 19m 49s (non-voting)
✔️ lightspeed-operator-kuttl SUCCESS in 48m 33s

malingatembo · 2026-10-05T13:23:20Z

recheck

centosinfra-prod-github-app · 2026-10-05T20:57:15Z

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/94c05be1ae6745a496f6495e206fccf4

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 39m 00s
❌ lightspeed-operator-deployment-crc FAILURE in 1h 17m 32s (non-voting)
✔️ lightspeed-operator-kuttl SUCCESS in 49m 19s

malingatembo · 2026-10-06T09:13:51Z

recheck

centosinfra-prod-github-app · 2026-10-07T03:36:18Z

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/2e276703acdb4589ac66f8ec30d8028e

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 01m 37s
✔️ lightspeed-operator-deployment-crc SUCCESS in 1h 36m 57s (non-voting)
✔️ lightspeed-operator-kuttl SUCCESS in 49m 19s

lpiwowar · 2026-10-07T07:16:31Z

Nice progress! The lightspeed operator got deployed with the mock LLM server:) 🎉

Just an observation. If we want to continue to use the mock LLM server and we hit problem during running of the tests then the problem might be missing /v1/models endpoint on the server side 2:

INFO     2026-10-07T02:59:37.142725Z  HTTP Request: GET                                                                 
         http://mock-llm-api-server-pod.openstack-lightspeed.svc.cluster.local:8000/v1/models "HTTP/1.1 404 Not Found"  
         category=uncategorized                                                                                         
ERROR    2026-10-07T02:59:37.145884Z  ogx.providers.utils.inference.openai_mixin:580 list_provider_model_ids() failed   
         category=providers::utils error=Error code: 404 - {'detail': 'Not Found'} provider=OpenAIInferenceAdapter      
WARNING  2026-10-07T02:59:37.147718Z  ogx.core.routing_tables.models:102 Model refresh skipped                          
         category=core::routing_tables provider_id=provider-default-model                                               
INFO     2026-10-07T02:59:37.358187Z  ogx.core.stack:880 starting registry refresh task   category=core                 
         interval_seconds=300

umago · 2026-10-07T09:06:37Z

Nice progress! The lightspeed operator got deployed with the mock LLM server:) 🎉

Just an observation. If we want to continue to use the mock LLM server and we hit problem during running of the tests then the problem might be missing /v1/models endpoint on the server side 2:

INFO     2026-10-07T02:59:37.142725Z  HTTP Request: GET                                                                 
         http://mock-llm-api-server-pod.openstack-lightspeed.svc.cluster.local:8000/v1/models "HTTP/1.1 404 Not Found"  
         category=uncategorized                                                                                         
ERROR    2026-10-07T02:59:37.145884Z  ogx.providers.utils.inference.openai_mixin:580 list_provider_model_ids() failed   
         category=providers::utils error=Error code: 404 - {'detail': 'Not Found'} provider=OpenAIInferenceAdapter      
WARNING  2026-10-07T02:59:37.147718Z  ogx.core.routing_tables.models:102 Model refresh skipped                          
         category=core::routing_tables provider_id=provider-default-model                                               
INFO     2026-10-07T02:59:37.358187Z  ogx.core.stack:880 starting registry refresh task   category=core                 
         interval_seconds=300

Nice find!

Maybe we can quickly add the v1/models to the mocked server then ? Even part of this work, should be straight forward

centosinfra-prod-github-app · 2026-10-09T03:50:12Z

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/6103f337490e47deb846051648a7568e

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 16m 52s
✔️ lightspeed-operator-deployment-crc SUCCESS in 1h 38m 37s (non-voting)
✔️ lightspeed-operator-kuttl SUCCESS in 55m 10s

lpiwowar · 2026-10-09T08:23:29Z

This is amazing! 🎉 -> https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/logs/df8/rdoproject.org/df89eec2e65145959fed02b30b506d5d/controller/ci-framework-data/logs/pre_tests_02_run_lightspeed_tests.log

We are not running the tests via test-operator though, right? I'm not sure what to think at the moment of it. If it works it works but also the ci-framework part is a bit complicated at the moment. Probably something to talk about:).

In this openstack-k8s-operators/lightspeed-tests#12 it is described how to run the tests using the tests operator. Running the tests using that configuration should be just about passing it via the corresponding ci-framework params (an example of AnsibleTest config in zuul.yaml)

centosinfra-prod-github-app · 2026-10-09T11:09:49Z

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/dec093f7311f4984bf297bd806ba0fd2

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 47m 45s
❌ lightspeed-operator-deployment-crc FAILURE in 1h 23m 47s (non-voting)
✔️ lightspeed-operator-kuttl SUCCESS in 51m 52s

centosinfra-prod-github-app · 2026-10-10T23:08:45Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/openstack-k8s-operators/ci-framework for 4222,2d1db6134f13b845d950bbce4ed1a19c8bfc7868

malingatembo · 2026-10-10T23:30:31Z

recheck

centosinfra-prod-github-app · 2026-10-11T01:24:32Z

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/cea0b0c41dd943708df08858ab6d0514

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 52m 26s
❌ lightspeed-operator-deployment-crc FAILURE in 1h 30m 25s (non-voting)
✔️ lightspeed-operator-kuttl SUCCESS in 46m 29s

malingatembo · 2026-10-11T07:30:18Z

recheck

centosinfra-prod-github-app · 2026-10-11T09:46:44Z

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/5bba08bca9be42b0964934dfe535edfa

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 06m 19s
❌ lightspeed-operator-deployment-crc FAILURE in 1h 36m 09s (non-voting)
✔️ lightspeed-operator-kuttl SUCCESS in 47m 50s

malingatembo · 2026-10-11T13:35:31Z

recheck

centosinfra-prod-github-app · 2026-10-11T15:36:34Z

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/5501e46cc9214bf58278794421d4b6a0

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 59m 26s
❌ lightspeed-operator-deployment-crc FAILURE in 1h 36m 32s (non-voting)
✔️ lightspeed-operator-kuttl SUCCESS in 48m 49s

centosinfra-prod-github-app · 2026-10-11T21:36:14Z

This change depends on a change that failed to merge.

Change openstack-k8s-operators/ci-framework#4222 is needed.

malingatembo · 2026-10-11T22:09:23Z

recheck

centosinfra-prod-github-app · 2026-10-12T00:16:13Z

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/02070809c0454a3e8312e25315cf018a

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 05m 13s
✔️ lightspeed-operator-deployment-crc SUCCESS in 1h 43m 22s (non-voting)
✔️ lightspeed-operator-kuttl SUCCESS in 45m 02s

lpiwowar

This is awesome:)

Notes from the meeting. The following two things can be done as part of next set of tickets:

Ticket no. 1: Connect it to OpenAI using the creds from Jan
Ticket no. 2: Improve the tests results gathering (make sure that they are in separate folder here)

We still need a cleanup on the side of the ci-framework PR and move the variables for the test-operator role here (ref).

BUT IT WORKS and it is AWESOME!:) Hopefully the cleanup will suffice, let's see 🤞

Point the deployment job at the mock provider supplied by the ci-framework hook instead of an unreachable placeholder endpoint. Related-Issue: OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>

Prepare the Lightspeed test repository alongside the operator and framework. Enable the mock explicitly, deploy Lightspeed, then run application integration tests and publish JUnit reports and runner logs. Related-Issue: OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>

Retain the inherited Tempest stage and run Lightspeed through the test-operator role after deployment. Propagate application test failures to the deployment job. Related-Issue: #OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>

Select the 0.7.0 OGX image containing the upstream Solr KNN fix. Keep the API image at its operator default because the final API image failed PostgreSQL startup in CRC validation. Related-Issue: #OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>

Configure the AnsibleTest runner directly through existing job settings and remove the preparation hook, extra Tempest stage and OGX image override. Keep result collection and authenticated test access. As per Lukas reviews. Use only the in-cluster mock and dummy provider token. Leave the real OpenAI secret and its lifecycle hooks disconnected from this job; real-provider validation is follow-up work. Merge lightspeed-tests openstack-k8s-operators#17 before final validation against upstream default. Related-Issue: #OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>

openshift-ci · 2026-10-20T11:33:09Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: malingatembo
Once this PR has been reviewed and has the lgtm label, please assign lpiwowar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Temporarily select the published tests PR openstack-k8s-operators#17 branch from the author fork so the native AnsibleTest clone includes its fixes before they are merged. Keep the fake LLM configuration and existing test authentication. Restore the upstream repository and default branch after openstack-k8s-operators#17 merges, then recheck that final configuration. Related-Issue: #OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>

malingatembo · 2026-10-20T13:28:37Z

recheck

Pass the configured service model to the Goose Responses API tests and exclude tests marked llm_dependent. Keep the fake provider endpoint and dummy token so ci can run without real llm creds. Signed-off-by: malingatembo <tembomalinga@gmail.com>

malingatembo · 2026-10-21T11:22:55Z

recheck

CI bootstrap saves the job parameters before the test runner and result PVC facts exist. Supply bootstrap defaults in the existing reader override while retaining the real runner identity and discovered PVCs at collection. Resolve the collector's nested instance-name template in mount paths so the generated reader can expose the result directories to log collection. Keep the reader UID, group and SELinux settings and the mock test selection. Signed-off-by: malingatembo <tembomalinga@gmail.com>

lpiwowar

Just couple of comments from the meeting:). Overall it looks really good! 👍 Tests are running. These are just comments related to the polishing of the code.

lpiwowar · 2026-10-23T09:36:52Z

+          memory: 2Gi
+      # Allow bootstrap to save parameters before runner and PVC facts exist.
+      # The collector supplies those facts when it creates this Pod.
+      cifmw_test_operator_log_pod_definition:


From the meeting (non-blocking): This has been added by AI. We need to understand why.

lpiwowar · 2026-10-23T09:39:21Z

+          type: ansibletest
+
+      cifmw_test_operator_namespace: openstack
+      cifmw_test_operator_privileged: true


From the meeting (non-blocking): Is this needed? If so, why?

lpiwowar · 2026-10-23T09:40:38Z

+      cifmw_test_operator_ansibletest_ansible_git_repo: https://github.com/malingatembo/lightspeed-tests
+      cifmw_test_operator_ansibletest_ansible_git_branch: feat/OSPRH-33344-ci-test-runner


issue (blocking): This needs to be updated before we merge.

lpiwowar · 2026-10-23T10:12:50Z

I just want to say that this is pretty much done. We can build on top of it and keep improving it. There are couple of comments on the ci-framework part as well but we are almost there:)

openshift-ci Bot requested review from Akrog and umago September 30, 2026 20:21

malingatembo marked this pull request as draft September 30, 2026 20:24

openshift-ci Bot added the do-not-merge/work-in-progress label Sep 30, 2026

lpiwowar reviewed Oct 16, 2026

View reviewed changes

Comment thread zuul.d/jobs.yaml Outdated

Comment thread zuul.d/jobs.yaml Outdated

Comment thread zuul.d/jobs.yaml

openshift-ci Bot added the needs-rebase label Oct 17, 2026

malingatembo added 3 commits October 20, 2026 13:25

[ci] Use mock LLM for Lightspeed CRC deployment

3029e4c

Point the deployment job at the mock provider supplied by the ci-framework hook instead of an unreachable placeholder endpoint. Related-Issue: OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>

malingatembo added 2 commits October 20, 2026 13:25

malingatembo force-pushed the feat/OSPRH-33344-use-mock-llm branch from 5f49f7a to b489b27 Compare October 20, 2026 11:33

openshift-ci Bot removed the needs-rebase label Oct 20, 2026

malingatembo added the ok-to-test label Oct 20, 2026

lpiwowar reviewed Oct 23, 2026

View reviewed changes

		cifmw_test_operator_ansibletest_ansible_git_repo: https://github.com/malingatembo/lightspeed-tests
		cifmw_test_operator_ansibletest_ansible_git_branch: feat/OSPRH-33344-ci-test-runner

Conversation

malingatembo commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Draft PR not reviewed

Uh oh!

centosinfra-prod-github-app Bot commented Oct 1, 2026

Uh oh!

malingatembo commented Oct 5, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Oct 5, 2026

Uh oh!

malingatembo commented Oct 6, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Oct 7, 2026

Uh oh!

lpiwowar commented Oct 7, 2026

Uh oh!

umago commented Oct 7, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Oct 9, 2026

Uh oh!

lpiwowar commented Oct 9, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Oct 9, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Oct 10, 2026

Uh oh!

malingatembo commented Oct 10, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Oct 11, 2026

Uh oh!

malingatembo commented Oct 11, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Oct 11, 2026

Uh oh!

malingatembo commented Oct 11, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Oct 11, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Oct 11, 2026

Uh oh!

malingatembo commented Oct 11, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Oct 12, 2026

Uh oh!

lpiwowar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci Bot commented Oct 20, 2026

Uh oh!

malingatembo commented Oct 20, 2026

Uh oh!

malingatembo commented Oct 21, 2026

Uh oh!

lpiwowar left a comment

Choose a reason for hiding this comment

Uh oh!

lpiwowar Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

lpiwowar Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

lpiwowar Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

lpiwowar commented Oct 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

malingatembo commented Sep 30, 2026 •

edited

Loading

coderabbitai Bot commented Sep 30, 2026 •

edited

Loading