feat: Implement a retry mechanism for kubernetes job spawning

[mlanth]

This PR adds a retry mechanism to the kubernetes executor spawn operation to mitigate hung runs caused by API rejection on busy clusters. Issue surfaced through testing large volume of concurrent runs causing etcd instability. While there are cluster measures to be taken to help boost etcd performance, it is out of scope and this PR focuses on implementing recoverability.

Problem

With the kubernetes executor, two bugs cause a run to hang in planning or applying with no pod, no log output and no error surfaced in the UI until its phase timeout expires. On a busy deployment this is invisible as the run sits there with no indication of why and just shows an active phase with no log output.

Both bugs are triggered by rejection from the Kubernetes API server. The issue surfaced as etcdserver: request timed out while running 100s of concurrent runs. The issue isn't that the API server occasionally rejects a write which is normal and recoverable, it's that OTF turns the recoverable rejection into a stranded run.

Bug 1: job is marked running before its pod exists

In the Runner.Start process jobs loop:

token, err := r.runners.StartJob(ctx, j.ID)   // job -> running, run -> planning/applying
...
if err := r.executor.SpawnOperation(ctx, g, j, token); err != nil {
    return fmt.Errorf("spawning job operation: %w", err)
}

StartJob commits the state transition before the pod is created. When SpawnOperation then fails, processJobs returns an error and the backoff.RetryNotify wrapper retries it but the job is now JobRunning, so the next line of the loop skips it forever:

for _, j := range jobs {
    if j.Status != JobAllocated {
        continue    // the failed job is never retried
    }

The allocator can't help either: it only reallocates jobs in JobAllocated, so nothing in the system ever touches that job again. It stays running with no corresponding Kubernetes Job, and its run hangs until PlanningTimeout / ApplyingTimeout.

The OTFD logs are the clearest way to see it. A healthy spawn produces five lines:

allocated job     ... phase=plan
received job
started plan
created kubernetes secret for job token
created kubernetes job

A failed one produces three, then nothing until the phase timeout fires:

allocated job     ... phase=apply
received job
started apply
                  <- no secret, no kubernetes job, ever
ERROR processing jobs  error="spawning job operation: creating kubernetes secret
                              for job token: etcdserver: request timed out"
...some time later...
canceled job      ... status=canceled

Bug 2: setting the job token secret's owner reference is fatal

SpawnOperation makes three sequential API calls:

#	Call	On failure today
1	secrets.Create	fatal - correct, nothing was created
2	jobs.Create	fatal - correct, no pod will run
3	secrets.Update (owner reference)	fatal - incorrect

By the time (3) runs, the kubernetes Job has already been created and its pod is going to run to completion. Returning an error there reports a job as having failed to spawn when it has in fact spawned and via Bug 1 that strands the run, for a job that is working as expected. The only real consequence of a missing owner reference is that the secret isn't garbage collected along with its job.

Fix

1. Retry the spawn

SpawnOperation now retries the whole spawn rather than each call individually. The new spawn struct records which steps have completed, so a retry resumes from the first incomplete one and issues exactly one API call, the one that failed rather than repeating work:

if !sp.secretCreated { ... }
if !sp.jobCreated    { ... }
// The kubernetes job now exists and is going to run to completion, so every
// remaining step is optional.
if sp.kjob == nil    { ... }   // recover the UID after a lost create response
if !sp.ownerRefSet   { ... }

AlreadyExists is treated as success, which is what makes the resume correct. etcdserver: request timed out is ambiguous, the write may have committed and only the response was lost so without this, retrying would convert a successful write into a hard failure.

2. An owner-reference failure no longer fails the spawn

Steps reached after the Job exists return an optionalStepError. They're retried like anything else, but if the budget is exhausted SpawnOperation returns nil, because the pod is running:

var optional *optionalStepError
if errors.As(err, &optional) {
    s.Logger.Error(optional, "spawned kubernetes job but could not complete an optional step; ...")
    return nil
}
return err

As previously mentioned, the only real consequence of a missing owner reference is that the secret isn't garbage collected along with its job.

3. Report the job errored when the spawn ultimately fails

When retries are exhausted, Runner.failJob reports the job as JobErrored, which errors its run phase. The run fails visibly with the API error attached, and is immediately re-runnable, instead of hanging for the phase timeout.

This needed no new service method, API endpoint, or state transition. Service.FinishJob only accepts the job itself as caller, and StartJob has already returned the job's token, so failJob builds a client via the existing operationClientCreator, the same mechanism DoOperation uses.