kubernetes executor: support node selector and tolerations - #2
Merged
Conversation
Add --kubernetes-node-selector and --kubernetes-tolerations flags so run jobs can be pinned to a dedicated/tainted node pool. Tolerations use the kubectl-taint syntax key[=value]:effect (Exists when no value).
Expose the new --kubernetes-node-selector and --kubernetes-tolerations flags through the otfd and otf-agent helm charts (via the runner library), regenerate READMEs, bump chart versions, and document the flags in flags.md and executors.md.
Member
Author
|
Superseded by the upstream draft PR leg100#973 (same branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two flags to the kubernetes executor so run Jobs can be pinned to a dedicated (and typically tainted) node pool, instead of scheduling anywhere in the cluster.
Changes
--kubernetes-node-selector— repeatablename=value, setsPodSpec.NodeSelector.--kubernetes-tolerations— repeatablekey[=value]:effect(kubectl-taint syntax; operator isExistswhen no value, elseEqual; empty effect tolerates all effects), setsPodSpec.Tolerations.Both are parsed in
newKubeExecutor(consistent with the existing--kubernetes-labelshandling) and applied to the Job pod template.Why
Lets terraform plan/apply pods target an isolated node pool so cluster-autoscaler can scale that pool with run volume, keeping the per-run CPU off shared nodes. The kubernetes executor previously exposed no scheduling controls (no nodeSelector/tolerations/affinity), and GKE does not expose the
PodNodeSelector/PodTolerationRestrictionadmission plugins to do this cluster-side.Testing
go build ./internal/runner/andgo vet ./internal/runner/pass.go test ./internal/runner/ -run TestNewKubeExecutor|TestKubeExecutor_SpawnOperationpasses.