fix(runtimes): set defaultMode on MPI SSH Secret volume

[h0pers]

What this PR does / why we need it:

Sets defaultMode and per-item mode on the MPI SSH Secret volume to restrict private key permissions. Without this fix, Kubernetes defaults all mounted Secret files to 0644 (world-readable), making the SSH private key accessible to any user in the container.

Supersedes #3368 and addresses its unresolved review feedback.

File modes:

• Private key (id_rsa): 0o640 (owner rw, group r) — needs group-read because builtin runtimes (deepspeed, mlx) run as runAsUser: 1000 while Secret volumes mount as root:root
• Public key and authorized_keys: 0o644 (owner rw, world r) — public data, unchanged from default
• Volume defaultMode: 0o400 (owner read-only) — defense-in-depth fallback for any future items added without an explicit per-item mode

Which issue(s) this PR fixes:
Fixes #3262

Checklist:

• Docs included if any changes are user facing

[github-actions]

🎉 Welcome to the Kubeflow Trainer! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.
• Our team will review your PR soon! cc @kubeflow/kubeflow-trainer-team

Join the community:

• Slack: Join our #kubeflow-trainer Slack channel.
• Meetings: Attend the Kubeflow AutoML and Training Working Group bi-weekly meetings.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[Copilot]

Pull request overview

This PR hardens the MPI plugin’s SSH Secret volume permissions by explicitly setting defaultMode and per-item mode values so the mounted SSH private key is no longer world-readable, aligning the runtime behavior with least-privilege expectations for in-pod SSH auth material.

Changes:

• Set defaultMode on the MPI SSH auth Secret volume and explicit mode on each mounted Secret item (private key, public key, authorized_keys).
• Centralize the file mode values as constants for reuse and consistency.
• Update unit, framework, runtime, and integration tests to assert the new volume permissions.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
pkg/runtime/framework/plugins/mpi/mpi.go	Sets Secret volume DefaultMode and per-item Mode for SSH key material in the MPI plugin.
pkg/constants/constants.go	Introduces constants for the SSH Secret volume default mode and per-file modes.
pkg/runtime/framework/plugins/mpi/mpi_test.go	Updates MPI plugin unit tests to expect the new defaultMode and item mode fields.
pkg/runtime/framework/core/framework_test.go	Updates framework component-builder expectations for the SSH Secret volume permissions.
pkg/runtime/core/trainingruntime_test.go	Updates runtime construction tests to include the new Secret volume modes.
test/integration/controller/trainjob_controller_test.go	Updates integration assertions for rendered volumes to include DefaultMode and per-item Mode.

[andreyvelich]

/ok-to-test
/retest

[andreyvelich]

Thanks for picking this up and addressing the prior feedback, @h0pers! This review was done using AI tooling. One small non-blocking nit inline.

[andreyvelich]

/assign @robert-bell @astefanutti @abhijeet-dhumal @tenzen-y
/lgtm

[google-oss-prow]

@andreyvelich: GitHub didn't allow me to assign the following users: robert-bell.

Note that only kubeflow members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Details

In response to this:

/assign @robert-bell @astefanutti @abhijeet-dhumal @tenzen-y
/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[robert-bell]

Thanks @h0pers this looks great!

/lgtm

[google-oss-prow]

@robert-bell: changing LGTM is restricted to collaborators

Details

In response to this:

Thanks @h0pers this looks great!

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[abhijeet-dhumal]

Thanks @h0pers ! 🙌
/lgtm

[andreyvelich]

/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment