Weightless support for all initializers#29607
Draft
chilo-ms wants to merge 11 commits into
Draft
Conversation
Add API surface for expanding weightless mode to cover all initializers (internal and external), working in both JIT and AOT flows: Session options: - Add ep.enable_weightless: unified weightless option for all initializers - Add ep.context_source_model_path: runtime override for source model location - Deprecate ep.enable_weightless_ep_context_nodes (external-only, AOT-only) OrtEpApi: - Add GetWeightlessSupport: EP capability query for the handshake. App requests weightless (session option), EP confirms support (this API). OrtCompileApi: - Add ModelCompilationOptions_SetWeightlessCache: enable weightless for AOT compilation path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Clarify that extending OrtValue lifetime past Compile() for direct pointer caching is planned but not yet implemented. Currently, EPs should use drop_constant_initializers=false and access initializers via KernelContext_GetInput() at Compute() time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… time Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… timing - ep.enable_weightless: clarify JIT uses KernelContext path (direct pointer caching is TODO), remove premature lifetime extension claim - ep.context_source_model_path: fix 'inference time' to 'session creation' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add two new OrtApi functions for providing the source model's initializer data when creating a session from a weightless EPContext model: - SessionOptionsSetWeightlessSourceModelPath: file path to source model, acts as runtime override for the onnx_model_filename EPContext attribute. - SessionOptionsSetWeightlessSourceModelFromBuffer: in-memory byte buffer for scenarios where the source model is not on disk. Caller must keep the buffer alive for the session lifetime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SessionOptionsSetWeightlessSourceModelPath: - Stores file path in OrtSessionOptions for runtime override of onnx_model_filename EPContext attribute - Clears any previously set buffer source SessionOptionsSetWeightlessSourceModelFromBuffer: - Stores borrowed buffer pointer + size in OrtSessionOptions - Caller must keep buffer alive for session lifetime - Clears any previously set file path source ModelCompilationOptions_SetWeightlessCache: - Adds use_weightless_cache_ flag to ModelCompilationOptions - Sets ep.enable_weightless=1 in session config options - Registered in ort_compile_api table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the app requests weightless mode and the EP confirms support via GetWeightlessSupport(), ORT automatically overrides drop_constant_initializers to false in GetCapability(). This ensures constant initializers remain as fused node inputs so the EP can access them via KernelContext_GetInput() at Compute() time. The override is logged at INFO level for diagnostics. Constructor queries the EP's weightless support at creation time and stores the result in weightless_enabled_ for use during GetCapability(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add kOrtEpDevice_EpMetadataKey_WeightlessSupport to onnxruntime_ep_device_ep_metadata_keys.h. EPs set this during GetSupportedDevices() so apps can query device-specific weightless capability before compilation via EpDevice_EpMetadata(). Valid values: 'none', 'external_only', 'all'. Update all new weightless APIs from version 1.28 to 1.29: - Session options: ep.enable_weightless, ep.context_source_model_path - OrtEpApi: OrtWeightlessSupport enum, GetWeightlessSupport - OrtApi: SessionOptionsSetWeightlessSourceModelPath, SessionOptionsSetWeightlessSourceModelFromBuffer - OrtCompileApi: ModelCompilationOptions_SetWeightlessCache - Version gate in PluginExecutionProvider constructor Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
javier-intel
reviewed
Jul 8, 2026
| // When weightless mode is enabled, override drop_constant_initializers to false so that ORT keeps | ||
| // constant initializers as fused node inputs. The EP can then access them via KernelContext_GetInput(). | ||
| bool drop_constant_initializers = node_grouping.fusion_options.drop_constant_initializers; | ||
| if (weightless_enabled_ && drop_constant_initializers) { |
Contributor
There was a problem hiding this comment.
A couple of comments:
- The override should only happen if the EP support level is all (2), otherwise if it's one this extends the initializers lifetime for no benefit; it's actually worse because EP would copy those to the compiled model. My suggestion is to make
weightless_enabled_of typeOrtWeightlessSupportand checkweightless_enabled_ >= OrtWeightlessSupport_ALL - Another thought is that EPs handle initializers in different ways, most use
KernelContext_GetInput()but OV doesn't do that, instead it directly uses the initializer values. I guess that once phase 2 is done to reuse the initializers from Compile() in Compute() this code will not be necessary, is that correct?
| ORT_API_STATUS_IMPL(OrtApis::SessionOptionsSetWeightlessSourceModelPath, _Inout_ OrtSessionOptions* options, | ||
| _In_ const ORTCHAR_T* source_model_path) { | ||
| API_IMPL_BEGIN | ||
| if (source_model_path == nullptr || source_model_path[0] == '\0') { |
Contributor
There was a problem hiding this comment.
Should it also validate that the path is valid? Say with std::filesystem::exists(std::filesystem::path(source_model_path))
| // Experimental API | ||
| ORT_API(OrtExperimentalFnPtr, GetExperimentalFunction, _In_ const char* name); | ||
|
|
||
| ORT_API_STATUS_IMPL(KernelContext_GetSyncStream, _In_ const OrtKernelContext* context, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Expand ORT's weightless mode to cover all initializers (internal and external), working in both JIT and AOT flows.
Problem
The existing ep.enable_weightless_ep_context_nodes session option is a string-based convention between the app and the EP with no ORT Core involvement. It only covers external initializers in the AOT flow, has no versioning or capability negotiation, and JIT flows cannot benefit.
Changes
Session options ( onnxruntime_session_options_config_keys.h ):
OrtEpApi ( onnxruntime_ep_c_api.h ):
OrtApi ( onnxruntime_c_api.h ):
OrtCompileApi ( onnxruntime_c_api.h ):
ORT Core behavior ( ep_plugin_provider_interfaces.cc ):
Future work
Motivation and Context