Skip to content

Adopt Foundation Models OS 27 surfaces: image attachments, capabilities, context size - #206

Open
james-333i wants to merge 2 commits into
huggingface:mainfrom
james-333i:feat/foundation-models-os27
Open

Adopt Foundation Models OS 27 surfaces: image attachments, capabilities, context size#206
james-333i wants to merge 2 commits into
huggingface:mainfrom
james-333i:feat/foundation-models-os27

Conversation

@james-333i

Copy link
Copy Markdown
Contributor

This updates the Foundation Models adapter for OS 27 while keeping the package building unchanged on Xcode 26.

Image input: Transcript image segments were previously dropped when converting to a FoundationModels transcript. On OS 27 they now map to Transcript.Segment.attachment with an ImageAttachment, built directly for URL sources and via ImageIO decoding for data sources. Verified end to end on an iPhone running iOS 27 (image prompt answered by the on-device model).

Capabilities and context: SystemLanguageModel now exposes contextSize (back-deployed by Apple to OS 26) and supportsImageInput, which reports the OS 27 vision capability and returns false on earlier OS versions.

Generation options: the adapter now sets maximumResponseTokens on FoundationModels.GenerationOptions. The property has existed since OS 26 but was never assigned, so response token limits were silently ignored on the Foundation Models backend. Verified on device: a capped request now terminates at the limit.

The OS 27 references are behind #if compiler(>=6.4) with runtime availability checks, so Xcode 26 builds still compile and the deployment floor is unchanged

@mattt

mattt commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Hi @james-333i. Thank you for this. The maximumResponseTokens fix alone is worth having, and the image-attachment mapping is a nice preview of what's coming.

Here's what I'm thinking for the project overall (details in #210):

  • Cut a 1.0 soon against the current (OS 26) Foundation Models API, with some cleanup of places where we've grown beyond it.
  • Start a 2.0 for the OS 27 API, which I expect to be source-breaking in a few places (session construction, image inputs, generation options).

Since everything here is availability-gated and doesn't change the OS 26 surface, I'm inclined to take it now rather than hold it for 2.0. Two small things before I merge: can you split out the maximumResponseTokens fix as its own commit so it's easy to cherry-pick, and let me look at how contextSize / supportsImageInput line up with what we do for the LanguageModel protocol more broadly?

Does that sound reasonable?

@james-333i
james-333i force-pushed the feat/foundation-models-os27 branch from bc12b2c to 4750a4f Compare September 2, 2026 15:58
@james-333i

Copy link
Copy Markdown
Contributor Author

@mattt Sounds good. I've split the maximumResponseTokens fix into its own commit so it cherry-picks cleanly. On contextSize and supportsImageInput, I kept both as passthroughs on SystemLanguageModel since the LanguageModel protocol has no capability surface today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants