Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion BLURTENGINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ func setTargetApp(_ app: NSRunningApplication?) async
func insert(_ text: String, after priorText: String?, windowTitle: String?) async throws
```

`KeyInjector.insert` **always** pastes: it saves the current pasteboard, writes the transcript, activates the captured target app, posts a synthesized ⌘V, waits for the target to read the clipboard (`pasteSettleDuration`, default 400 ms, tunable in the initializer), then restores the prior pasteboard contents. That restore never destroys what it can't put back: an unreadable pasteboard snapshots as `nil` (not as empty) and is skipped, and the replacement items are built before `clearContents()`, with the plain-string flavor as a floor when promised representations can't be materialized. There is no keystroke-by-keystroke typing path and no length threshold. If the target app is gone or nothing editable is focused it leaves the text on the clipboard and throws `.targetAppLost` / `.noEditableTarget` — which the session turns into the quiet `.noTarget` outcome. `priorText` (the text before the caret, captured at press time) drives `withLeadingSeparator`, which joins consecutive dictations with a space so they don't run together. When `priorText` is unreadable (an Accessibility-opaque editor, or a browser tab like Google Docs whose canvas-rendered body exposes no AX text), `separatorBasis` falls back to what was last pasted — but only when both the target app **and** `windowTitle` match the last successful insert, so the fallback tracks "the same window," not just "the same process" (a browser hosts many unrelated tabs/documents under one PID).
`KeyInjector.insert` **always** pastes: it saves the current pasteboard, writes the transcript, activates the captured target app, posts a synthesized ⌘V, waits for the target to read the clipboard (`pasteSettleDuration`, default 400 ms, tunable in the initializer), then restores the prior pasteboard contents. That restore never destroys what it can't put back: an unreadable pasteboard snapshots as `nil` (not as empty) and is skipped, and the replacement items are built before `clearContents()`, with the plain-string flavor as a floor when promised representations can't be materialized. There is no keystroke-by-keystroke typing path and no length threshold. If the target app is gone or nothing editable is focused it leaves the text on the clipboard and throws `.targetAppLost` / `.noEditableTarget` — which the session turns into the quiet `.noTarget` outcome. AX-opaque targets — Electron editors and web browsers, per `FocusCapture.isAXOpaqueApp` — are exempt from the editability gate and are pasted into even with no editable AX signal: web content is routinely invisible to Accessibility (Chromium builds its tree lazily; `contenteditable` composers expose no editable role), so "no signal" there usually means "AX can't see the field," not "no field," and the accepted trade-off is a rare beep over dropping the user's words to copy-only. `priorText` (the text before the caret, captured at press time) drives `withLeadingSeparator`, which joins consecutive dictations with a space so they don't run together. When `priorText` is unreadable (an Accessibility-opaque editor, or a browser tab like Google Docs whose canvas-rendered body exposes no AX text), `separatorBasis` falls back to what was last pasted — but only when both the target app **and** `windowTitle` match the last successful insert, so the fallback tracks "the same window," not just "the same process" (a browser hosts many unrelated tabs/documents under one PID).

The session calls `setTargetApp` at press time with the app that was frontmost when recording started — so the paste lands where the user was, even if focus moved during transcription.

Expand Down
71 changes: 59 additions & 12 deletions Sources/BlurtEngine/FocusCapture/FocusCapture+Editability.swift
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,11 @@ extension FocusCapture {
/// no readable role — is treated as not editable, so we copy rather than beep a
/// ⌘V into a target that can't take it.
///
/// AX-opaque Electron editors (VS Code, Slack) expose *none* of these signals
/// even for a genuine text field, so this returns false for them too — but the
/// injector still pastes into those via a separate Electron-app check (see
/// `isElectronApp` / `KeyInjector.insert`), so the user's words aren't dropped
/// to copy-only there.
/// AX-opaque apps — Electron editors (VS Code, Slack) and web browsers — can
/// expose *none* of these signals even for a genuine text field, so this
/// returns false for them too. The injector still pastes into those via a
/// separate app-identity check (see `isAXOpaqueApp` / `KeyInjector.insert`),
/// so the user's words aren't dropped to copy-only there.
static func isEditableTarget(role: String?, valueSettable: Bool, hasInsertionPoint: Bool) -> Bool {
if let role, editableRoles.contains(role) { return true }
return valueSettable || hasInsertionPoint
Expand All @@ -41,17 +41,63 @@ extension FocusCapture {
/// Whether `app` is an Electron/Chromium-based app, detected by the bundled
/// Electron framework. Such apps ship with their accessibility tree off, so even
/// a focused text field exposes no editable AX signal and
/// `hasEditableFocusedElement` reads them as non-editable. They're the one case
/// the injector still pastes into on no signal (dropping the user's words into a
/// copy-only fallback would be the worse mistake). A native app with genuinely no
/// editable focus bundles no such framework and correctly falls back to copy.
/// `hasEditableFocusedElement` reads them as non-editable. A native app with
/// genuinely no editable focus bundles no such framework and correctly falls
/// back to copy.
static func isElectronApp(_ app: NSRunningApplication?) -> Bool {
guard let bundleURL = app?.bundleURL else { return false }
let electronFramework = bundleURL.appendingPathComponent(
"Contents/Frameworks/Electron Framework.framework")
return FileManager.default.fileExists(atPath: electronFramework.path)
}

/// Bundle-identifier prefixes of known web browsers. Prefix-matched so channel
/// variants classify with their stable siblings (`com.google.Chrome.beta`,
/// `com.apple.SafariTechnologyPreview`).
private static let browserBundleIDPrefixes: [String] = [
"com.apple.Safari", // Safari + Safari Technology Preview
"com.google.Chrome", // Chrome + Beta/Dev/Canary
"org.chromium.Chromium",
"com.microsoft.edgemac", // Edge + Beta/Dev/Canary
"com.brave.Browser", // Brave + Beta/Nightly
"com.operasoftware.Opera",
"com.vivaldi.Vivaldi",
"company.thebrowser.Browser", // Arc
"org.mozilla.firefox",
"com.duckduckgo.macos.browser",
"com.kagi.kagimacOS", // Orion
]

/// Pure decision behind `isBrowserApp`: does this bundle identifier belong to a
/// known browser? Split from the `NSRunningApplication` wrapper so the
/// classification is unit-testable without live running apps.
static func isBrowserBundleID(_ bundleID: String?) -> Bool {
guard let bundleID else { return false }
return browserBundleIDPrefixes.contains { bundleID.hasPrefix($0) }
}

/// Whether `app` is a known web browser. Web content is AX-opaque in practice:
/// Chromium builds its accessibility tree lazily (the first query after launch
/// resolves only a bare `AXWebArea` with no editable signal), and even with the
/// tree live, a `contenteditable` composer (ChatGPT's ProseMirror field) can
/// surface as a generic group with no settable value. So "no editable signal"
/// in a browser usually means "AX can't see the field," not "no field."
static func isBrowserApp(_ app: NSRunningApplication?) -> Bool {
isBrowserBundleID(app?.bundleIdentifier)
}

/// Whether `app` is AX-opaque — an Electron editor or a web browser — where a
/// focused text field can expose no editable AX signal at all. These are the
/// one case the injector still pastes into on no signal: dropping the user's
/// words into a copy-only fallback there would be the worse mistake. The
/// accepted trade-off is a rare ⌘V beep when such an app truly has nothing
/// editable focused.
static func isAXOpaqueApp(_ app: NSRunningApplication?) -> Bool {
// Browser first: it's a string prefix check, whereas isElectronApp probes
// the disk (FileManager.fileExists) — skip that I/O for the common case.
isBrowserApp(app) || isElectronApp(app)
}

/// Whether the system-wide focused element can accept pasted text right now.
/// Read by `KeyInjector` (off the main actor, after it has activated the target
/// app) just before pasting — the Accessibility *client* read APIs are
Expand All @@ -66,9 +112,10 @@ extension FocusCapture {
// AX is trusted but reports no focused element — e.g. a native app frontmost
// with nothing editable focused (Finder, the desktop, a button-only window).
// Posting ⌘V there only beeps, so treat it as non-editable and copy instead.
// AX-opaque Electron apps (VS Code, Slack) also expose no focused element
// here, but the injector's Electron-app check still pastes into those (see
// `KeyInjector.insert` / `isElectronApp`).
// AX-opaque apps (Electron editors like VS Code/Slack, and browsers before
// Chromium's lazy accessibility tree is built) also expose no focused
// element here, but the injector's app-identity check still pastes into
// those (see `KeyInjector.insert` / `isAXOpaqueApp`).
return false
}

Expand Down
29 changes: 16 additions & 13 deletions Sources/BlurtEngine/Injection/KeyInjector.swift
Original file line number Diff line number Diff line change
Expand Up @@ -88,13 +88,15 @@ public actor KeyInjector: InjectorProtocol {
/// host's live focus (defaults to "editable" there).
private let hasEditableTarget: @Sendable () -> Bool

/// Whether the captured target app is an AX-opaque Electron/Chromium editor
/// (VS Code, Slack), which exposes no editable AX signal even for a real text
/// field. When `hasEditableTarget` reads false but this is true, we still paste
/// rather than copy — dropping the user's words into an Electron editor they're
/// clearly typing in would be the worse mistake. Injectable so tests don't
/// depend on which apps are installed (defaults to "not Electron").
private let isAXOpaqueEditor: @Sendable (NSRunningApplication?) -> Bool
/// Whether the captured target app is AX-opaque — an Electron/Chromium editor
/// (VS Code, Slack) or a web browser — which can expose no editable AX signal
/// even for a real text field (Chromium builds its accessibility tree lazily,
/// and `contenteditable` composers like ChatGPT's surface no editable role).
/// When `hasEditableTarget` reads false but this is true, we still paste
/// rather than copy — dropping the user's words into a field they're clearly
/// typing in would be the worse mistake. Injectable so tests don't depend on
/// which apps are installed (defaults to "not opaque").
private let isAXOpaqueApp: @Sendable (NSRunningApplication?) -> Bool

/// The pasteboard the paste reads, writes, and restores. Behind a seam so
/// tests exercise the save/restore + changeCount logic against an in-memory
Expand All @@ -111,7 +113,7 @@ public actor KeyInjector: InjectorProtocol {
waitForTargetActivation: KeyInjector.waitUntilFrontmost,
isAccessibilityTrusted: KeyInjector.accessibilityTrusted,
hasEditableTarget: FocusCapture.hasEditableFocusedElement,
isAXOpaqueEditor: FocusCapture.isElectronApp)
isAXOpaqueApp: FocusCapture.isAXOpaqueApp)
}

init(
Expand All @@ -121,7 +123,7 @@ public actor KeyInjector: InjectorProtocol {
waitForTargetActivation: @escaping @Sendable (NSRunningApplication) async -> Bool = { _ in true },
isAccessibilityTrusted: @escaping @Sendable () -> Bool = { true },
hasEditableTarget: @escaping @Sendable () -> Bool = { true },
isAXOpaqueEditor: @escaping @Sendable (NSRunningApplication?) -> Bool = { _ in false },
isAXOpaqueApp: @escaping @Sendable (NSRunningApplication?) -> Bool = { _ in false },
clipboard: any ClipboardAccess = SystemClipboard()
) {
self.pasteSettleDuration = pasteSettleDuration
Expand All @@ -130,7 +132,7 @@ public actor KeyInjector: InjectorProtocol {
self.waitForTargetActivation = waitForTargetActivation
self.isAccessibilityTrusted = isAccessibilityTrusted
self.hasEditableTarget = hasEditableTarget
self.isAXOpaqueEditor = isAXOpaqueEditor
self.isAXOpaqueApp = isAXOpaqueApp
self.clipboard = clipboard
}

Expand Down Expand Up @@ -232,9 +234,10 @@ public actor KeyInjector: InjectorProtocol {
// a synthesized ⌘V would just make macOS beep. Leave the transcript on the
// clipboard so the user can paste it by hand, and signal the pipeline to show
// a quiet "copied" notice instead of typing. The exception is an AX-opaque
// Electron editor (VS Code, Slack), which reports no editable signal even for
// a real text field — there we still paste rather than drop the user's words.
guard hasEditableTarget() || isAXOpaqueEditor(target) else {
// app — an Electron editor (VS Code, Slack) or a web browser — which can
// report no editable signal even for a real text field; there we still paste
// rather than drop the user's words.
guard hasEditableTarget() || isAXOpaqueApp(target) else {
clipboard.write(finalText)
throw BlurtError.noEditableTarget
}
Expand Down
62 changes: 62 additions & 0 deletions Tests/BlurtEngineTests/BrowserBundleIDTests.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
import Testing

@testable import BlurtEngine

/// Pins the browser classification behind the injector's AX-opaque exemption
/// (see `FocusCapture.isAXOpaqueApp`): a known browser pastes even when the
/// focused element exposes no editable AX signal, because web content is
/// routinely opaque (Chromium's lazy accessibility tree, `contenteditable`
/// composers like ChatGPT's) — "no signal" there means "AX can't see the
/// field," not "no field."
@Suite("FocusCapture.isBrowserBundleID")
struct BrowserBundleIDTests {
@Test(
Comment thread
alexkroman marked this conversation as resolved.
"known browser bundle IDs classify as browsers",
arguments: [
"com.apple.Safari",
"com.google.Chrome",
"org.chromium.Chromium",
"com.microsoft.edgemac",
"com.brave.Browser",
"com.operasoftware.Opera",
"com.vivaldi.Vivaldi",
"company.thebrowser.Browser",
"org.mozilla.firefox",
"com.duckduckgo.macos.browser",
"com.kagi.kagimacOS",
])
func knownBrowsers(bundleID: String) {
#expect(FocusCapture.isBrowserBundleID(bundleID))
}

@Test(
"channel variants classify with their stable siblings (prefix match)",
arguments: [
"com.apple.SafariTechnologyPreview",
"com.google.Chrome.beta",
"com.google.Chrome.canary",
"com.microsoft.edgemac.Dev",
"com.brave.Browser.nightly",
])
func channelVariants(bundleID: String) {
#expect(FocusCapture.isBrowserBundleID(bundleID))
}

@Test(
"non-browser apps are not browsers — they keep the copy-don't-beep fallback",
arguments: [
"com.apple.finder",
"com.apple.TextEdit",
"com.apple.dt.Xcode",
"com.microsoft.VSCode", // Electron: exempted by isElectronApp, not here
"com.googlecode.iterm2", // "com.google" lookalike must not prefix-match
])
func nonBrowsers(bundleID: String) {
#expect(!FocusCapture.isBrowserBundleID(bundleID))
}

@Test("a nil bundle ID is not a browser")
func nilBundleID() {
#expect(!FocusCapture.isBrowserBundleID(nil))
}
}
5 changes: 3 additions & 2 deletions Tests/BlurtEngineTests/EditableTargetTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,9 @@ struct EditableTargetTests {
func unknownRoleWithoutSignalCopies() {
// A focused element that reports an unrecognized role and exposes no settable
// value or insertion point isn't a text target — copy rather than beep a ⌘V
// into it. (AX-opaque Electron editors, which also land here, are pasted into
// via the injector's separate Electron-app check, not this signal test.)
// into it. (AX-opaque apps — Electron editors and browsers — also land here,
// but are pasted into via the injector's separate app-identity check, not
// this signal test.)
#expect(
!FocusCapture.isEditableTarget(
role: "AXWebArea", valueSettable: false, hasInsertionPoint: false))
Expand Down
2 changes: 1 addition & 1 deletion Tests/BlurtEngineTests/KeyInjectorInsertTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ struct KeyInjectorInsertTests {
return true
},
hasEditableTarget: { false }, // Electron/Chromium exposes no editable AX signal
isAXOpaqueEditor: { _ in true }, // …but it *is* an Electron editor
isAXOpaqueApp: { _ in true }, // …but it *is* an AX-opaque app (Electron editor)
clipboard: clip)

// Must not throw noEditableTarget: the Electron exception keeps the paste.
Expand Down