[Fix] Fix cua tools - #40
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR: Align the CUA execution stack with the real cua-driver 0.19.3 wire contract
🎯 Wire Contract Alignment
ax.tree→get_window_state{pid, window_id}(was sending an invalidappfield); AX actions map via a data table to the discreteclick/double_click/right_click/type_text/set_valuetools withx/ypixel args; shortcuts parse intohotkey{keys: [...]}arrays (bare keys route topress_key); scroll speaksdirection/amount;app.activate→bring_to_frontby pid (launch_appis backgrounded by design); untargeted input gainsscope:"desktop"; screenshot routes toget_desktop_state/get_window_state(the standalone tool no longer exists); reverse-DNS bundle ids are recognized and the retiredpathfield is dropped.ok: Trueenvelope (errors still raise structuredRpcError); image blocks andeffect/escalationverdicts survive unwrapping; local clipboard returns the{text, change_count, change_ts}port contract instead of a bare string.🪟 Addressing Model: list_windows → pid/window_id → element_index
list_windowstool (Methods.AX_LIST): the model discovers window targets itself — no hidden app→pid resolution.observe_ui(pid, window_id, query?)snapshots one window with indexed elements; action tools address byelement_index, translated to the driver's staleness-checkedelement_token. Stale/missing snapshots return structured guidance instead of failing opaquely.switch_apptakes pid/windows from the launch response, activates viabring_to_front, verifies against that exact window.📦 Flat UISnapshot Replaces the UINode Tree
parent_index, noactions, norootkey) — the tree parser returned zero elements against a real driver while the mock's invented tree kept tests green. FrozenUIElement/UISnapshotnow mirror the records verbatim (enabled,selected, tokens).ui_selector.py,ui_summarizer.py, selector TTL caches, role vocabularies, and theAXScrollAreaheuristic (targetless scroll uses the driver's focused-scroller path).elements_complete,capture_coverage— e.g. browser page content invisible in window scope) now reach the model with recovery guidance.🔐 Safety Gate Restored
SendActionRule(the send-confirmation gate matched the retiredselectorparam). NewToolBridge.register(describer=…)hook resolveselement_index→ "Button 发送" before policy evaluation; click/right_click wired, both directions tested.🖥️ Platform Fixes
-Commandre-parses argv — any text with spaces brokeSet-Clipboard); reads useGet-Clipboard -Raw+ UTF-8 with the one appended newline stripped — byte-faithful round-trip incl. CJK and multi-line.exec_shellruns locally with a configurable timeout (it was sent to theclicktool — the driver has no shell tool); screenshots land on disk viascreenshot_out_file; undo backups use the platform tempdir instead of/tmp..gitattributesenforces LF repository-wide.🧪 Tests
test_darwin_adapter.pycovers all execution methods, including delete→undo restore and a notepad type/copy/scroll round-trip verified through the clipboard (gated byLEAPFLOW_TEST_INTERACTIVE=1).