Skip to content

[Fix] Fix cua tools - #40

Merged
wangxingjun778 merged 6 commits into
modelscope:mainfrom
fanqiNO1:fix_cua
Aug 14, 2026
Merged

[Fix] Fix cua tools#40
wangxingjun778 merged 6 commits into
modelscope:mainfrom
fanqiNO1:fix_cua

Conversation

@fanqiNO1

@fanqiNO1 fanqiNO1 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

PR: Align the CUA execution stack with the real cua-driver 0.19.3 wire contract

🎯 Wire Contract Alignment

  • Request side: ax.treeget_window_state{pid, window_id} (was sending an invalid app field); AX actions map via a data table to the discrete click/double_click/right_click/type_text/set_value tools with x/y pixel args; shortcuts parse into hotkey{keys: [...]} arrays (bare keys route to press_key); scroll speaks direction/amount; app.activatebring_to_front by pid (launch_app is backgrounded by design); untargeted input gains scope:"desktop"; screenshot routes to get_desktop_state/get_window_state (the standalone tool no longer exists); reverse-DNS bundle ids are recognized and the retired path field is dropped.
  • Response side: dict payloads gain a uniform ok: True envelope (errors still raise structured RpcError); image blocks and effect/escalation verdicts survive unwrapping; local clipboard returns the {text, change_count, change_ts} port contract instead of a bare string.

🪟 Addressing Model: list_windows → pid/window_id → element_index

  • New list_windows tool (Methods.AX_LIST): the model discovers window targets itself — no hidden app→pid resolution.
  • observe_ui(pid, window_id, query?) snapshots one window with indexed elements; action tools address by element_index, translated to the driver's staleness-checked element_token. Stale/missing snapshots return structured guidance instead of failing opaquely.
  • switch_app takes pid/windows from the launch response, activates via bring_to_front, verifies against that exact window.

📦 Flat UISnapshot Replaces the UINode Tree

  • Real payloads are flat element arrays (sparse parent_index, no actions, no root key) — the tree parser returned zero elements against a real driver while the mock's invented tree kept tests green. Frozen UIElement/UISnapshot now mirror the records verbatim (enabled, selected, tokens).
  • Retired: ui_selector.py, ui_summarizer.py, selector TTL caches, role vocabularies, and the AXScrollArea heuristic (targetless scroll uses the driver's focused-scroller path).
  • Driver blind-spot statements (elements_complete, capture_coverage — e.g. browser page content invisible in window scope) now reach the model with recovery guidance.

🔐 Safety Gate Restored

  • Index addressing had silently disabled SendActionRule (the send-confirmation gate matched the retired selector param). New ToolBridge.register(describer=…) hook resolves element_index → "Button 发送" before policy evaluation; click/right_click wired, both directions tested.

🖥️ Platform Fixes

  • Windows clipboard: text travels as base64 (PowerShell -Command re-parses argv — any text with spaces broke Set-Clipboard); reads use Get-Clipboard -Raw + UTF-8 with the one appended newline stripped — byte-faithful round-trip incl. CJK and multi-line.
  • exec_shell runs locally with a configurable timeout (it was sent to the click tool — the driver has no shell tool); screenshots land on disk via screenshot_out_file; undo backups use the platform tempdir instead of /tmp.
  • .gitattributes enforces LF repository-wide.

🧪 Tests

  • Mocks rewritten isomorphic to real payloads (flat records, launch returning pid/windows), closing the "mock green, real broken" drift. Full mock layer: 1714 passing.
  • Unit suites lock the 0.19.3 mapping, index addressing, and response shapes; real-driver test_darwin_adapter.py covers all execution methods, including delete→undo restore and a notepad type/copy/scroll round-trip verified through the clipboard (gated by LEAPFLOW_TEST_INTERACTIVE=1).

@wangxingjun778
wangxingjun778 merged commit 69a98ca into modelscope:main Aug 14, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants