DE version is available. Content is displayed in original English for accuracy.
The main problem: when a UI automation process controls a desktop app today, it usually takes over the human’s session. Your cursor moves, keyboard focus gets stolen, windows jump to the front, and you have to stop working until the agent is done. That is why we have historically avoided encouraging users to run these processes directly on their host machine, instead relying on VMs or GUI containers for concurrency and background execution.
But computer-use - the tools we give agents to operate computers like humans - does not scale cleanly that way. As models get smarter, agents need to share hosts safely, run in the background, and avoid collisions with the human or other agents using the same machine.
We realized macOS has no first-class API for "drive this app without touching the cursor". CGEventPost routes through the hardware input stream, so it moves your cursor. CGEvent.postToPid avoids the cursor warp, but Chromium treats those events as untrusted and silently drops clicks at the renderer boundary. Activating the target app first raises the window and pulls focus, defeating the point of background execution.
Cua Driver is our attempt at a real fix: a background computer-use driver for macOS that lets an agent click, type, scroll, and read native apps while your cursor, frontmost app, and Space stay where they are. The default interface is a CLI, so it is easy to script or call from any coding agent shell.
Try it on macOS 14+:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-d...)"
The first internal use case was delegated demo recording. We ask Claude Code to drive an app while 'cua-driver recording start' captures the trajectory, screenshots, actions, and click markers. The result is an agent-generated product demo, Screen Studio inspired.
Other things we have used it for:
- Replacing Vercel’s agent-browser and other browser-use CLIs. With Claude Code and Cua Driver, you do not need Chrome DevTools Protocol at all.
- A dev-loop QA agent that reproduces a visual bug, edits code, rebuilds, and verifies the UI while my editor stays frontmost.
- Personal-assistant flows that use iMessage from Claude Code, Hermes, or other general-purpose agent CLIs.
- Pulling visual context from Chrome, Figma, Preview, or YouTube windows I am not looking at, without relying on their APIs.
What made this harder than expected:
- CGEventPost warps the cursor because it goes through the HID stream.
- CGEvent.postToPid does not warp the cursor, but Chromium drops it at the renderer IPC boundary.
- Activating the target first raises the window and can drag you across Spaces.
- Electron apps stop keeping useful AX trees alive when windows are occluded without a private remote-aware SPI.
The unlock was SkyLight. SLEventPostToPid is a sibling of the public per-PID call, but it travels through a WindowServer channel Chromium accepts as trusted. Pair it with yabai’s focus-without-raise pattern, plus an off-screen primer click at (-1, -1), and the click lands without the window ever raising.
One thing we learned: the right addressing mode depends on the app. Native macOS apps usually have rich AX trees, Chromium-family apps often need a hybrid of AX and screenshots, and apps like Blender or CAD tools may expose almost no useful AX surface. The mistake is defaulting to pixels everywhere - or defaulting to AX everywhere.
Long technical writeup: https://github.com/trycua/cua/blob/main/blog/inside-macos-wi...
I would like feedback from people building Mac automation, agent harnesses, or accessibility tooling. If it breaks on an macOS app you care about, that is useful data for us.
Discussion (22 Comments)Read Original on HackerNews
[1] https://news.ycombinator.com/item?id=47799128
If you want to use it directly as an automation framework, you can take a Swift dependency on 'CuaDriverCore': https://cua.ai/docs/cua-driver/guide/getting-started/swift-i...
My only criticism is enabling telemetry by default. I'm a fan of having people opt-in.
Ironically enough the opposite happens with opt-out telemetry, for the same reason: a lot of power users will turn off telemetry, thus you will never see their usage patterns and will have to infer them. Dogfooding helps.
You claim power users opt in to telemetry, and then immediately say power users opt out.
From that you get two situations.
Opt-in:
- Regular users: click all 'ok' through setup at lightning speed, no telemetry enabled.
- Most power users: consciously don't check the box to opt-in because of privacy worries.
- Big picture power users: consciously check the opt-in box given they trust you (because they want their usage patterns to be profiled and optimized for).
Opt-out:
- Regular users: click all 'ok' through setup at lightning speed, telemetry enabled.
- Most power users: consciously check the box to opt-out because of privacy worries.
- Big picture power users: consciously don't check the opt-out box given they trust you (because they want their usage patterns to be profiled and optimized for).
The overwhelming majority of people don't care about digital privacy because the cost is opaque to them.
Also, telemetry when done right isn't "spying". Again, it is anonymized and used to see, for example, where the hot paths and paper cuts in applications are.
One useful trick is to cua-driver 'launch_app' instead of the default 'open' or other osascript, since it can start the app without raising/focusing it, and the tests don't disturb your active desktop while they run
The audit trail question is interesting and I haven't seen it come up much. When an agent clicks through an ERP or edits a file, you've got logs, but how do you explain the "why" behind each decision to, say, a compliance team?
Curious if that's something you're thinking about or if it's too early.