Computer Use is 45x more expensive than structured APIs

Hang on, that sounds like common corporate SaaS apps.

The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility.

The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab`

Why accessibility? Well, turns out that it's just a good DOM in general. It's structure for apps. Not all apps implement it perfectly, but enough do to make it wildly useful.

[1] https://getinvoke.com - note that the landing page is targeted towards creatives right now and doesn't talk about this use case yet

If one agent just explores the UI, maybe in a test environment, and outputs a somewhat-structured description of the various UI elements and their behavior, then another agent was given that description, would the other agent perform better that an agent that both explores the UI and tries to accomplish the given task at the same time?

With an example UI I made up, the description (API-like interface definition) could be something like:

  Get all reviews:

  To get all the reviews you need to go to each page and click "show full review" for every review summary in that page.

  Go to each page:

  Start at page 1 (the default when in the Reviews tab). Continue by clicking the "next" button until the "next" button is no longer available (as you've reached the last page).

Or am I misunderstanding the article completely? Probably. But it's interesting nonetheless. Sorry if it makes no sense.

To me the browser is a translation layer. Working on the browser directly while hard enables big advantages on compatibility. The only thing I miss as of now which is on the todo is ocr of the images in the browser into text out. But an api would need to do that anyways to work.

The main loss in my view of pure API based is, where do you get the data? We won't replicate human work without seeing that done. Humans work in the UI that's it. Computer use to me is the promise of being able to replicate end to end actions a human does. API can do that in theory but the data to do that is also near impossible to collect properly.

Recently, I asked Claude to log into my local grocery store chain's website and add all of the items from my shopping list to a cart. It was hilariously slow, but it did get the job done.

Unless I missed it, the article doesn't explictly mention speed in the copy, but the results do show a 17 minute (!!!) total time for the vision agent vs. 0.5s - 2.8s for the API approach.

A big part of the challenge with vision is that to manipulate the DOM, you first have to be sure the entire (current) DOM is loaded. In my experience this ends up in adding a lot of artificial waits for certain elements to exist on the page.

The problem is that not everything from the 'past' can be accessed via APIs. It would be a fun time - remember Prism [1] - I would just run that and get all the API calls in a nice format and then replay them over and over to do things in succession.

In the new world, we have access to OpenAPI.json and whatnot, but in the world where things were built in the days pre-OpenAPI and pre-specs and best practices...I am not so sure! (and a lot of world lives then)

Alas, this works for a good chunk of things but not everything. Which is why the other technnology exists.

[1] https://stoplight.io/open-source/prism

Apps written now will have mcp servers / AI compatibility when relevant

The issue that still needs solving is how to make llms interact with everything we already have and use (efficiently, not with screenshot, read, screenshot, ...)

Most of the time that means reverse engineering, either the app itself or the APIs it uses

From github (not my projects):

https://github.com/SimoneAvogadro/android-reverse-engineerin... => reverse engineer android app APIs from APKs

https://github.com/HKUDS/CLI-Anything => convert ooen-source GUI apps to clis

https://github.com/kalil0321/reverse-api-engineer => API reverse engineering from traffic (claude skills)

My take at the same issue (very young project):

Also api reverse engineering from traffic captures, with a focus on mobile app, safety & community mcp generation

https://getspectral.sh

https://github.com/spectral-mcp/spectral

In particular, the vision-based approach used in the evaluation has clear limitations with regard to efficiency due to its nature (small observation window, heterogeneous modality)

At Smooth we use an hybrid DOM/vision approach and we index very strongly on small models. An interesting fact is that UIs are generally designed to minimize ambiguity and supply all and only the necessary context as token-efficient as possible, and the UX is cabled up to abstract the APIs in well-understood interface patterns e.g. dropdowns or autocompletes. This makes navigation easier and that's why small models can do it, which is another dimension that must be considered

We typically recommend using APIs/MCP where available and well designed, but it's genuinely surprising how token-efficient agentic browser navigation can actually be

I think OpenAI designing their own phone is the next logical step. I hope they succeed which should bring major competition to Apple and Android.

The only reason you wouldn’t choose an API is if it wasn’t viable.

I embedded a Google Calendar widget on my Book a demo page, I don't know the API and Google doesn't expose/maintain one either.

What we are doing at Retriever AI is to instead reverse engineer the website APIs on the fly and call them directly from within the webpage so that auth/session tokens propoagate for free: https://www.rtrvr.ai/blog/ai-subroutines-zero-token-determin...

idk.. not really thought out too much, but has to be better

I can see the appeal in pixel route given universality but wow that seems ugly on efficiency

> To make the comparison apples-to-apples, we rewrote the vision prompt as an explicit UI walkthrough, naming the sidebar items, tabs, and form fields the agent should interact with at each step. Fourteen numbered instructions covering the navigation the agent had failed to figure out on its own.

This is a model problem, though. Because the model failed to understand it could scroll, you forced it to consume multiples of the tokens. Could you come up with an alternative here?

Do you know what the vision model was trained on? Because often people see “vision model” and think “human-level GUI navigator” when afaik the latter has yet to be built.

I don't think any new app should ever be specifically designed for AI to interact with them through computer use

There are usecases where the vision agent is the more obvious, or only choice though, e.g. prorprietary/locked-down desktop apps that lack an automation layer.

1. https://github.com/SawyerHood/dev-browser

Me: hmm, this title confuses and infuriates Rob.

[Clicks link]

Me: Sees same title, repeat feelings of confusion and infuration

[Scrolls article down on my smartphone]

Me: Sees jpg with the same title, repeat feelings of co fusion and infuriation.

[Closes tab]

[Continues living rest of my life]

I hope this feedback is well received and understood.

The problem is, all the tasks are essentially: a) things agents probably just can't do, and b) things that absolutely cannot afford to be hallucinated or otherwise fucked up. So far the tasks I've thought of:

- Taxes. So it needs a lot of sensitive information to get W2's. Since I have to look up a lot of this stuff in the physical world anyway, it's not like I can just let it run wild.

- Background check for a new job. It took me 3 hrs to fill out one of them (mostly because the website was THAT bad). Being myself, I already was making mistakes just forgetting things like move in dates from 10 years ago, and having to do a lot of searching in my email for random documents. No way I'm trusting an agent with this.

- Setting up an LLC. Nope nope nope. There's a lot of annoying work involved with this, but I'm not trusting an LLM to do this.

Anyway, I guess my point is that even if an LLM was good at using my computer (so far, it seems like it wouldn't be), the kind of things I'd want an agent for are things that an LLM can't be trusted with.

When you think of everything it takes for an AI to use what the article calls a "vision agent" then it seems as if using a purpose-made API ought to be MANY orders of magnitude faster.

Electron uses 10x more RAM than regular apps. But it's so convenient.

Python is 100x slower than C. It's in the top 3 of languages now.

Worse but more convenient always wins.

Using CLI tools is much faster and token-efficient. I developed ten apps in the last two months. One reached 10,000+ monthly active users.

I ask Codex to generate SVG line by line and backtrack edit, ask it to use Inkscape to generate icons, etc...

I developed all this on $20 codex sub.