You can't solve computer use by ignoring the interface

juancn•about 1 hour ago

You lost me at the annoying trail on mouse movement.

You advocate for good UI and immediately make some crap that makes it hard to read/use your site. It's kind of ironic.

Please stop messing with: selection, mouse cursor, copy/paste, etc.

bluGill•18 minutes ago

Lost me even before I realized what was going on. My computer is too slow to handle the tricks they are pulling. I think you are right, but their "cool" things were too slow and made a mess of my screen without having any relation to what I am doing now (because I did it seconds ago)

bellowsgulch•17 minutes ago

They work on real-time computer use input. One of the areas in this space, mouse movement, is sophisticated enough there are people with PhDs working on the problem of human-like mouse movement, which requires you to record time-series (trailing) data of physical mouse movement based on human hand, wrist, and arm movement and replicating position, acceleration, and velocity data.

It's also just a mouse trail, get over it. It's not an interesting comment.

kvark•33 minutes ago

We have 2 paths here:

- either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls . Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates.

- or make AI systems use the screen like humans do - what the article is arguing for

Both of these paths are possible. One of them is much more efficient than the other in the long term. Another one may be inevitable.

winddude•16 minutes ago

> - either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls . Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates.

Yea, hasn't worked so far, devs and orgs have been pushing for APIs ages, same with structured data, eg schema.org

The third option is agent to agent communication, mcp, etc.

gandreani•40 minutes ago

Meta: I like that the first comments are not about the model at all but the website's mouse trail. One is in favor and one is against. Sums up society pretty well hahaha.

On the model. Really cool that it can act every 70ms. That's more than 10x faster than TTFT in most inference APIs. It HAS to be a local model. No idea which family or which one it is or what hardware it runs in. These would be nice details to tease!

nvch•42 minutes ago

Looking how much agents like to use and push to have functional a11y trees, we may accidentally solve accessibility as well

dzonga•about 2 hours ago

that's what a lot of 'a.i' companies miss - unless you've formerly QA devs deployed as Forward Automation Engineers - writing scripts to automate most tasks done by humans in a service economy - by the time the a.i bubble busts it will all about the promise about a.i to transform the economy while it was near achieved.

browser automation, RPA etc can bring us closer to agentic workflows since most software doesn't provide an API.

euphetar•about 1 hour ago

Agreed and a lot of companies seem to have their head in the sand, just hoping AI agents will magic their way through the legacy

But I don't think RPA is the way. Or at least, not in the current state. I think AI agents with proper computer use are what RPA should be. Usual RPA is too brittle and maintaining it often requires more work than just doing the work

speak_plainly•about 1 hour ago

The little pencil effect is a nice touch.

luciana1u•27 minutes ago

every decade someone rediscovers that the bottleneck was never the model, it was the mouse. we built APIs to avoid this exact problem and then asked the AI to ignore them and click buttons like an intern

You can't solve computer use by ignoring the interface

⚡ Community Insights

Discussion (11 Comments)Read Original on HackerNews