Nvidia wants to put a watchdog chip next to every AI agent

> Put a human in the loop and you just end up bottlenecking it and throwing away any purported productivity gains. Auto mode doesn't matter either, it's trivial to trick and for the agent to break out.

Productivity gains are still enormous compared to what we used to do before agents. But, I know that people don't want to stop there.

I feel like we are missing many shades of grey in the middle.

Semi-automation (human in the loop) can still result in a dramatic uplift in productivity. You can't run a combine harvester 100% autonomous but that doesn't stop anyone from trying to get as close to that limit as possible.

Agreed- this is the same problem we have with trusted admins or devs who have elevated privileges on their networks. We have to trust that the admins won't use their power to steal company secrets or misuse company resources. If you don't trust the admins, then they can't fix things on your network and there is no point in having them.

If you want an agent to act on its own, like pushing to a git repo, managing dependencies, building and testing, etc., then you have to trust it as much as any other privileged user.

If you don't want to trust it, then you're just forcing yourself into the reverse centaur role, where the agent edits some code, but then has to stop and ask you to push the changes or build the software again and run the unit tests.

I suppose there is a principled way of doing things like "I trust you do do basic commits but I will handle merge conflicts" and "you can build modules in this directory but you can't build outside of it" but this is just a lot of effort that most orgs won't bother with.

An agent doesn't inherently need wide access to be useful. The most popular application for agents today is writing code. A coding agent needs write access to the source code and read/execute access to tools needed to build and test the code, but not much more. There is little added utility from giving coding agent access to things like ssh keys.

The hypothesis I've had in my head since OpenClaw has been the following and I haven't seen contradictory evidence yet. Agents have a fundamental unresolvable tension between usefulness, safety, alignment, and accuracy. You have to restrict access to ensure an agent acts safely because alignment and accuracy cannot be perfect. But restricting access makes the agent less useful. You can play with the sliding scale and get more and more granular with access restrictions but at some point you need to draw some line. And then finally, even access restrictions cannot be made perfect, so improvements to model accuracy without corresponding improvements to alignment make detailed access controls less useful.

In other words, better models need blunter access controls which negates whatever improvement in utility they provide.

> a new chip solves nothing

It does allow Nvidia to sell more chips. This is no genuine attempt to solve anything, imo.

I don't see why it needs wide unattended access. There's no getting around spending some human time on expressing your wishes and constraints, but we have choices about what form that takes. Markdown files and wide access seems to work, but so does custom handcuffs for each job. You just have to shift your guidance out of documentation and into interactive help, error messages, or other facets of the handcuffs (e.g. a custom CLI for this task which is the only way for the agent to act outside of its sandbox).

"Inherently needs wide unattended access"

And what if you could? What if you could give a space secure enough it could have direct control over your bank account. It may do something dumb but it's boundaries are beyond the agent.

It could use your routing number and run your gmail without risk of abusing the routing number.

There should be hope for some fields, right? Naively, I can imagine giving an airgapped model an offline copy of the web and once it cures a form of cancer, printing out the details for a researcher to verify.

Running untrusted workloads have been done at scale for a long time.

Every cloud provider dealt with it and concluded that virtual machine technology is an important part of that stack.

Couple it with the right observability, tooling I do think we can curb risks posed by agents.

Pretty sure the chip will need regular updates and therefore a subscription.

Generating even more revenue for Nvidia.

The bomber always gets through?

It is Custodians all the way son, you can’t fool me!

This feels suspiciously good for Nivida, yes.

I wonder how this will impact other chip manufacturers? What about people running local models on older hardware? Does this imply vendor lockout is coming in the future or is this restricted to datacenter hardware?

Both can be true at the same time.

This is like using "protect the children" as an argument for dragnet surveillance.

That a person can see such endless pages of people having various forms of disagreement, and yet somehow manage to conclude that this observed chaos constitutes a clear exhibition of cohesive groupthink is just...stunningly amazing to me.

I don't know why I find it so amazing since it happens with such regularity, but I'm always amazed by it anyway.

https://www.calcalistech.com/ctechnews/article/uwoyygmsu some might even say, something about known state sponsors of supply chain terrorist attacks being trusted to monitor every ai agent..

Airgap what network? How is it gonna order you a burrito on doordash without a network?

Or push to github?

Where is the contradiction? There is a trivial solution that does not impinge on our freedoms, so why on Earth would the existence of the trivial solution that could be used to avoid the tyrannical solution justify accepting the tyrannical solution?

You're only revealing your own inability to appreciate the nuance between these two situations.

So?