Gemini Robotics 2 brings whole body intelligence to robots

Google DeepMind was at the forefront of AI, and they are currently developing new concepts while others chase benchmark percentages.

And protein folding + Nobel Prize

It's only because Anthropic and OpenAI will mog Deepmind once they turn their attention to these things. Deepmind is good at shipping demos and benchmarks, Anthropic/OpenAI delightful products that are useful.

I wouldn't be surprised if Anthropic beats Demis/Isomorphic labs in Drug Discovery too leveraging alphafold/Jumper

For house keeping tasks the robotics companies love showing off... it honestly doesn't matter if it takes a robot longer than a human to clean your house. As long as it gets done before you get home from work, no big deal.

The time between gpt 2 and 3 was 15 months. The time between gemini robotics 1 and 2 was also 15 months: https://blog.google/products-and-platforms/products/gemini/h...

The difference between gpt 2 and 3 was insane. 2 could generate limericks when it wasn't repeating a word 300x. 3 could actually do some things. By comparison gemini robotics has hardly changed at all.

I will also point out that slow, non-fluid robotics is on a totally different level of difficulty from fast fluid motion. Asimov could walk pretty smoothly, but it didn't fall over because it used a very careful sequence that was never unbalanced; you could pause at any point without falling over. Move faster, like boston dynamics, and you need to account for the change in balance from your arms swinging... or rather, you need to be able to account for the rotational inertia etc from moving multiple masses along complex paths with multiple points of articulation at hundreds or thousands of times per second.

An algorithm to fold tshirts 90% of the time is easy. The cloth hangs down by gravity and you can just look for right angles (corners), find their coordinates with binocular matching, and move them to meet each other. Getting 99%, or folding them quickly, so that the fabric is actually moving instead of just hanging still- incredibly, incredibly more complex.

Gotta admit google is pretty ballsy for directly attacking Moravec's Paradox: https://en.wikipedia.org/wiki/Moravec%27s_paradox

They're fluid, but they are slow, which is highly likely to be a safety thing, which will have the side effect of making many motions more difficult, not less.

LLMs aren't themselves hurting anyone, no matter how much certain people like to pretend otherwise, whereas an AI in a robot with significant motors in absolutely can and will. There are reasons industrial ones live in safety cages after all.

Expecting them to be like inverse kinematic driven digital dolls is wrong because the optimization won't be for matching that but something like "net reduce energy consumption" which for electric motors in multi joint arms will look a bit odd.

These robots are unusable in their current form. Slow and expensive, humans perform this work better and cheaper. Looks like every other robot before it, a bunch of servos and no brain.

We don't get virtuous feedback loops in hardware though. Muscles are incredible engineering that took half a billion years. Token generation (speech) is low hundred thousands maybe. Unless of course improved AI navigates the space of ideas so well that it can give us alloys or synthetic flesh that does twitch as fast...

> If progress is as fast as LLMs

Is this the new "x technology is a year away" ?

I believe that it's also for safety, these robots would easily destroy themselves if not restrained (and cause damage).

yes at 1k tok/sec i think these robots will be fast and very fluid.

I work in this field, and I wrote my bachelor's thesis here, not with humanoids but with VLAs (think chatgpt connected to a robot arm)

It's certainly not there yet for anything practical, there's also certain bits and structures that don't have accurate names during construction, and it is important to keep that in mind - so a robot is unlikely to understand what it means to say "put the left bit of this box onto this right bit" due to ambiguity, a human would understand that

Plus we have no good reliable accuracy testing data in most cases (most tests occur on a few demos, but that isn't a good representation of how must things work), popular benchmarks, such as libero have been saturated, and nearly everything gets 95% there, most companies and researchers have their own benchmarks here.

Plus companies lie alot, and do very dangerous things in thier videos, I.e. these robots should not be standing very close to humans, because of being dangerous.

There are also legitimate concerns of misuse of these robots that need to be accounted for, misuse does not have to be warfare, but can be as simple as confusing it while it is cutting tomatoes with a knife.

Turning doorknob is easy, and fail recovery is also being worked on, but we don't have reliable statistics anywhere on that. The hard part is on practical things, as in when placing bricks or attaching a part during manufacturing it needs to ensure that it is aligning everything correctly....and that's hard, while it is impressive, it is very irresponsible to keep humanoids at home (people are irresponsible when untrained), for example, lawnmowers injure about 6400 people a year...and that is not an everything machine.

Humanoids in general are...not appealing in specific, due to maintainable of joints, complexity, but robot arms in particular, expecially on wheels (check mobile aloha), are likely to be able to do tasks such as clean up in hotels, after a guest had left, or replace some cooks in restaurants (if their work is consistent)

It’s still at GPT-1 level, but GPT-2 moment feels imminent.

IMHO the real test is if any robotic startup currently selling (or planning to sell) robots as a service for homes not just use it but gets returning users from it.

I did professionally few prototypes with robots and progress is real yet very far from what the average customer would find reliably useful in menial tasks.

FWIW I do think https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... remains relevant, namely dexterity is also a hardware problem, grippers aren't hands. They even clarify "multi-finger dexterous manipulation remains challenging." and those aren't even fingers with a lot of sensors.

There is a LOT of monkey business when it comes to robotics industry, a lot of demos you see are well crafted for that specific demo, a lot of times there’s even actual fraud “like claiming it’s an autonomous but it was actually teleoperated during the demo”, a lot robots that make most hypes are practically useless (dog robots or qugv for example) no matter how they try to create a usecase, when it comes to practicality, most fail. Robotics as an industry is niche (not to be confused with cobotics) the only exception is drones which is something I said in a workshop back in 2020 that drones are the only potential and will dominate if we solved the endurance issue, which is also why they immediately got weaponized because humans are usually evil and will utilize useful stuff in bad things.

Humanoids are far from being remotely useful in real life scenarios, yet.

Fun fact: most (if bot all) qugv can’t go reverse on a stairway.

Good stuff, yay, good.

I do want to pick a nit in this one,

> and the only lab where

I do think Allen Institution for AI (AI2) is has coverage across most of these domains ( albeit their frontier isn't nearly so far out, hopefully the $152m NSF awarded them + Nvidia is a fruitful partnership there).

For example, robotics: MolmoBot, MolmoSpaces, MolmoAct, https://allenai.org/embodied-ai

> Today, humans convert their labor to capital. Capital holders need labor (humans) to acquire more capital. When the price of inference for these robots becomes less than the price of labor then capital holders don’t need labor.

Ignoring quibbling about inference not being the only cost and other issues and just accepting the proposed end state: this is very good if capital is effectively democratized, and apocalyptically bad if it remains highly concentrated in a narrow class.

Not sure it is just about inference costs, errors in handling things and beings in the real world have a very different "surface" from immaterial applications - robots might need to have failsafes, independent limiters, etc. there.

and this is good?

>automated rubbish disposal can be integrated into houses and flats, grocery delivery and storage could be done via an automated intake, cataloguing, and storage system

This is a complete non-starter for already-built apartment blocks, terraced homes, and even most semi-detached. It's an interesting but costly solution for new detached homes.

Humanoids are a useful form factor because the already-built human world is, definitionally, built for humanoids.

Except Roombas can only clean 80-90% of my floor, and not as well as a real vacuum. And we have a dog that leaves toys everywhere so probably it does more like 60% of the job when accounting for clearing its path, emptying it, etc… To get to 100% you basically need the whole human(oid).

> Humanoids are just creepy and you can't really trust them.

Ultimately I shouldn't have to trust my robots. If my roomba or my dishwasher go haywire they won't pinch my finger off. They physically can't listen to me or spy on me. These are good robots.

I think that people will argue around that. Self-driving cars are already strong enough to kill people and even if you only made the robot strong enough to lift 15 pounds of weight, that's probably strong enough to slit someone's throat with a kitchen knife while they sleep... There probably isn't a level of strength that's weak enough to be completely safe but strong enough to be useful.

So at some point you have to trust that the tech is safe. Both in terms of "robot won't go off the rails", but also in terms of hostile actors can't remotely take over your robot while you sleep.

In terms of sleeping, personally, I would like for the law to mandate that robots must have a physical off switch, in a very visible location, that physically disconnects power. The switch should be illuminated while in the ON position. What makes me a bit pessimistic there is that we don't even have laws to mandate webcam indicator lights (e.g. a very tiny red LED) must be ON in hardware.

On latency: that is true now but likely false in the near future. There is so much potential left over for optimizing inference further.

On PID: the field has been stuck trying to do analytical/optimization-based control for decades, and end-to-end control has shown incredible performance (e.g., SoTA cost of transport in legged locomotion) and robustness (e.g., not falling over when stepping on a pile of leaves) - while being far scalable (in terms of how fast it is to get a new robot up and running). Which is not to say it's perfect but it seems like it's a step in the right direction.

Better yet, companies like Physical Intelligence are doing good with hierarchical ("fast-slow") architecture to address both the intelligence and latency fronts.

AFAIK NVidia has a VLA running on their chips with 2B params at 10Hz. They’ve openly published a .5B model running at 10Hz

Why not both?

The AI will need generalized robots to build the highly specialized ones.

falling down and breaking things?

I do think robotics would come up with more safety mechanisms (provably safe motion planning etc) just because the risk is a lot more serious than LLMs spitting half-truths

I actually like Tau's videos a lot better than Google's. An agile robot getting out of a car carrying a tote is actually novel and difficult and useful. In contrast, what Google is showing here is mostly glorified pick and place with low success rates on a needlessly complex robot and a voice LLM slapped on top to pre-announce its moves like an anime character, with fancy video editing and bubbly music to distract you from how slow it is.

Sure Tau is teleoperated, but teleoperating an agile movement like that is actually really hard to get right and still involves AI to keep the robot balanced. Tau is a lot closer to real deployment than Google, and when it performs useful tasks it is simultaneously collecting the data to eventually automate those tasks.

Welsh Labs has a great video on JEPA and robotics. https://www.youtube.com/watch?v=kYkIdXwW2AE

A 36% success rate on screwing in a light bulb means there's probably something to LeCunn's take that VLM/VLA models aren't going to be the thing powering tomorrow's robots, but only time will tell.

Humanoid robots can be dropped in wherever human labor is currently used.

That makes them extremely easy to sell to industry. This unlocks, in principle, a large chunk of the potential untapped industrial automation market that still relies on human labor because the ROI of redesigning production lines didn't make sense.

ABB, EPSON, FANUC, Kawasaki... Virtually every cars in past 50-60 years have had robots and some form of AI involved in assembly. Some of them already support standard humanoid/dogbot tools in their products.

Humanoid startup execs just don't talk about that.

In the Xiaomi Model Demo, they have non-humanoid robots

https://robotics.xiaomi.com/xiaomi-robotics-1.html

They already do: https://qwen.ai/blog?id=qwen-robotsuite

Nvidia also releases their Cosmos series models.

Cost, weight, durability. For cameras, bandwidth.

I would caution against enthusiasm. Anyone opposing those who end up controlling these robots won't be alive for long.

Also, "better" is doing a lot of lifting here, meaning it still fails regularly.

Yes. That's a whole other level of difficulty to accomplish safely.

No one has any idea how to achieve something like this.

Yeah, but that doesn’t mean the west won’t try to catch up

Still waiting for self driving car to bring me one robotic plumber to my construction site.

What's the point of not releasing it? Let people play with it and get some experience with it for a price.

Gemini is the only LLM I've ever experienced that decided on its own to tell me that something was a bad idea. Which, ok, in itself it's not a bad thing, we actually need more of that instead of the pathological sycophancy that all current models suffer from.

But in a physical robot? Yeah, that thing is going to punch me in the face, eventually. Or worse.

That makes it sound like Waymo is copying Tesla when in fact Waymo has been in the works before Tesla even considered self driving.

Another way to look at it is that they're avoiding some of the expensive parts of making products that use AI, while leaning into their strengths. No Tesla required within the explanation.

Bizarre oversimplification.

For both manipulation and autonomous driving, google has invested in approaches with custom hardware, and off-the-shelf hardware, and a blend (which is what waymo is).

Seems smart. There are dozens of companies that make low-margin cars. There's only one company that has managed to make a working autonomous software driver.