RU version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
88% Positive
Analyzed from 1876 words in the discussion.
Trending Topics
#world#model#atlas#input#images#camera#explicit#context#views#different
Discussion (49 Comments)Read Original on HackerNews
If you have a robot deployed in an environment, generating synthetic views of the environment you're in doesn't have any obvious value. What does have obvious value is the latent knowledge that the model could have used to generate those synthetic views.
For instance, the fact that Atlas is capable of identifying regions of the input images that look like "floors", and smoothly interpolating them and filling in gaps with more floor, suggests that it has a concept of "floor-like walkability" which it's learned from the examples in its training data. And being able to identify the regions of 3D space that correspond to that semantic label would obviously be useful for robot path planning.
There's plenty of literature about e.g. using neural networks to estimate walkable areas from a point cloud. And you could imagine just bolting one of those methods to the front of Atlas, using the synthesized point cloud (instead of traditional photogrammetry or LIDAR) as input. But that seems like it's throwing away a lot of potentially useful semantic information, on top of being needlessly inefficient.
I don't know if that undercuts what you're saying entirely. But if they're just protecting the splat in real-time based on the camera path through a 3d world, would the model truly understand that it's a floor?
Furthermore, being able to extract and process world geometry and 3D objects from Atlas could reduce friction in the early stages of indy development, where developer time is stretched thinner.
I'm very excited about AI tooling moving forward if this is a glimpse into the future.
You still need the technical proficiency to be able to make edits beyond that, but I think studios would be foolish to let go of their experts in the first place.
They show it working with videos that have motion, but it seems like time is always frozen while the camera is moving, and they always return to a ground truth camera view before advancing time again. Maybe the temporal consistency isn't very good? This surprises me given how well it understands space. I guess modeling physics and time is the next step in the development of this kind of model.
But this is certainly one area we plan to improve going forward.
- Renderers output pixels or views of a world, either the real world or a generated world
- Simulators predict how the world responds to actions
- Planners predict actions for an agent to make to affect some change in the world
Under this taxonomy, Atlas is somewhere between renderer and simulator. It predicts high-quality views of a consistent 3D world (renderer) but it can also predict explicit 3D to aid in simulation. Atlas doesn't currently predict actions to directly drive a robot, so it isn't natively a planner; but its simulation abilities can help train another model to be planner.
Source: https://www.worldlabs.ai/blog/taxonomy-of-world-models
Best definition I've heard is: AI systems that can build an internal map of their surroundings to anticipate what happens next and make decisions based on their predictions about the consequences the different actions they can take would have.
There's a bunch of different approaches people are trying:
- World labs (linked in this post) is going down the route of neural 3D representation work (NeRFs, 3D Gaussian Splatting)
- Yann LeCun is pretty famously betting on JEPA architectures (check out the excellent Welch Labs videos for more)
- Google is betting on generative video
- Karl Friston was pursuing 'active interference,' which is just traditional RL techniques with different reward functions
However essentially a world model is something that has the understanding of 3d world and can generate novel view point given either text or image input.
The use I have seen is for robotics. You feed in the current view and describe the action you want it to do, and then it plans the arm movements. (really useful for softbody manipulation.
There are other meanings. but essentially a world model is able to reason in 3d, rather than text.
Another big differentiator is multimodality. Genie 3 only outputs pixels. Atlas also outputs pixels, but it can also output explicit 3D for the cases where you need it (such as plugging into game engines, simulators, or VFX workflows)
An ideal workflow would be something semi-interactive that you can use to quickly iterate on an idea, followed by a longer offline bake-out to generate final production-quality assets.
Atlas is an auto-regressive diffusion model, so context length limitations apply similar to LLMs and video models.
Where Atlas has an edge is that its context comprised of an arbitrary sequence of images with camera poses, which lends itself to managing the context in creative ways (we called this "context juggling" in our RTFM blog, https://www.worldlabs.ai/blog/rtfm). So yes through clever context management you could potentially build an entire 3D model of the world.
My first thought is a detailed Hogwarts that is fully explorable using scenes from the movies (or even descriptions from the books?)
Also, big fan of Dr.Li!
Potentially very significant for accelerating the data flywheel challenge for robotics
If you only want to reconstruct the content in the input images without any imagination, Atlas can do sparse 3D reconstruction where it predicts a depth map for each input pixel, without predicting anything occluded or not visible in the input views. This could be used to implement your "fog of war" view.
But sometimes you want the model to stitch together views and imagine a coherent completion; this is a powerful tool for building and authoring generative worlds. For example the 1 minute video we show in the post was built by positioning seven unrelated input images in space and prompting the model to interpolate between them along a hand-specified camera path.
Dedicated depth sensors used to be a must on advanced robotics platforms - the only way to get anything close to reliable 3D point clouds was to spin a LiDAR. But by now, I wouldn't be surprised to see more and more robots ship with smartphone-like camera blocks - varying FoVs and focal depths, but not a lot of explicit depth sensing, if any at all.
Also, I wonder if this very model can be retrofit into a true robotics VLA? If it already takes text and image guidance, performs autoregressive diffusion of novel views, and handles temporal dynamics - why not diffusion of actions too?
“It generates both image frames from novel views and explicit 3D outputs”
Model can indeed generate novel views but I don’t think in real time (I could be wrong). If you want to navigate a space in real time as user above was asking gotta rely on the 3D output and the explicit representation will provide continuity. User likely was referring to the limitations we’ve seen on video-style world models like genie and precursors.