Quick framing, since the post is long: I did robotic manipulation research at OpenAI from 2017–2020, and the tabletop setup back then cost roughly 10x this one and took a team to run. This project is me testing whether a single person can now do meaningful work on the same class of problems: starting with physical and software setup.
A few decisions I'm least settled on, and would love some pushback/feedback on:
- single arm vs. bimanual (I went single for cost/space, knowing it rules out things like folding cloth)
- not calibrating camera extrinsics/intrinsics for now
- RGB vs. RGB-D for from-scratch policies (ACT / Diffusion Policy)
And one I'm more confident about but expect disagreement on: not building on ROS 2 / LeRobot, and writing my own stack instead. Happy to get into the reasoning.
Discussion (7 Comments)Read Original on HackerNews
>I do not intend to calibrate the camera’s extrinsics or intrinsics for now.
Sensible choice, although I suggest it's good in the long run to do at early stage in your setup, especially if you intend to collect data for policy learning.
Debugging trained policies for visual manipulation task can be a headache and having as much context on variables that can change is a good practice.
My previous setup was in Japan, a earthquake prone place and I wasted some time after realizing the camera got misaligned due to earthquake. A simple solution is just to place an Aruco marker on the table that tracks the relative extrinsic position of camera, and add it as metadata to collected teleoperation dataset.
It would be interesting to explore how RL can be applied on top of my (flawed) human demos to optimize beyond what I’m able to do.
Would like to know your reasoning on not going with LeRobot.
How have you found it?
(The author does explain his reasons for not using LeRobot in the post - although "I also use LeRobot for training and running baseline policies, and the vendor SDKs for the hardware.")
- I've heard the advantage of ROS besides the architecture is the ecosystem (driver integrations, etc). Is that not an issue because the arm supports a Python SDK OOTB?
- Any issues you've been running into with this setup?
- How do you determine if a session recording is good enough for training? Is 50/100 samples really all you need?