pi_at_home

—

This is my productionized deployment environment for my real world AI robot policies, and a place to create training data for reinforcement learning and imitation learning.

Video Demo:

It’s also the first project I’ve built using create-ros-app, a template I’m developing to make it easier for anyone to create and deploy production ready ROS2 applications.

Check out the large-scale synthetic dataset that’s been generated with this project!

Project Roadmap

Phase 1

Phase 1 focuses on leveraging imitation learning + synthetic data to improve the robustness of imitation-learning datasets with regards to lighting, camera position, and other environmental factors.

Phase 2

Add Reinforcement learning pipelines with ROS2 and Isaac-Sim support

Phase 3

Support long-horizon tasks involving multiple policies, with a focus on VLMs and language-grounded interaction with the robot.

Capabilities

Simple Teleoperation

The simplest thing you can do is teleoperation with your robot arm copied in Isaac Sim. To do that, run:

docker/launch teleop_myarm

This will automatically build and run everything needed for the project. Then, open http://localhost/ on your browser to view the project logs.

For in-depth documentation on the repository features, read the About Template documentation.

Synthetic Data Generation

There are three steps to synthetic data generation. First, open up the synthetic_generation launch profile:

docker/launch synthetic_generation

This will spin up Isaac Sim. Set up extensions using this guide. This extension is custom built, and stored in isaac_src/extensions. Next, plug in your robot. When using a real world robot, you will need to add a parameters.overrides.yaml file to the root of launch-profiles/synthetic_generation to configure your specific robots drivers.

Next, record an episode! Configure the “Trajectory Recorder” window, and click “Start Recording”:

Once you’ve recorded an episode, you can re-render with domain randomization using the “Trajectory Renderer” extension:

This will generate a configurable number of episodes while varying everything possible- including joint interpolation to move the robot faster or slower than the original demonstration.

Finally, convert the rendered episodes to a dataset:

docker/run convert_isaac_to_lerobot \
  --episodes_dir /robot/synthetic-output/recordings \
  --repo-id hf_username/dataset_name \
  --fps 30 \
  --task "Pick up a cube and place it in the basket" \
  --upload

Dependencies

• Docker, and optionally Nvidia Container Toolkit for hardware acceleration.
• Poetry, in order to use linting tooling.

This repository was initialized by the create-ros-app template. Contributions are welcome!