Stretch AI

Stretch AI is designed to help researchers and developers build intelligent behaviors for the Stretch 3 mobile manipulator from Hello Robot. It contains code for:

• grasping
• manipulation
• mapping
• navigation
• LLM agents
• text to speech and speech to text
• visualization and debugging

Much of the code is licensed under the Apache 2.0 license. See the LICENSE file for more information. Parts of it are derived from the Meta HomeRobot project and are licensed under the MIT license.

Hardware Requirements

We recommend the following hardware to run Stretch AI. Other GPUs and other versions of Stretch may support some of the capabilities found in this repository, but our development and testing have focused on the following hardware.

• Stretch 3 from Hello Robot
  • When Checking Hardware, stretch_system_check.py should report that all hardware passes.
• Computer with an NVIDIA GPU
  • The computer should be running Ubuntu 22.04. Later versions might work, but have not been tested.
  • Most of our testing has used a high-end CPU with an NVIDIA GeForce RTX 4090.
• Dedicated WiFi access point
  • Performance depends on high-bandwidth, low-latency wireless communication between the robot and the GPU computer.
  • The official Stretch WiFi Access Point provides a tested example.
• (Optional) Stretch Dexterous Teleop Kit.
  • To use the learning-from-demonstration (LfD) code you’ll need the Stretch Dexterous Teleop Kit.

Quick-start Guide

Artificial intelligence (AI) for robots often has complex dependencies, including the need for trained models. Consequently, installing stretch-ai from source can be challenging.

First, you will need to install software on your Stretch robot and another computer with a GPU (GPU computer). Use the following link to go to the installation instructions: Instructions for Installing Stretch AI

Once you’ve completed this installation, you can start the server on your Stretch robot. Prior to running the script, you need to have homed your robot with stretch_robot_home.py. Then, run the following command:

./scripts/run_stretch_ai_ros2_bridge_server.sh

After this, we recommend trying the Language-Directed Pick and Place demo.

Experimental support for Older Robots

The older model of Stretch, the Stretch RE2, did not have an camera on the gripper. If you want to use this codebase with an older robot, you can purchase a Stretch 2 Upgrade Kit to give your Stretch 2 the capabilities of a Stretch 3. Alternatively, you can run a version of the server with no d405 camera support on your robot.

Note that many demos will not work with this script (including the Language-Directed Pick and Place demo) and learning from demonstration. However, you can still run the simple motions demo and view images with this script.

./scripts/run_stretch_ai_ros2_bridge_server.sh --no-d405

Optional: Docker Quickstart

To help you get started more quickly, we provide two pre-built Docker images that you can download and use with two shell scripts.

On your remote machine, you can install docker as normal, then, you can start the client on your GPU computer:

./scripts/run_stretch_ai_gpu_client.sh

This script will download the Docker image and start the container. You will be able to run Stretch AI applications from within the container.

Language-Directed Pick and Place

Now that you have the server running on Stretch, we recommend you try a demonstration of language-directed pick and place.

For this application, Stretch will attempt to pick up an object from the floor and place it inside a nearby receptacle on the floor. You will use words to describe the object and the receptacle that you’d like Stretch to use.

While attempting to perform this task, Stretch will speak to tell you what it is doing. So, it is a good idea to make sure that you have the speaker volume up on your robot. Both the physical knob on Stretch’s head and the volume settings on Stretch’s computer should be set so that you can hear what Stretch says.

Now, on your GPU computer, run the following commands in the Docker container that you started with the script above.

You need to let the GPU computer know the IP address (#.#.#.#) for your Stretch robot.

./scripts/set_robot_ip.sh #.#.#.#

Please note that it’s important that your GPU computer and your Stretch robot be able to communicate via the following ports 4401, 4402, 4403, and 4404. If you’re using a firewall, you’ll need to open these ports.

Next, run the application on your GPU computer:

python -m stretch.app.ai_pickup

It will first spend time downloading various models that it depends on. Once the program starts, you will be able to bring up a Rerun-based GUI in your web browser.

Then, in the terminal, it will ask you to specify an object and a receptacle. For example, in the example pictured below, the user provided the following descriptions for the object and the receptacle.

Enter the target object: plush rabbit toy
Enter the target receptacle: yellow chair 

At Hello Robot, people have successfully commanded the robot to pick up a variety of objects from the floor and place them in nearby containers, such as baskets and boxes.

Find out more about the LLM-based AI agent in its documentation. And once you’re ready to learn more about Stretch AI, you can try out the variety of applications (apps) that demonstrate various capabilities.

Next Steps

Check out additional documentation for ways to use Stretch AI:

• LLM Agent – How to use the LLM agent for language-directed pick and place
• Add a New LLM Task – How to add a new task that can be called by an LLM
• DynaMem – Run the LLM agent in dynamic scenes, meaning you can walk around and place objects as the robot explores
• Data Collection for Learning from Demonstration – How to collect data for learning from demonstration
• Learning from Demonstration – How to train and evaluate policies with LfD
• Open-Vocabulary Mobile Manipulation – Experimental code which can handle more complex language commands
• Apps – List of many different apps that you can run
• Simple API – How to use the simple API to control the robot over wireless

Development

Clone this repo on your Stretch and PC, and install it locally using pip with the “editable” flag:

cd stretch_ai/src
pip install -e .[dev]
pre-commit install

Then follow the quickstart section. See CONTRIBUTING.md for more information. There is some information on how to debug and update the codebase.

You can test out most code in the simulation environment, which is a good way to test code without needing a robot.

Updating Code on the Robot

See the update guide for more information. Code installed from git must be updated manually, including code from this repository.

You can also pull the latest docker image on the robot with the following command:

./scripts/run_stretch_ai_ros2_bridge_server.sh --update

Building Docker Images

Docker build and other instructions are located in the docker guide. Generally speaking, from the root of the project, you can run the docker build process with:

docker build -t stretch-ai_cuda-11.8:latest .

See the docker guide for more information and troubleshooting advice.

Acknowledgements

Parts of this codebase were derived from the Meta HomeRobot project, and is licensed under the MIT license. We thank the Meta team for their contributions.

The stretch_ros2_bridge package is based on the OK robot project’s Robot Controller, and is licensed under the Apache 2.0 license.

We use LeRobot from HuggingFace for imitation learning, though we use our own fork.

License

This code is licensed under the Apache 2.0 license. See the LICENSE file for more information.

Contributing to Stretch AI

We welcome contributions to Stretch AI! Please read the following guidelines before submitting a pull request.

This repository is in an early state of development. Guidelines are subject to change and tests will be by necessity incomplete.

Before contributing, you will need to sign a Contributor License Agreement. For more information, you can check out the Stretch Contributor License Agreement page for more information.

We use the SAP CLA Assistant bot, which will open a comment and give you instructions when you open a pull request.

Setup

Install the code and set up the pre-commit hooks:

git clone https://github.com/hello-robot/stretch_ai.git --recursive
cd stretch_ai/src
pip install -e .[dev]
pre-commit install

Style

We use black and flake8 to format our code. In addition, we use isort for sorting imports, mypy for static type checking, and codespell for spell checking, among other things.

You can run the pre-commit hooks on all files with:

pre-commit run --all-files

Please make sure that all changes are made and that the pre-commit hooks pass before submitting a pull request.

If you need to temporarily commit something that is not passing, use:

git commit --no-verify

However, pull requests with failing pre-commit hooks will not be merged.

Pull Requests

We follow a squash-and-merge strategy for pull requests, which means that all commits in a PR are squashed into a single commit before merging. This keeps the git history clean and easy to read.

Please make sure your PR is up-to-date with the latest changes in the main branch before submitting. You can do this by rebasing your branch on the main branch:

git checkout main
git pull
git checkout <your-branch>
git rebase main

Draft PRs

If a PR is still a work-in-progress and not ready for review, please open it with “WIP: (final PR title)” in the title to indicate that it is still a work in progress. This will indicate to reviewers that the PR is not ready for review yet. In addition, use the “Draft” PR status on github to indicate that the PR is not ready yet.

Documentation

Please make sure to update the documentation if you are adding new features or changing existing ones. This includes docstrings, README, and any other relevant documentation. Use type hints to make the code more readable and maintainable.

For example:

def add(a: int, b: int) -> int:
    return a + b

This shows what a and b are expected to be and what the function returns – in this case, all are int variables.

Testing

We use pytest for testing. Please make sure to add tests for new features or changes to existing code. You can run the tests with:

cd src
pytest

File Structure

The code is organized as follows. Inside the core package src/stretch:

• core is basic tools and interfaces
• app contains individual endpoints, runnable as python -m stretch.app.<app_name>, such as mapping, discussed above.
• motion contains motion planning tools, including algorithms like RRT.
• mapping is broken up into tools for voxel (3d / ok-robot style), instance mapping
• perception contains tools for perception, such as object detection and pose estimation.
  • perception/encoders contains tools for encoding vision and language features, such as the SiglipEncoder class.
  • perception/captioners contains tools for generating captions from images, such as the MoonbeamCaptioner class.
• agent is aggregate functionality, particularly robot_agent which includes lots of common tools including motion planning algorithms.
  • In particular, agent/zmq_client.py is specifically the robot control API, an implementation of the client in core/interfaces.py. there’s another ROS client in stretch_ros2_bridge.
  • agent/robot_agent.py is the main robot agent, which is a high-level interface to the robot. It is used in the app scripts.
  • agent/base contains base classes for creating common and sequentially executed robot operations through the ManagedOperation class.
  • agent/task contains task-specific code, such as for the pickup task. This is divided between “Managers” like pickup_manager.py which are composed of “Operations.” Each operation is a composable state machine node with pre- and post-conditions.
  • agent/operations contains the individual operations, such as move_to_pose.py which moves the robot to a given pose.

The stretch_ros2_bridge package is a ROS2 bridge that allows the Stretch AI code to communicate with the ROS2 ecosystem. It is a separate package that is symlinked into the ament_ws workspace on the robot.

Trying individual components

Perception

The perception package contains tools for perception, such as object detection and pose estimation.

Here are some images you can use to test the perception tools:

Object Image	Receptacle Image

You can try the captioners with:

# Moonbeam captioner
# Gives: "A plush toy resembling a spotted animal is lying on its back on a wooden floor, with its head and front paws raised."
python -m stretch.perception.captioners.moonbeam_captioner --image_path object.png
# Gives: "An open cardboard box rests on a wooden floor, with a stuffed animal lying next to it."
python -m stretch.perception.captioners.moonbeam_captioner --image_path receptacle.png

# ViT + GPT2 captioner
# Gives: "a cat laying on the floor next to a door"
python -m stretch.perception.captioners.vit_gpt2_captioner --image_path object.png
# Gives: "a box with a cat laying on top of it"
python -m stretch.perception.captioners.vit_gpt2_captioner --image_path receptacle.png

# Blip Captioner
# Gives: "a stuffed dog on the floor"
python -m stretch.perception.captioners.blip_captioner --image_path object.png
# Gives: "a dog laying on the floor next to a cardboard box"
python -m stretch.perception.captioners.blip_captioner --image_path receptacle.png

# Git Captioner
# Gives: "a stuffed animal that is laying on the floor"
python -m stretch.perception.captioners.git_captioner --image_path object.png
# Gives: "a cardboard box on the floor"
python -m stretch.perception.captioners.git_captioner --image_path receptacle.png

Developing on the Robot

You can run the server and ROS nodes separately on the robot. Run the server itself with:

ros2 run stretch_ros2_bridge server

You can then run the SLAM and other ROS nodes:

ros2 launch stretch_ros2_bridge startup_stretch_hector_slam.launch.py

Or, if you are using ORB-SLAM3:

ros2 launch stretch_ros2_bridge startup_stretch_orbslam.launch.py