RAI

[!IMPORTANT]
Development Status: RAI is currently undergoing significant development on the development branch, focusing on version 2.0. This major version update will introduce substantial improvements and is not backward compatible with version 1.0. For the latest stable release, please refer to the main branch.

RAI is a flexible AI agent framework to develop and deploy Embodied AI features for your robots.

Overview

The RAI framework aims to:

• Supply a general multi-agent system, bringing Gen AI features to your robots.
• Add human interactivity, flexibility in problem-solving, and out-of-box AI features to existing robot stacks.
• Provide first-class support for multi-modalities, enabling interaction with various data types.

Limitations

• Limitations of LLMs and VLMs in use apply: poor spatial reasoning, hallucinations, jailbreaks, latencies, costs, …
• Resource use (memory, CPU) is not addressed yet.​
• Requires connectivity and / or an edge platform.​

Table of Contents

• Features
• Setup (docker)
• Setup (local)
• Usage examples (demos)
• Debugging Assistant
• Developer resources

Features

• Voice interaction (both ways).
• Customizable robot identity, including constitution (ethical code) and documentation (understanding own capabilities).
• Accessing camera (“What do you see?”), utilizing VLMs.
• Summarizing own state through ROS logs.
• ROS 2 action calling and other interfaces. The Agent can dynamically list interfaces, check their message type, and publish.
• Integration with LangChain to abstract vendors and access convenient AI tools.
• Tasks in natural language to nav2 goals.
• NoMaD integration.
• Tracing.
• Grounded SAM 2 integration.
• Improved Human-Robot Interaction with voice and text.
• Additional tooling such as GroundingDino.
• Support for at least 3 different AI vendors.
• Debugging assistant for ROS 2.
• SDK for RAI developers.
• UI for configuration to select features and tools relevant for your deployment.

Setup (docker)

Currently, docker images are experimental. See the docker for instructions.

Setup (local)

Before going further, make sure you have ROS 2 (Jazzy or Humble) installed and sourced on your system. If you don’t have ROS 2 follow the installation documentation for Humble or Jazzy. Make sure that ros-dev-tools are installed.

1. Setting up the workspace:

1.1 Install poetry

RAI uses Poetry for python packaging and dependency management. Install poetry with the following line:

curl -sSL https://install.python-poetry.org | python3 -

Alternatively, you can opt to do so by following the official docs.

1.2 Clone the repository:

git clone https://github.com/RobotecAI/rai.git
cd rai

1.3 Create poetry virtual environment and install dependencies:

poetry install
rosdep install --from-paths src --ignore-src -r -y

[!TIP]
RAI is modular. If you want to use features such as speech-to-speech, simulation and benchmarking suite, openset detection, or NoMaD integration, install additional dependencies:

poetry install --with openset,nomad,s2s,simbench

1.4 Configure RAI

Run the configuration tool to set up your vendor and other settings:

poetry run streamlit run src/rai_core/rai/frontend/configurator.py

[!TIP]
If the web browser does not open automatically, open the URL displayed in the terminal manually.

2. Build the project:

2.1 Build RAI workspace

colcon build --symlink-install

2.2 Activate a virtual environment:

source ./setup_shell.sh

3. Setting up vendors

RAI is vendor-agnostic. Use the configuration in config.toml to set up your vendor of choice for RAI modules. Vendor choices for RAI and our recommendations are summarized in Vendors Overview.

We strongly recommend you to use of best-performing AI models to get the most out of RAI!

Pick your local solution or service provider and follow one of these guides:

• Ollama
• OpenAI
• AWS Bedrock

What’s next?

Once you know your way around RAI, try the following challenges, with the aid the developer guide:

• Run RAI on your own robot and talk to it, asking questions about what is in its documentation (and others!).
• Implement additional tools and use them in your interaction.
• Try a complex, multi-step task for your robot, such as going to several points to perform observations!

Debugging Assistant

Use the debugging assistant to inspect ROS 2 network state and troubleshoot issues.

Simulation demos

Try RAI yourself with these demos: | Application | Robot | Description | Docs Link | | —————————————— | ———————— | ———————————————————————————————————————————————— | ————————————————————- | | Mission and obstacle reasoning in orchards | Autonomous tractor | In a beautiful scene of a virtual orchard, RAI goes beyond obstacle detection to analyze best course of action for a given unexpected situation. | link | | Manipulation tasks with natural language | Robot Arm (Franka Panda) | Complete flexible manipulation tasks thanks to RAI and Grounded SAM 2 | link | | Autonomous mobile robot demo | Husarion ROSbot XL | Demonstrate RAI’s interaction with an autonomous mobile robot platform for navigation and control | link | | Turtlebot demo | Turtlebot | Showcase RAI’s capabilities with the popular Turtlebot platform | link | | Speech-to-speech interaction with autonomous taxi | Simulated car | Demonstrate RAI’s speech-to-speech interaction capabilities for specifying destinations to an autonomous taxi in awsim with autoware environment | link |

Community

Embodied AI Community Group

RAI is one of the main projects in focus of the Embodied AI Community Group. If you would like to join the next meeting, look for it in the ROS Community Calendar.

Publicity

• A talk about RAI at ROSCon 2024.

RAI Q&A

Please take a look at Q&A.

Developer Resources

See our Developer Guide for a deeper dive into RAI, including instructions on creating a configuration specifically for your robot.

Contributing

You are welcome to contribute to RAI! Please see our Contribution Guide.

Contributing

We are very happy you want to contribute to RAI, and we welcome all input. This document outlines guidelines for contributors to follow.

Our philosophy is strongly aligned with the philosophy of the ROS development process. These guidelines have been therefore strongly influenced by the ROS2 Contributing document.

Tenets

• Engage Robotec.ai as early as possible

  • Start discussions with Robotec.ai and the community early. Long time RAI contributors may have a clearer vision of the big picture. If you implement a feature and send a pull request without discussing with the community first, you are taking the risk of it being rejected, or you may be asked to largely rethink your design.
  • Opening issues or using Discourse to socialize an idea before starting the implementation is generally preferable.

• Adopt community best-practices whenever possible instead of ad-hoc processes

  Think about the end-users experience when developing and contributing. Features accessible to a larger amount of potential users, utilising widely available solutions are more likely to be accepted.

• Think about the community as a whole

  Think about the bigger picture. There are developers building different robots with different constraints. The landscape of available AI models is rapidly changing, coming with different capabilities and constraints. RAI wants to accommodate requirements of the whole community.

There are a number of ways you can contribute to the RAI project.

Discussions and support

Some of the easiest ways to contribute to RAI involve engaging in community discussions and support. This can be done by creating Issues and RFCs on github.

Contributing code

Setting up the development environment

Set up your development environment following the instructions.

Starting the discussion

Always try to engage in discussion first. Browse Issues and RFCs or start a discussion on RAI Discord to see if a feature you want to propose (or a similar one) has already been mentioned. If that is the case feel free to offer that you’ll work on it, and propose what changes/additions you will make. One of the project maintainers will assign the issue to you, and you can start working on the code.

Submitting code changes

To submit a change begin by forking this repository and making the changes on the fork. Once the changes are ready to be proposed create a pull request back to the repository. In order to maintain a linear and clear commit history please:

• make sure that all commits have meaningful messages
• if batches of “cleanup” or similar commits are present - squash them together
• rebase onto the main branch of repository before making the PR

Always make sure that both all tests are passing before making the PR. Once this is done open your PR describing what changes have been made and how to test if it’s working. Request review from the maintainer who assigned you the issue.

If the review requires modifications please make them on your forked repository and go through the above process. Once the PR is accepted it will be merged into the repository. Congratulations, and thank you for your contributions to the development of RAI.