ROS index logo ROS Index
  1. Home
  2. Packages
  3. rai_open_set_vision
 
No version for distro humble. Known supported distros are highlighted in the buttons above.
No version for distro jazzy. Known supported distros are highlighted in the buttons above.
No version for distro rolling. Known supported distros are highlighted in the buttons above.

rai_open_set_vision package from rai repo

rai_whatisee rai_bringup rai_nomad rai_open_set_vision rai_interfaces rai_state_logs

API Docs
Browse Code

Package Summary

Tags No category tags.
Version 0.1.0
License Apache-2.0
Build type AMENT_PYTHON
Use RECOMMENDED

Repository Summary

Description RAI is an agentic framework for robotics, utilizing Langchain and ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more.
Checkout URI https://github.com/robotecai/rai.git
VCS Type git
VCS Version development
Last Updated 2025-04-24
Dev Status UNKNOWN
CI status No Continuous Integration
Released UNRELEASED
Tags ai robotics ros2 vlm multimodal embodied-artificial-intelligence embodied-agent embodied-ai o3de llm generative-ai ai-agents-framework embodied-agents robotec
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Package Description

Package enabling open set vision for RAI

Additional Links

No additional links.

Maintainers

  • Kajetan Rachwał

Authors

No additional authors.
src/rai_extensions/rai_open_set_vision/README.md

RAI Open Set Vision

This package provides a ROS2 Node which is an interface to the Idea-Research GroundingDINO Model. It allows for open-set detection.

Installation

In your workspace you need to have an src folder containing this package rai_open_set_vision and the rai_interfaces package.

Preparing the GroundingDINO

Add required ROS dependencies:

rosdep install --from-paths src --ignore-src -r

Build and run

In the base directory of the RAI package install dependencies:

poetry install --with openset

Source the ros installation

source /opt/ros/${ROS_DISTRO}/setup.bash

Run the build process:

colcon build --symlink-install

Source the environment

source setup_shell.sh

Run the ROS node using ros2 launch:

ros2 launch rai_open_set_vision gdino_launch.xml [weights_path:=PATH/TO/WEIGHTS]

[!NOTE] By default the weights will be downloaded to $(ros2 pkg prefix rai_open_set_vision)/share/weights/. You can change this path if you downloaded the weights manually or moved them.

RAI Tools

This package provides the following tools:

  • GetDetectionTool This tool calls the grounding dino service to use the model to see if the message from the provided camera topic contains objects from a comma separated prompt.

    Example call

  x = GetDetectionTool(node=RaiBaseNode(node_name="test_node"))._run(
      camera_topic="/camera/camera/color/image_raw",
      object_names=["chair", "human", "plushie", "box", "ball"],
  )

Example output

  I have detected the following items in the picture - chair, human

  • GetDistanceToObjectsTool This tool calls the grounding dino service to use the model to see if the message from the provided camera topic contains objects from a comma separated prompt. Then it utilises messages from depth camera to create an estimation of distance to a detected object.

    Example call

  x = GetDistanceToObjectsTool(node=RaiBaseNode(node_name="test_node"))._run(
      camera_topic="/camera/camera/color/image_raw",
      depth_topic="/camera/camera/depth/image_rect_raw",
      object_names=["chair", "human", "plushie", "box", "ball"],
  )

Example output

  I have detected the following items in the picture human: 1.68 m away, chair: 2.20 m away

Example

An example client is provided with the package as rai_open_set_vision/talker.py

You can see it working by running:

ros2 launch rai_open_set_vision example_communication_launch.xml image_path:=src/rai_extensions/rai_open_set_vision/images/sample.jpg [dino_weights_path:=PATH/TO/DINO/WEIGHTS] [sam_weights_path:=PATH/TO/SAM/WEIGHTS]

If everything was set up properly you should see a couple of detections with classes dinosaur, dragon, and lizard.

CHANGELOG
No CHANGELOG found.

Wiki Tutorials

This package does not provide any links to tutorials in it's rosindex metadata. You can check on the ROS Wiki Tutorials page for the package.

Package Dependencies

Deps Name
vision_msgs
rai_interfaces
ament_copyright
ament_pep257

System Dependencies

Name
python3-pytest

Dependant Packages

No known dependants.

Launch files

  • launch/gsam_launch.xml
    • ~ Copyright 2024 Robotec.ai ~ ~ Licensed under the Apache License, Version 2.0 (the "License"); ~ you may not use this file except in compliance with the License. ~ You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License.
      • weights_path [default: ]
  • launch/gdino_launch.xml
    • ~ Copyright 2024 Robotec.ai ~ ~ Licensed under the Apache License, Version 2.0 (the "License"); ~ you may not use this file except in compliance with the License. ~ You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License.
      • weights_path [default: ]
  • launch/example_communication_launch.xml
    • ~ Copyright 2024 Robotec.ai ~ ~ Licensed under the Apache License, Version 2.0 (the "License"); ~ you may not use this file except in compliance with the License. ~ You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License.
      • dino_weights_path [default: ]
      • sam_weights_path [default: ]
      • image_path

Messages

No message files found.

Services

No service files found

Plugins

No plugins found.

Recent questions tagged rai_open_set_vision at Robotics Stack Exchange

No version for distro noetic. Known supported distros are highlighted in the buttons above.
No version for distro ardent. Known supported distros are highlighted in the buttons above.
No version for distro bouncy. Known supported distros are highlighted in the buttons above.
No version for distro crystal. Known supported distros are highlighted in the buttons above.
No version for distro eloquent. Known supported distros are highlighted in the buttons above.
No version for distro dashing. Known supported distros are highlighted in the buttons above.
No version for distro galactic. Known supported distros are highlighted in the buttons above.
No version for distro foxy. Known supported distros are highlighted in the buttons above.
No version for distro iron. Known supported distros are highlighted in the buttons above.
No version for distro lunar. Known supported distros are highlighted in the buttons above.
No version for distro jade. Known supported distros are highlighted in the buttons above.
No version for distro indigo. Known supported distros are highlighted in the buttons above.
No version for distro hydro. Known supported distros are highlighted in the buttons above.
No version for distro kinetic. Known supported distros are highlighted in the buttons above.
No version for distro melodic. Known supported distros are highlighted in the buttons above.