ETRI’s Dual Hand-Arm Robot in Issac Sim

This repository provides USD files and ROS 2 packages for ETRI’s dual hand-arm robot in Isaac Sim, designed for manipulation skill learning. ETRI’s dual hand-arm is a 49 DOF robot consisting of two sets of Kinova Gen3 arms and Allegro Hands(For more details on the robot system, refer to this documentation). Using ROS 2 message communication, you can control this robot by sending joint commands and receiving feedback on the robot’s joint states, camera images, and transform information at the same time. Please use this repository for your robot research.

Interface between the user’s program and the robot in Isaac Sim

A demo of the robot grasping objects with each arm separately. Please note that this example is not included in this repository. Robot manipulation or motion planning is up to the user.

Prerequisites

Intermediate or higher-level skills in both ROS 2 and Isaac Sim are required. And here is the version of ROS 2 and Isaac Sim that we used:

NVIDIA Omniverse / Isaac Sim

• Omniverse Launcher 1.9.14
• Isaac Sim 2023.1.1

  ROS

• ROS 2 Humble Hawksbill (Ubuntu 22.04.4 LTS)

*NOTE: It is recommended to perform ‘ROS and ROS 2 Installation for Isaac Sim’ in advance to ensure Isaac Sim’s compatibility with ROS 2.

Installation

Clone this git repository using the command line or download zip.

git clone https://github.com/DonghyungKim/ETRI-Dual-Hand-Arm-Robot.git

Copy usd_files to your IsaacSim workspace. Also, copy etri_dualarm_cmd_msg_converter_sim and sample_etri_dualarm_ctr to the src directory of your ROS 2 workspace. And build your ROS 2 workspace.

• usd_files can be placed in any folder chosen by the user
• Let’s say your ROS 2 workspace is ~/robot_ws, then the path to etri_dualarm_cmd_msg_converter_sim and sample_etri_dualarm_ctr should be ~/robot_ws/src/sample_etri_dualarm_ctr. You need to run colcon build, e.g.

cd ~/robot_ws/src
colcon build --symlink-install

File Description

USD files (/usd_files)

• etri_dualarm_ros2_ctr.usd: main USD file for Isaac Sim simulation.
• etri_dualarm_robot.usd: USD file of ETRI’s dual hand-arm robot automatically generated by Isaac Sim’s URDF Importer.
• rsd455.usd: USD file of Intel RealSense Camera D455.

Command message converter (/etri_dualarm_cmd_msg_converter_sim): ROS 2 package that converts the user’s joint commands into suitable joint commands for Isaac Sim.

The default control for the robot joints is position control. However, in cases like visual servoing for teleoperation, some users prefer velocity control. Therefore, this repository provides two types of command message converters:

• /etri_dualarm_cmd_msg_converter_sim/run_both_arms_pos_ctrl.py: Command message converter for position control (49 DOF position control).
• /etri_dualarm_cmd_msg_converter_sim/run_both_arms_pos_ctrl.py: Command message converter with velocity control for the arm joints only, and position control for all other joints (7X2 DOF velocity control + 35 DOF position control)

Sample codes (/sample_etri_dualarm_ctr)

• /sample_etri_dualarm_ctr/sample_sequence_joint_pos_ctr.py: Sample code for sending sequential joint position commands to the robot.
• /sample_etri_dualarm_ctr/sample_sinusoidal_joint_pos_ctrl.py: Sample code for sending sinusoidal joint position commands to the robot
• /sample_etri_dualarm_ctr/sample_sinusoidal_arm_vel_ctrl.py: Sample code for sending sinusoidal joint velocity commands for the arms and position commands for all other joints.

How to Run

Step 1. Open etri_dualarm_ros2_ctr.usd in Isaac Sim

Launch the Isaac Sim and open etri_dualarm_ros2_ctr.usd(File -> Open). Then ETRI’s dual hand-arm robot will appear in the Viewport. Start the simulation by pressing play button ( :arrow_forward: ). If the USD file loads correctly without any issues, you can see the list of topic messages in the terminal by using ros2 topic list command. | Topic name | Description | |—|:—:| | /joint_states_isaac | The current state of the robot published by Isaac Sim. Please refer this documentation for the details.| | /joint_command_isaac | The robot’s joint command subscribed to by Isaac Sim. When the user publishes to /joint_command, this message is converted to /joint_command_isaac using the command message converter, and the controller within Isaac Sim moves the robot’s joints. | | /head_camera/color/camera_info, /head_camera/color/image_raw, /head_camera/depth/camera_info, /head_camera/depth/image_rect_raw, /head_camera/depth/color/points | The topics related to the robot’s head camera (RGB, depth, point cloud) | | /left_wrist_camera/color/camera_info, /left_wrist_camera/color/image_raw, /left_wrist_camera/depth/camera_info, /left_wrist_camera/depth/image_raw, /right_wrist_camera/color/camera_info, /right_wrist_camera/color/image_raw, /right_wrist_camera/depth/camera_info, /right_wrist_camera/depth/image_raw | The topics related to the robot’s wrist camera (RGB, depth) | | /tf | Transform information (TF) for the robot’s tool frames and camera frames relative to the base link |

• Please note that the robot in etri_dualarm_ros2_ctr.usd is different from the robot in etri_dualarm_robot.usd, as I have refined various parameter values, including Joint Drive Gains, through trial and error.
• In IsaacSim’s Stage panel, the Controller, Cameras, and TFs under the etri_dualarm_robot are Action Graphs responsible for the following:
  • Controller: Handles robot joint state feedback and control.
  • Cameras: Publishes camera-related messages.
  • TFs: Publishes TF (Transformation) messages.
• If the topic messages listed above do not appear, check whether UseDomain ID Env Var is unchecked in the ROS2 Context Node within the Action Graphs. If it is unchecked and you are using a domain_id different from your environment’s ROS_DOMAIN_ID, all ROS 2 nodes in Isaac Sim may fail to be discovered. For more details on ROS 2 domain ID issues, read Graph Explained here.

Step 2. Run the command message converter

After starting the simulation, you must run the command message converter. This is the ROS 2 node that convert /joint_command to /joint_command_isaac which is suitable joint commands for Isaac Sim.

For 49 DOF position control:

If you want to run the command message converter for position control, use the following command:

ros2 run etri_dualarm_cmd_msg_converter_sim run_both_arms_pos_ctrl

For 7X2 DOF velocity control(two arms) + 35 DOF position control:

If you want to run the command message converter with velocity control for the arm joints only and position control for all other joints, you first need to determine the control frequency. Then, enter this frequency as the hz argument for the node. For example, if the control frequency is 30 Hz, run the following command:

ros2 run etri_dualarm_cmd_msg_converter_sim run_both_arms_vel_ctrl hz:=30

Step 3. Commanding the robot

Now you’re ready to move the robot! You can control the robot by publishing to /joint_command. See the definition of this message.

The following is an example command using ros2 topic pub to move the robot to a sample position. Try it after running the command message converter for position control, and check if the robot moves in IsaacSim.

ros2 topic pub --once /joint_command sensor_msgs/msg/JointState \
"{name: [liftkit_extension,
        head_pan_joint, head_tilt_joint,
        left_arm_joint_1, left_arm_joint_2, left_arm_joint_3, left_arm_joint_4, left_arm_joint_5, left_arm_joint_6, left_arm_joint_7,
        right_arm_joint_1, right_arm_joint_2, right_arm_joint_3, right_arm_joint_4, right_arm_joint_5, right_arm_joint_6, right_arm_joint_7,
        left_hand_joint_0, left_hand_joint_1, left_hand_joint_2, left_hand_joint_3, left_hand_joint_4, left_hand_joint_5, left_hand_joint_6, left_hand_joint_7, left_hand_joint_8, left_hand_joint_9, left_hand_joint_10, left_hand_joint_11, left_hand_joint_12, left_hand_joint_13, left_hand_joint_14, left_hand_joint_15,
        right_hand_joint_0, right_hand_joint_1, right_hand_joint_2, right_hand_joint_3, right_hand_joint_4, right_hand_joint_5, right_hand_joint_6, right_hand_joint_7, right_hand_joint_8, right_hand_joint_9, right_hand_joint_10, right_hand_joint_11, right_hand_joint_12, right_hand_joint_13, right_hand_joint_14, right_hand_joint_15], 
position: [0.4, 
            0.0, 1.0557, 
            1.274, 1.003, 0.119, 0.849, -0.323, 0.639, 0.254,
            -1.308, 0.925, -0.119, 0.906, -2.836, -0.760, -0.357,
            0.0, 0.0, 0.0, 0.0, 0.0, 1.551, 1.058, 1.448, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
            0.0, 0.0, 0.0, 0.0, 0.0, 1.551, 1.058, 1.448, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}"

Run Sample code (ROS 2 package)

After running Isaac Sim and the command message converter, let’s execute the following sample code based on the type of converter.

For 49 DOF position control:

The following node makes the robot’s joint positions move in a sinusoidal pattern.

ros2 run sample_etri_dualarm_ctr sample_sinusoidal_joint_pos_ctrl

The following node makes the robot move through a sequence of 5 joint positions in order.

ros2 run sample_etri_dualarm_ctr sample_sequence_joint_pos_ctr

For 7X2 DOF velocity control(two arms) + 35 DOF position control:

The following node also moves the robot’s joint positions in a sinusoidal pattern, but sends a velocity command instead.

ros2 run sample_etri_dualarm_ctr sample_sinusoidal_arm_vel_ctrl 

I will cover the details of the sample code in here.

How to Use

This chapter describes how to use ETRI’s dual hand-arm robot to your research.

Preperation

For your Let’s make a copy of etri_dualarm_ros2_ctr.usd and rename it. Since we are going to use new environment called ‘simple room’ with the robot, the name of the new file is:

etri_dualarm_simple_room_example.usd

Changing the environment and adding the objects

Let’s use the simple room from the Environment Assets. In version 2023.1.1, the USD file path for the simple room is as follows. Delete the default environment and add the simple room. Then, change the position and orientation of the robot and the room as you want.

omniverse://localhost/NVIDIA/Assets/Isaac/2023.1.1/Isaac/Environments/Simple_Room/simple_room.usd

Let’s add objects like YCB objects to the simple room. The following is the path to the USD file.

omniverse://localhost/NVIDIA/Assets/Isaac/2023.1.1/Isaac/Props/YCB/

Here are examples of using various environments from Isaac Sim’s assets, including the simple room.

We suggest you start by using the assets in Isaac Sim. Feel free to make your own custom environment.

Setting initial joint positions

The default joint positions of the robot needed to be changed depending on the robot’s task and environments. You can modify the initial joint positions using the Physics Inspector, such as the joint angles of the robot arm, the pan/tilt angles of the camera head, and the stroke length of the lifting column. Refer to the details on the Physics Inspector here.

Acknowledgements

This work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government foundation. [24ZB1200, Research of Human-centered autonomous intelligence system original technology]