hri_fullbody

Overview

:warning: some of the links are yet to be updated and may be pointing to the original ROS page. As soon as all the involved components will be officially documented for ROS 2 as well, we will update this document.

hri_fullbody is a ROS4HRI-compatible 3D body pose estimation node.

It is built on top of Google Mediapipe 3D body pose estimation.

The node provides the 3D pose estimation for the detected humans in the scene, implementing a robust solution to self-occlusions.

This node can be used in two different modes: single body and multi-body. In the former case, the node performs the whole body detection pipeline, publishing information under the ROS4HRI naming convention regarding the body id (on the /humans/bodies/tracked topic), the body bounding box, and the jointstate of the body’s skeleton. In the latter, the node only publishes jointstate information, expecting body detection to be performed by another node and subscribing to /humans/bodies/tracked to read the ids of the currently tracked bodies.

These two modes are required since Mediapipe body estimation is not able to estimate more than one body pose per frame. Therefore, in case we want more than one body pose estimated, the adopted strategy is to rely on some other package implementing a human body detection (and tracking in the best case) algorithm. Such package should provide the hri_fullbody detect node the IDs of the detected bodies and the relative images, cropped from the original image following the detected bounding boxes.

To estimate the body position, the node does not need a RGB-D camera, only RGB is required. However, using RGB-D camera provides a more accurate depth estimation.

Important: to estimate the body depth without using a depth sensor, a calibrated RGB camera is required. You can follow this tutorial to properly calibrate your camera.

Launch

ros2 launch hri_fullbody hri_fullbody_with_args.launch.py <parameters>

ROS API

Parameters

Node parameters:

• single_body (default: True): whether or not running in single body mode (see above for the single body vs multi-body modes).
• use_depth (default: False): whether or not to rely on depth images for estimating body movement in the scene. When this is False, the node estimates the body position in the scene solving a P6P problem for the face and approximating the position from this, using pinhole camera model geometry.
• stickman_debug (default: False): whether or not to publish frames representing the body skeleton directly using the raw results from mediapipe 3D body pose estimation. These debug frames are not oriented to align with the body links (ie, only the 3D location of the frame is useful).
• human_description_<body_id> (read-only): the URDF generated for body_<body_id>. The node generates a URDF for each detected body and uses them to perform kinematically-consistent 3D body pose estimation.

hri_fullbody.launch parameters:

• single_body (default: True): equivalent to single_body node parameter.
• use_depth (default: False): equivalent to use_depth node parameter.
• stickman_debug (default: False): equivalent to stickman_debug node parameter.
• rgb_camera (default: ` `): rgb camera topics namespace.
• rgb_camera_topic (default: $(arg rgb_camera)/image_raw): rgb camera raw image topic.
• rgb_camera_info (default: $(arg rgb_camera)/camera_info): rgb camera info topic.
• depth_camera (default: ` `): depth camera topics namespace.
• depth_camera_topic (default: $(arg depth_camera)/image_rect_raw): depth camera rectified raw image topic.
• depth_camera_info (default: $(arg depth_camera)/camera_info): depth camera info topic.

Topics

hri_fullbody follows the ROS4HRI conventions (REP-155). In particular, refer to the REP to know the list and position of the 2D/3D skeleton points published by the node.

Subscribed topics

• /camera_info (sensor_msgs/CameraInfo): rgb camera meta information
• /depth_image (sensor_msgs/Image): depth image used to estimate the 3D body position with respect to the camera.
• /depth_info (sensor_msgs/CameraInfo): depth camera meta information

Single body mode only:

• /image (sensor_msgs/Image): rgb image, processed for body detection and 3D body pose estimation.

Multi-body mode only:

• /humans/bodies/tracked (hri_msgs/IdsList): list of the bodies currently detected.
• /humans/bodies/<body_id>/cropped (sensor_msgs/Image): image used to estimate the 3D body pose.
• /humans/bodies/<body_id>/roi (hri_msgs/NormalizedRegionOfInterest2D): body bounding box in normalized image coordinates.

Published topics

• /humans/bodies/<body_id>/skeleton2d (hri_msgs/Skeleton2D): detected 2D skeleton points.
• /humans/bodies/<body_id>/joint_states (sensor_msgs/JointState): skeleton joints state.
• /humans/bodies/<body_id>/position: (geometry_msgs/PointStamped): filtered body position, representing the point between the hips of the tracked body. Only published when use_depth = True.
• /humans/bodies/<body_id>/velocity: (geometry_msgs/TwistStamped): filtered body velocity. Only published when use_depth = True.

Single body mode only:

• /humans/bodies/tracked (hri_msgs/IdsList): list of the bodies currently detected. There will be only one body in the list.
• /humans/bodies/<body_id>/roi (hri_msgs/NormalizedRegionOfInterest2D): body bounding box in normalized image coordinates.

Visualization

It is possible to visualize the results of the body pose estimation in rviz using the hri_rviz Skeleton plugin. A visualization example is reported in the image above.

Changelog for package hri_fullbody

2.2.2 (2024-07-04)

• linting
• Contributors: Séverin Lemaignan

2.2.1 (2024-07-04)

• launch: use get_pal_configuration from launch_pal
• Contributors: Séverin Lemaignan

2.2.0 (2024-05-27)

• Updated topics naming convention in source code and configuration file. Fiexed bugs
• add configuration management
• removed unused import os
• Contributors: Luka Juricic, Raquel Ros, lorenzoferrini

2.1.1 (2024-05-21)

• removed old ros command
• Contributors: lorenzoferrini

2.1.0 (2024-05-07)

• Support 32bits depth
• Publish bodies' URDF on a topic instead of a parameter
• Contributors: ferrangebelli

2.0.1 (2024-04-24)

• add module
• Contributors: lorenzoferrini

2.0.0 (2024-02-14)

• removing bridging bug when using CvBridge.imgmsg_to_cv2 function without specifying the desired encoding to "bgr8", the returned image only had two channels if it's coming from the usb_cam node. Fixed the issue
• ros 2 porting
• clarify documentation for multibody case
• fix multibody diagnostics bug
• Contributors: Luka Juricic, lorenzoferrini

0.1.10 (2023-07-14)

• reset mediapipe backend in case of matrix inversion error
• Contributors: Luka Juricic

0.1.9 (2023-07-05)

• fix bug in left ear tragion publish
• change RoI message type to hri_msgs/NormalizedRegionOfInterest2D
• Contributors: Luka Juricic

0.1.8 (2023-05-23)

• disable publishing of face_XXX and gaze_XXX frames See comment in code for rationale
• Contributors: Séverin Lemaignan

0.1.7 (2023-05-18)

• diagnostic_msgs: exec_depend -> depend
• Contributors: Séverin Lemaignan

0.1.6 (2023-05-18)

• add diagnostics
• Contributors: lukajuricic

0.1.5 (2023-03-08)

• ensure mediapipe can not be called from 2 threads in parallel This would cause internal mediapipe errors related to non-monotonic timestamps
• Contributors: Séverin Lemaignan

0.1.4 (2023-02-03)

• Merge branch 'devel' into 'master' Revert "have threshold params" See merge request ros4hri/hri_fullbody!7
• Merge branch 'master' into 'devel' # Conflicts: # src/hri_fullbody/fullbody_detector.py
• Revert "have threshold params" This reverts commit 78264248652879a1b572397e8edeee6202685bc6 and commit 5cc5c34f7a93f8030623ff78945095778374cd36.
• Merge branch 'devel' into 'master' Introduced requirements.txt See merge request ros4hri/hri_fullbody!6
• Merge branch 'uncalibrated_monocular_camera' into 'master' Correctly managing uncalibrated camera See merge request ros4hri/hri_fullbody!5
• Introduced requirements.txt Introduced requirements.txt. Currently, the only requirement specified is ikpy. Using the requirements.txt file syntax, it was possible to specify the requested version, that is, 3.2.2
• Enabled human jointstate estimation for uncalibrated RGB cameras It was previously not possible to estimate human joint angles when the RGB camera was not calibrated. Additionally, the node is now displaying a warning message when the received camera intrinsic matrix has all zero values, which usually means that it needs to be calibrated.
• [Doc] Specified camera calibration requirement It is now specified in the README.md file that camera calibration is required to estimate body depth without using a depth sensor.
• REP-155 compliance: skeleton must be published on /skeleton2d topic
• Merge branch 'master' of gitlab:ros4hri/hri_fullbody
• have threshold params
• have threshold params
• generate only letter-based IDs (for consistency with other nodes)
• code formatting
• Contributors: Séverin Lemaignan, lorenzoferrini, saracooper

0.1.3 (2022-09-27)

• add dependency on ikpy 3.2.2
• Contributors: Séverin Lemaignan

0.1.2 (2022-09-02)

• update to hri_msgs-0.8.0
• Contributors: Séverin Lemaignan

0.1.0 (2022-08-02)

• Body position estimation without accessing depth information
• clean up launch files
• multibody depth detection
• Depth information for multipeople 3D pose estimation
• remove RegionOfInterestStamped + remove unused imports
• Multi-people detection We are currently working on the implementation of multi-people detection, usign a two stage approach: first detecting the people in the scene through a face detector, then estimating the holistic model of each one of the people detected. Work in progress!
• Killing robot_state_publishers with rosnode kill Using the robot_state_publisher_body_id.terminate method was not updating the rosnode list (however, it was actually terminating the process).
• Defined a consistency check for the face bounding box Testing the ROS4HRI plugin we are developing, it came out that there was a consistency problem in the published face bounding box. A consistency check on the published bounding box avoids this problem, verifying that its size matches the requirements imposed by the current image height and width.
• More debug parameters in launch file Updated the launch file to include two more debug parameters: stickman, to visualize simple frame representation using 3D Mediapipe estimation without inverse kinematic, and textual, to let the node print in the terminal information about the detected faces and bodies.
• One urdf per detected body hri_fullbody node now creating and managing one urdf file for each detected body.
• install as well the rviz config
• completed launch file to also start the webcam
• do not use CompressedImage directly
• import protobuf_to_dict The library is in the public domain: https://github.com/benhodgson/protobuf-to-dict
• Added README.md and video example of the expected result for human model visualization on RViz
• Now publishing Faces ROI, Bodies ROI, human model jointstate and 2D Skeleton message from ros4hri_msgs. Human model visualization via RViz included
• Code refined, HolistiDtector class defined. Face Detection message publishing according to ROS4HRI protocol
• Holistic node publishing skeleton message with chance of visual and textual debugging
• Contributors: Séverin Lemaignan, lorenzoferrini