Reaction Analyzer

Description

The main purpose of the reaction analyzer package is to measure the reaction times of various nodes within a ROS-based autonomous driving simulation environment by subscribing to pre-determined topics. This tool is particularly useful for evaluating the performance of perception, planning, and control pipelines in response to dynamic changes in the environment, such as sudden obstacles. To be able to measure both control outputs and perception outputs, it was necessary to divide the node into two running_mode: planning_control and perception_planning.

Planning Control Mode

In this mode, the reaction analyzer creates a dummy publisher for the PredictedObjects and PointCloud2 topics. In the beginning of the test, it publishes the initial position of the ego vehicle and the goal position to set the test environment. Then, it spawns a sudden obstacle in front of the ego vehicle. After the obstacle is spawned, it starts to search reacted messages of the planning and control nodes in the pre-determined topics. When all the topics are reacted, it calculates the reaction time of the nodes and statistics by comparing reacted_times of each of the nodes with spawn_cmd_time, and it creates a csv file to store the results.

Perception Planning Mode

In this mode, the reaction analyzer reads the rosbag files which are recorded from AWSIM, and it creates a topic publisher for each topic inside the rosbag to replay the rosbag. It reads two rosbag files: path_bag_without_object and path_bag_with_object. Firstly, it replays the path_bag_without_object to set the initial position of the ego vehicle and the goal position. After spawn_time_after_init seconds , it replays the path_bag_with_object to spawn a sudden obstacle in front of the ego vehicle. After the obstacle is spawned, it starts to search the reacted messages of the perception and planning nodes in the pre-determined topics. When all the topics are reacted, it calculates the reaction time of the nodes and statistics by comparing reacted_times of each of the nodes with spawn_cmd_time, and it creates a csv file to store the results.

Point Cloud Publisher Type

To get better analyze for Perception & Sensing pipeline, the reaction analyzer can publish the point cloud messages in 3 different ways: async_header_sync_publish, sync_header_sync_publish or async_publish. (T is the period of the lidar’s output)

• async_header_sync_publish: It publishes the point cloud messages synchronously with asynchronous header times. It means that each of the lidar’s output will be published at the same time, but the headers of the point cloud messages includes different timestamps because of the phase difference.
• sync_header_sync_publish: It publishes the point cloud messages synchronously with synchronous header times. It means that each of the lidar’s output will be published at the same time, and the headers of the point cloud messages includes the same timestamps.
• async_publish: It publishes the point cloud messages asynchronously. It means that each of the lidar’s output will be published at different times.

Usage

The common parameters you need to define for both running modes are output_file_path, test_iteration, and reaction_chain list. output_file_path is the output file path is the path where the results and statistics will be stored. test_iteration defines how many tests will be performed. The reaction_chain list is the list of the pre-defined topics you want to measure their reaction times.

IMPORTANT: Ensure the reaction_chain list is correctly defined:

• For perception_planning mode, do not define Control nodes.
• For planning_control mode, do not define Perception nodes.

Prepared Test Environment

• Download the demonstration test map from the link here. After downloading, extract the zip file and use its path as [MAP_PATH] in the following commands.

Planning Control Mode

• You need to define only Planning and Control nodes in the reaction_chain list. With the default parameters, you can start to test with the following command:

ros2 launch reaction_analyzer reaction_analyzer.launch.xml running_mode:=planning_control vehicle_model:=sample_vehicle sensor_model:=sample_sensor_kit map_path:=[MAP_PATH]

After the command, the simple_planning_simulator and the reaction_analyzer will be launched. It will automatically start to test. After the test is completed, the results will be stored in the output_file_path you defined.

Perception Planning Mode

• Download the rosbag files from the Google Drive link here.
• Extract the zip file and set the path of the .db3 files to parameters path_bag_without_object and path_bag_with_object.
• You can start to test with the following command:

ros2 launch reaction_analyzer reaction_analyzer.launch.xml running_mode:=perception_planning vehicle_model:=sample_vehicle sensor_model:=awsim_labs_sensor_kit map_path:=[MAP_PATH]

• On the first run of the tool in perception_planning mode, initialization might take longer than expected. Please allow some time for the process to complete.

After the command, the e2e_simulator and the reaction_analyzer will be launched. It will automatically start to test. After the test is completed, the results will be stored in the output_file_path you defined.

Prepared Test Environment

Scene without object:

Scene object:

Custom Test Environment

If you want to run the reaction analyzer with your custom test environment, you need to redefine some of the parameters. The parameters you need to redefine are initialization_pose, entity_params, goal_pose, and topic_publisher ( for perception_planning mode) parameters.

• To set initialization_pose, entity_params, goal_pose:
• Run the AWSIM environment. Tutorial for AWSIM can be found here.
• Run the e2e_simulator with the following command:

ros2 launch autoware_launch e2e_simulator.launch.xml vehicle_model:=sample_vehicle sensor_model:=awsim_labs_sensor_kit map_path:=[MAP_PATH]

• After the EGO is initialized, you can position the ego vehicle in the desired location using the 2D Pose Estimate button in RViz.
• After the EGO located in desired position, please localize the dummy obstacle by using the traffic controller. You can access the traffic control section by pressing the ‘ESC’ key.

After localize EGO and dummy vehicle, we should write the positions of these entities in the map frame in reaction_analyzer.param.yaml. To achieve this:

• Get initialization pose from /awsim/ground_truth/vehicle/pose topic.
• Get entity params from /perception/object_recognition/objects topic.
• Get goal pose from /planning/mission_planning/goal topic.

PS: initialization_pose is only valid for planning_control mode.

• After the parameters were noted, we should record the rosbags for the test. To record the rosbags, you can use the following command:

ros2 bag record --all

• You should record two rosbags: one without the object and one with the object. You can use the traffic controller to spawn the object in front of the EGO vehicle or remove it.

NOTE: You should record the rosbags in the same environment with the same position of the EGO vehicle. You don’t need to run Autoware while recording.

• After you record the rosbags, you can set the path_bag_without_object and path_bag_with_object parameters with the paths of the recorded rosbags.

Results

The results will be stored in the csv file format and written to the output_file_path you defined. It shows each pipeline of the Autoware by using header timestamp of the messages, and it reports Node Latency, Pipeline Latency, and Total Latency for each of the nodes.

• Node Latency: The time difference between previous and current node’s reaction timestamps. If it is the first node in the pipeline, it is same as Pipeline Latency.
• Pipeline Latency: The time difference between published time of the message and pipeline header time.
• Total Latency: The time difference between the message’s published timestamp and the spawn obstacle command sent timestamp.

Parameters

Limitations

• Reaction analyzer has some limitation like PublisherMessageType, SubscriberMessageType and ReactionType. It is currently supporting following list:

• Publisher Message Types:

  • sensor_msgs/msg/PointCloud2
  • sensor_msgs/msg/CameraInfo
  • sensor_msgs/msg/Image
  • geometry_msgs/msg/PoseWithCovarianceStamped
  • sensor_msgs/msg/Imu
  • autoware_vehicle_msgs/msg/ControlModeReport
  • autoware_vehicle_msgs/msg/GearReport
  • autoware_vehicle_msgs/msg/HazardLightsReport
  • autoware_vehicle_msgs/msg/SteeringReport
  • autoware_vehicle_msgs/msg/TurnIndicatorsReport
  • autoware_vehicle_msgs/msg/VelocityReport

• Subscriber Message Types:

  • sensor_msgs/msg/PointCloud2
  • autoware_perception_msgs/msg/DetectedObjects
  • autoware_perception_msgs/msg/TrackedObjects
  • autoware_perception_msgs/msg/PredictedObject
  • autoware_planning_msgs/msg/Trajectory
  • autoware_control_msgs/msg/Control

• Reaction Types:

  • FIRST_BRAKE
  • SEARCH_ZERO_VEL
  • SEARCH_ENTITY

Future improvements

• The reaction analyzer can be improved by adding more reaction types. Currently, it is supporting only FIRST_BRAKE, SEARCH_ZERO_VEL, and SEARCH_ENTITY reaction types. It can be extended by adding more reaction types for each of the message types.

Changelog for package reaction_analyzer

0.43.0 (2025-03-21)

• Merge remote-tracking branch 'origin/main' into chore/bump-version-0.43
• chore: rename from [autoware.universe]{.title-ref} to [autoware_universe]{.title-ref} (#10306)
• Contributors: Hayato Mizushima, Yutaka Kondo

0.42.0 (2025-03-03)

• Merge remote-tracking branch 'origin/main' into tmp/bot/bump_version_base
• feat(autoware_utils): replace autoware_universe_utils with autoware_utils (#10191)
• fix: add missing includes to autoware_universe_utils (#10091)
• Contributors: Fumiya Watanabe, Ryohsuke Mitsudome, 心刚

0.41.2 (2025-02-19)

• chore: bump version to 0.41.1 (#10088)
• Contributors: Ryohsuke Mitsudome

0.41.1 (2025-02-10)

0.41.0 (2025-01-29)

0.40.0 (2024-12-12)

• Merge branch 'main' into release-0.40.0
• Revert "chore(package.xml): bump version to 0.39.0 (#9587)" This reverts commit c9f0f2688c57b0f657f5c1f28f036a970682e7f5.
• fix: fix ticket links in CHANGELOG.rst (#9588)
• chore(package.xml): bump version to 0.39.0 (#9587)
  • chore(package.xml): bump version to 0.39.0
  • fix: fix ticket links in CHANGELOG.rst

  * fix: remove unnecessary diff ---------Co-authored-by: Yutaka Kondo <<yutaka.kondo@youtalk.jp>>

• fix: fix ticket links in CHANGELOG.rst (#9588)
• fix(cpplint): include what you use - tools (#9574)
• 0.39.0
• update changelog
• fix: fix ticket links to point to https://github.com/autowarefoundation/autoware_universe (#9304)
• fix: fix ticket links to point to https://github.com/autowarefoundation/autoware_universe (#9304)
• chore(package.xml): bump version to 0.38.0 (#9266) (#9284)
  • unify package.xml version to 0.37.0
  • remove system_monitor/CHANGELOG.rst
  • add changelog

  * 0.38.0

• Contributors: Esteve Fernandez, Fumiya Watanabe, M. Fatih Cırıt, Ryohsuke Mitsudome, Yutaka Kondo

0.39.0 (2024-11-25)

• fix: fix ticket links to point to https://github.com/autowarefoundation/autoware_universe (#9304)
• fix: fix ticket links to point to https://github.com/autowarefoundation/autoware_universe (#9304)
• chore(package.xml): bump version to 0.38.0 (#9266) (#9284)
  • unify package.xml version to 0.37.0
  • remove system_monitor/CHANGELOG.rst
  • add changelog

  * 0.38.0

• Contributors: Esteve Fernandez, Yutaka Kondo

0.38.0 (2024-11-08)

• unify package.xml version to 0.37.0
• docs(reaction_analyzer): update bag files and the README (#8633)
  • docs(reaction_analyzer): update bag files and the README
• fix(reaction_analyzer): fix include hierarchy of tf2_eigen (#8663) Fixed include hierarchy of tf2_eigen
• fix(reaction_analyzer): fix variableScope (#8450)
  • fix:variableScope

  * fix:clang format ---------

• fix(reaction_analyzer): fix constVariableReference (#8063)
  • fix:constVariableReference
  • fix:constVariableReference
  • fix:constVariableReference

  * fix:suppression constVariableReference ---------

• refactor(universe_utils/motion_utils)!: add autoware namespace (#7594)
• refactor(motion_utils)!: add autoware prefix and include dir (#7539) refactor(motion_utils): add autoware prefix and include dir
• feat(autoware_universe_utils)!: rename from tier4_autoware_utils (#7538) Co-authored-by: kosuke55 <<kosuke.tnp@gmail.com>>
• feat!: replace autoware_auto_msgs with autoware_msgs for tools (#7250) Co-authored-by: Cynthia Liu <<cynthia.liu@autocore.ai>> Co-authored-by: NorahXiong <<norah.xiong@autocore.ai>> Co-authored-by: beginningfan <<beginning.fan@autocore.ai>>
• feat(reaction_analyzer): add reaction anaylzer tool to measure end-to-end delay in sudden obstacle braking response (#5954)
  • feat(reaction_analyzer): add reaction anaylzer tool to measure end-to-end delay in sudden obstacle braking response
  • feat: implement message_filters package, clean up
  • feat: update style and readme
  • feat: add predicted path for the PredictedObject and add publish_only_pointcloud_with_object
  • feat: add wrong initialize localization protection, improve code readability
  • feat: launch occupancy_grid_map from reaction analyzer's own launch file
  • feat: update
  • feat: change function names
  • feat: update
  • feat: improve style, change csv output stringstream
  • fix: ci/cd
  • feat: update for new sensor setup, fix bug, optimize code, show pipeline latency, update readme
  • fix: container die problem

  * feat: update stats, check path param, add marker, warn user for wrong reaction_chain ---------

• Contributors: Batuhan Beytekin, Berkay Karaman, Kosuke Takeuchi, Ryohsuke Mitsudome, SakodaShintaro, Takayuki Murooka, Yutaka Kondo, kobayu858

0.26.0 (2024-04-03)