Planning Evaluator

Purpose

This package provides nodes that generate metrics to evaluate the quality of planning and control.

Inner-workings / Algorithms

The evaluation node calculates metrics each time it receives a trajectory T(0). Metrics are calculated using the following information:

• the trajectory T(0) itself.
• the previous trajectory T(-1).
• the reference trajectory assumed to be used as the reference to plan T(0).
• the current ego pose.
• the current ego odometry.
• the set of objects in the environment.

These information are maintained by an instance of class MetricsCalculator which is also responsible for calculating metrics.

Stat

Each metric is calculated using a autoware_utils::Accumulator instance which contains the minimum, maximum, and mean values calculated for the metric as well as the number of values measured.

Metric calculation and adding more metrics

All possible metrics are defined in the Metric enumeration defined include/planning_evaluator/metrics/metric.hpp. This file also defines conversions from/to string as well as human readable descriptions to be used as header of the output file.

The MetricsCalculator is responsible for calculating metric statistics through calls to function:

Accumulator<double> MetricsCalculator::calculate(const Metric metric, const Trajectory & traj) const;

Adding a new metric M requires the following steps:

• metrics/metric.hpp: add M to the enum, to the from/to string conversion maps, and to the description map.
• metrics_calculator.cpp: add M to the switch/case statement of the calculate function.
• Add M to the selected_metrics parameters.

Inputs / Outputs

Inputs

Outputs

Each metric is published on a topic named after the metric name.

If output_metrics = true, the evaluation node writes the statics of the metrics measured during its lifetime to <ros2_logging_directory>/autoware_metrics/<node_name>-<time_stamp>.json when shut down.

Parameters

{{ json_to_markdown(“evaluator/autoware_planning_evaluator/schema/autoware_planning_evaluator.schema.json”) }}

Assumptions / Known limits

There is a strong assumption that when receiving a trajectory T(0), it has been generated using the last received reference trajectory and objects. This can be wrong if a new reference trajectory or objects are published while T(0) is being calculated.

Precision is currently limited by the resolution of the trajectories. It is possible to interpolate the trajectory and reference trajectory to increase precision but would make computation significantly more expensive.

Future extensions / Unimplemented parts

• Use Route or Path messages as reference trajectory.
• RSS metrics (done in another node https://tier4.atlassian.net/browse/AJD-263).
• Add option to publish the min and max metric values. For now only the mean value is published.
• motion_evaluator_node.
  • Node which constructs a trajectory over time from the real motion of ego.
  • Only a proof of concept is currently implemented.

Changelog for package autoware_planning_evaluator

0.43.0 (2025-03-21)

• Merge remote-tracking branch 'origin/main' into chore/bump-version-0.43
• chore: rename from [autoware.universe]{.title-ref} to [autoware_universe]{.title-ref} (#10306)
• feat(control_evaluator): add a new stop_deviation metric (#10246)
  • add metric of stop_deviation
  • fix bug
  • remove unused include.
  • add unit test and schema
  • pre-commit

  * update planning_evaluator schema ---------Co-authored-by: t4-adc <<grp-rd-1-adc-admin@tier4.jp>>

• Contributors: Hayato Mizushima, Kem (TiankuiXian), Yutaka Kondo

0.42.0 (2025-03-03)

• Merge remote-tracking branch 'origin/main' into tmp/bot/bump_version_base
• feat(autoware_utils): replace autoware_universe_utils with autoware_utils (#10191)
• feat(autoware_planning_evaluator): add resampled_relative_angle metrics (#10020)
  • feat(autoware_planning_evaluator): add new large_relative_angle metrics
  • fix copyright and vehicle_length_m
  • style(pre-commit): autofix
  • del: resample trajectory
  • del: traj points check
  • rename msg and speed optimization
  • style(pre-commit): autofix
  • add unit_test and fix resample_relative_angle
  • style(pre-commit): autofix
  • include tuple to test
  • target two point, update unit test value
  • fix abs

  * fix for loop bag and primitive type ---------Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]\@users.noreply.github.com>

• Contributors: Fumiya Watanabe, Kazunori-Nakajima, 心刚

0.41.2 (2025-02-19)

• chore: bump version to 0.41.1 (#10088)
• Contributors: Ryohsuke Mitsudome

0.41.1 (2025-02-10)

0.41.0 (2025-01-29)

• Merge remote-tracking branch 'origin/main' into tmp/bot/bump_version_base
• feat: tier4_debug_msgs changed to autoware_internal_debug_msgs in fil… (#9859) feat: tier4_debug_msgs changed to autoware_internal_debug_msgs in files evaluator/autoware_planning_evaluator
• fix(planning_evaluator): update lateral_trajectory_displacement to absolute value (#9696)
  • fix(planning_evaluator): update lateral_trajectory_displacement to absolute value

  * style(pre-commit): autofix ---------Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]\@users.noreply.github.com>

• fix(autoware_planning_evaluator): rename lateral deviation metrics (#9801)
  • refactor(planning_evaluator): rename and add lateral trajectory displacement metrics

  * fix typo ---------

• feat(planning_evaluator): add evaluation feature of trajectory lateral displacement (#9718)
  • feat(planning_evaluator): add evaluation feature of trajectory lateral displacement
  • feat(metrics_calculator): implement lookahead trajectory calculation and remove deprecated method

  * fix(planning_evaluator): rename lateral_trajectory_displacement to trajectory_lateral_displacement for consistency ---------

• fix(autoware_planning_evaluator): fix bugprone-exception-escape (#9730) fix: bugprone-exception-escape
• feat(planning_evaluator): add lateral trajectory displacement metrics (#9674)
  • feat(planning_evaluator): add nearest pose deviation msg
  • update comment contents
  • update variable name

  * Revert "update variable name" This reverts commit ee427222fcbd2a18ffbc20fecca3ad557f527e37.

  • move lateral_trajectory_displacement position

  * prev.dist -> prev_lateral_deviation ---------

• Contributors: Fumiya Watanabe, Kazunori-Nakajima, Kyoichi Sugahara, Vishal Chauhan, kobayu858

0.40.0 (2024-12-12)

• Merge branch 'main' into release-0.40.0
• Revert "chore(package.xml): bump version to 0.39.0 (#9587)" This reverts commit c9f0f2688c57b0f657f5c1f28f036a970682e7f5.
• fix: fix ticket links in CHANGELOG.rst (#9588)
• chore(package.xml): bump version to 0.39.0 (#9587)
  • chore(package.xml): bump version to 0.39.0
  • fix: fix ticket links in CHANGELOG.rst

  * fix: remove unnecessary diff ---------Co-authored-by: Yutaka Kondo <<yutaka.kondo@youtalk.jp>>

• fix: fix ticket links in CHANGELOG.rst (#9588)
• fix(cpplint): include what you use - evaluator (#9566)
• feat(planning_evaluator): add a trigger to choice whether to output metrics to log folder (#9476)
  • tmp save
  • planning_evaluator build ok, test it.
  • add descriptions to output.
  • pre-commit.
  • add parm to launch file.
  • move output_metrics from config to launch file.

  * fix unit test bug. ---------

• refactor(evaluators, autoware_universe_utils): rename Stat class to Accumulator and move it to autoware_universe_utils (#9459)
  • add Accumulator class to autoware_universe_utils
  • use Accumulator on all evaluators.
  • pre-commit
  • found and fixed a bug. add more tests.
  • pre-commit

  * Update common/autoware_universe_utils/include/autoware/universe_utils/math/accumulator.hpp Co-authored-by: Kosuke Takeuchi <<kosuke.tnp@gmail.com>> ---------Co-authored-by: Kosuke Takeuchi <<kosuke.tnp@gmail.com>>

• feat(planning_evaluator): add processing time pub (#9334)
• 0.39.0
• update changelog
• Merge commit '6a1ddbd08bd' into release-0.39.0
• fix: fix ticket links to point to https://github.com/autowarefoundation/autoware_universe (#9304)
• fix(evaluator): missing dependency in evaluator components (#9074)
• fix: fix ticket links to point to https://github.com/autowarefoundation/autoware_universe (#9304)
• feat(tier4_metric_msgs): apply tier4_metric_msgs for scenario_simulator_v2_adapter, control_evaluator, planning_evaluator, autonomous_emergency_braking, obstacle_cruise_planner, motion_velocity_planner, processing_time_checker (#9180)
  • first commit
  • fix building errs.
  • change diagnostic messages to metric messages for publishing decision.
  • fix bug about motion_velocity_planner
  • change the diagnostic msg to metric msg in autoware_obstacle_cruise_planner.
  • tmp save for planning_evaluator
  • change the topic to which metrics published to.
  • fix typo.
  • remove unnesessary publishing of metrics.
  • mke planning_evaluator publish msg of MetricArray instead of Diags.
  • update aeb with metric type for decision.
  • fix some bug
  • remove autoware_evaluator_utils package.
  • remove diagnostic_msgs dependency of planning_evaluator
  • use metric_msgs for autoware_processing_time_checker.
  • rewrite diagnostic_convertor to scenario_simulator_v2_adapter, supporting metric_msgs.
  • pre-commit and fix typo
  • publish metrics even if there is no metric in the MetricArray.
  • modify the metric name of processing_time.
  • update unit test for test_planning/control_evaluator

  * manual pre-commit ---------

• chore(package.xml): bump version to 0.38.0 (#9266) (#9284)
  • unify package.xml version to 0.37.0
  • remove system_monitor/CHANGELOG.rst
  • add changelog

  * 0.38.0

• Contributors: Esteve Fernandez, Fumiya Watanabe, Kazunori-Nakajima, Kem (TiankuiXian), M. Fatih Cırıt, Ryohsuke Mitsudome, Yutaka Kondo, ぐるぐる

0.39.0 (2024-11-25)

• Merge commit '6a1ddbd08bd' into release-0.39.0
• fix: fix ticket links to point to https://github.com/autowarefoundation/autoware_universe (#9304)
• fix: fix ticket links to point to https://github.com/autowarefoundation/autoware_universe (#9304)
• feat(tier4_metric_msgs): apply tier4_metric_msgs for scenario_simulator_v2_adapter, control_evaluator, planning_evaluator, autonomous_emergency_braking, obstacle_cruise_planner, motion_velocity_planner, processing_time_checker (#9180)
  • first commit
  • fix building errs.
  • change diagnostic messages to metric messages for publishing decision.
  • fix bug about motion_velocity_planner
  • change the diagnostic msg to metric msg in autoware_obstacle_cruise_planner.
  • tmp save for planning_evaluator
  • change the topic to which metrics published to.
  • fix typo.
  • remove unnesessary publishing of metrics.
  • mke planning_evaluator publish msg of MetricArray instead of Diags.
  • update aeb with metric type for decision.
  • fix some bug
  • remove autoware_evaluator_utils package.
  • remove diagnostic_msgs dependency of planning_evaluator
  • use metric_msgs for autoware_processing_time_checker.
  • rewrite diagnostic_convertor to scenario_simulator_v2_adapter, supporting metric_msgs.
  • pre-commit and fix typo
  • publish metrics even if there is no metric in the MetricArray.
  • modify the metric name of processing_time.
  • update unit test for test_planning/control_evaluator

  * manual pre-commit ---------

• chore(package.xml): bump version to 0.38.0 (#9266) (#9284)
  • unify package.xml version to 0.37.0
  • remove system_monitor/CHANGELOG.rst
  • add changelog

  * 0.38.0

• Contributors: Esteve Fernandez, Kem (TiankuiXian), Yutaka Kondo

0.38.0 (2024-11-08)

• unify package.xml version to 0.37.0
• refactor(autoware_planning_evaluator): devops node dojo (#8746)
• feat(motion_velocity_planner,planning_evaluator): add stop, slow_down diags (#8503)
  • tmp save.
  • publish diagnostics.
  • move clearDiagnostics func to head
  • change to snake_names.
  • remove a change of launch.xml
  • pre-commit run -a
  • publish diagnostics on node side.
  • move empty checking out of 'get_diagnostics'.
  • remove get_diagnostics; change reason str.
  • remove unused condition.

  * Update planning/motion_velocity_planner/autoware_motion_velocity_planner_node/src/planner_manager.cpp Co-authored-by: Kosuke Takeuchi <<kosuke.tnp@gmail.com>> * Update planning/motion_velocity_planner/autoware_motion_velocity_planner_node/src/planner_manager.cpp Co-authored-by: Kosuke Takeuchi <<kosuke.tnp@gmail.com>> ---------Co-authored-by: Kosuke Takeuchi <<kosuke.tnp@gmail.com>>

• fix(autoware_planning_evaluator): fix unreadVariable (#8352)
  • fix:unreadVariable

  * fix:unreadVariable ---------

• feat(evalautor): rename evaluator diag topics (#8152)
  • feat(evalautor): rename evaluator diag topics

  * perception ---------

• refactor(autoware_universe_utils): changed the API to be more intuitive and added documentation (#7443)
  • refactor(tier4_autoware_utils): Changed the API to be more intuitive and added documentation.
  • use raw shared ptr in PollingPolicy::NEWEST
  • update
  • fix

  * Update evaluator/autoware_control_evaluator/include/autoware/control_evaluator/control_evaluator_node.hpp Co-authored-by: danielsanchezaran <<daniel.sanchez@tier4.jp>> ---------Co-authored-by: danielsanchezaran <<daniel.sanchez@tier4.jp>>

• feat(cruise_planner,planning_evaluator): add cruise and slow down diags (#7960)
  • add cruise and slow down diags to cruise planner
  • add cruise types

  * adjust planning eval ---------

• feat(planning_evaluator,control_evaluator, evaluator utils): add diagnostics subscriber to planning eval (#7849)
  • add utils and diagnostics subscription to planning_evaluator
  • add diagnostics eval

  * fix input diag in launch ---------Co-authored-by: kosuke55 <<kosuke.tnp@gmail.com>>

• feat(planning_evaluator): add planning evaluator polling sub (#7827)
  • WIP add polling subs
  • WIP
  • update functions
  • remove semicolon

  * use last data for modified goal ---------

• feat(planning_evaluator): add lanelet info to the planning evaluator (#7781) add lanelet info to the planning evaluator
• refactor(universe_utils/motion_utils)!: add autoware namespace (#7594)
• refactor(motion_utils)!: add autoware prefix and include dir (#7539) refactor(motion_utils): add autoware prefix and include dir
• feat(autoware_universe_utils)!: rename from tier4_autoware_utils (#7538) Co-authored-by: kosuke55 <<kosuke.tnp@gmail.com>>
• feat(planning_evaluator): rename to include/autoware/{package_name} (#7518)
  • fix

  * fix

• Contributors: Kosuke Takeuchi, Takayuki Murooka, Tiankui Xian, Yukinari Hisaki, Yutaka Kondo, danielsanchezaran, kobayu858, odra

0.26.0 (2024-04-03)