autoware_shape_estimation

Purpose

This node calculates a refined object shape (bounding box, cylinder, convex hull) in which a pointcloud cluster fits according to a label.

Inner-workings / Algorithms

Fitting algorithms

• bounding box

  • L-shape fitting: See reference below for details
  • ML based shape fitting: See ML Based Shape Fitting Implementation section below for details

• cylinder

  cv::minEnclosingCircle

• convex hull

  cv::convexHull

Inputs / Outputs

Input

Output

Parameters

{{ json_to_markdown(“perception/autoware_shape_estimation/schema/shape_estimation.schema.json”) }}

ML Based Shape Implementation

The model takes a point cloud and object label(provided by camera detections/Apollo instance segmentation) as an input and outputs the 3D bounding box of the object.

ML based shape estimation algorithm uses a PointNet model as a backbone to estimate the 3D bounding box of the object. The model is trained on the NuScenes dataset with vehicle labels (Car, Truck, Bus, Trailer).

The implemented model is concatenated with STN (Spatial Transformer Network) to learn the transformation of the input point cloud to the canonical space and PointNet to predict the 3D bounding box of the object. Bounding box estimation part of Frustum PointNets for 3D Object Detection from RGB-D Data paper used as a reference.

The model predicts the following outputs for each object:

• x,y,z coordinates of the object center
• object heading angle classification result(Uses 12 bins for angle classification - 30 degrees each)
• object heading angle residuals
• object size classification result
• object size residuals

Training ML Based Shape Estimation Model

To train the model, you need ground truth 3D bounding box annotations. When using the mmdetection3d repository for training a 3D object detection algorithm, these ground truth annotations are saved and utilized for data augmentation. These annotations are used as an essential dataset for training the shape estimation model effectively.

Preparing the Dataset

Install MMDetection3D prerequisites

Step 1. Download and install Miniconda from the official website.

Step 2. Create a conda virtual environment and activate it

conda create --name train-shape-estimation python=3.8 -y
conda activate train-shape-estimation

Step 3. Install PyTorch

conda install pytorch torchvision -c pytorch

Install mmdetection3d

Step 1. Install MMEngine, MMCV, and MMDetection using MIM

pip install -U openmim
mim install mmengine
mim install 'mmcv>=2.0.0rc4'
mim install 'mmdet>=3.0.0rc5, <3.3.0'

Step 2. Install Autoware’s MMDetection3D fork

git clone https://github.com/autowarefoundation/mmdetection3d.git
cd mmdetection3d
pip install -v -e .

Preparing NuScenes dataset for training

Step 1. Download the NuScenes dataset from the official website and extract the dataset to a folder of your choice.

Note: The NuScenes dataset is large and requires significant disk space. Ensure you have enough storage available before proceeding.

Step 2. Create a symbolic link to the dataset folder

ln -s /path/to/nuscenes/dataset/ /path/to/mmdetection3d/data/nuscenes/

Step 3. Prepare the NuScenes data by running:

cd mmdetection3d
python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes --only-gt-database True

Clone Bounding Box Estimator model

git clone https://github.com/autowarefoundation/bbox_estimator.git

Split the dataset into training and validation sets

cd bbox_estimator
python3 utils/split_dbinfos.py --dataset_path /path/to/mmdetection3d/data/nuscenes --classes 'car' 'truck' 'bus' 'trailer'  --train_ratio 0.8

Training and Deploying the model

Training the model

# Detailed training options can be found in the training script
# For more details, run `python3 train.py --help`
python3 train.py --dataset_path /path/to/mmdetection3d/data/nuscenes

Deploying the model

# Convert the trained model to ONNX format
python3 onnx_converter.py --weight_path /path/to/best_checkpoint.pth --output_path /path/to/output.onnx

Give the output path of the ONNX model to the model_path parameter in the shape_estimation node launch file.

Assumptions / Known limits

TBD

References/External links

L-shape fitting implementation of the paper:

@conference{Zhang-2017-26536,
author = {Xiao Zhang and Wenda Xu and Chiyu Dong and John M. Dolan},
title = {Efficient L-Shape Fitting for Vehicle Detection Using Laser Scanners},
booktitle = {2017 IEEE Intelligent Vehicles Symposium},
year = {2017},
month = {June},
keywords = {autonomous driving, laser scanner, perception, segmentation},
}

Frustum PointNets for 3D Object Detection from RGB-D Data:

@inproceedings{qi2018frustum,
title={Frustum pointnets for 3d object detection from rgb-d data},
author={Qi, Charles R and Liu, Wei and Wu, Chenxia and Su, Hao and Guibas, Leonidas J},
booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
pages={918--927},
year={2018}
}
```

Changelog for package autoware_shape_estimation

0.43.0 (2025-03-21)

• Merge remote-tracking branch 'origin/main' into chore/bump-version-0.43
• chore: rename from [autoware.universe]{.title-ref} to [autoware_universe]{.title-ref} (#10306)
• refactor: add autoware_cuda_dependency_meta (#10073)
• Contributors: Esteve Fernandez, Hayato Mizushima, Yutaka Kondo

0.42.0 (2025-03-03)

• Merge remote-tracking branch 'origin/main' into tmp/bot/bump_version_base
• feat(autoware_utils): replace autoware_universe_utils with autoware_utils (#10191)
• Contributors: Fumiya Watanabe, 心刚

0.41.2 (2025-02-19)

• chore: bump version to 0.41.1 (#10088)
• Contributors: Ryohsuke Mitsudome

0.41.1 (2025-02-10)

0.41.0 (2025-01-29)

• Merge remote-tracking branch 'origin/main' into tmp/bot/bump_version_base
• feat(autoware_shape_estimation): tier4_debug_msgs chnaged to autoware_internal_debug_msgs in autoware_shape_estimation (#9897) feat: tier4_debug_msgs chnaged to autoware_internal_debug_msgs in files perception/autoware_shape_estimation
• refactor(autoware_tensorrt_common): multi-TensorRT compatibility & tensorrt_common as unified lib for all perception components (#9762)
  • refactor(autoware_tensorrt_common): multi-TensorRT compatibility & tensorrt_common as unified lib for all perception components
  • style(pre-commit): autofix
  • style(autoware_tensorrt_common): linting

  * style(autoware_lidar_centerpoint): typo Co-authored-by: Kenzo Lobos Tsunekawa <<kenzo.lobos@tier4.jp>> * docs(autoware_tensorrt_common): grammar Co-authored-by: Kenzo Lobos Tsunekawa <<kenzo.lobos@tier4.jp>>

  • fix(autoware_lidar_transfusion): reuse cast variable
  • fix(autoware_tensorrt_common): remove deprecated inference API

  * style(autoware_tensorrt_common): grammar Co-authored-by: Kenzo Lobos Tsunekawa <<kenzo.lobos@tier4.jp>> * style(autoware_tensorrt_common): grammar Co-authored-by: Kenzo Lobos Tsunekawa <<kenzo.lobos@tier4.jp>>

  • fix(autoware_tensorrt_common): const pointer
  • fix(autoware_tensorrt_common): remove unused method declaration
  • style(pre-commit): autofix

  * refactor(autoware_tensorrt_common): readability Co-authored-by: Kotaro Uetake <<60615504+ktro2828@users.noreply.github.com>>

  • fix(autoware_tensorrt_common): return if layer not registered

  * refactor(autoware_tensorrt_common): readability Co-authored-by: Kotaro Uetake <<60615504+ktro2828@users.noreply.github.com>>

  • fix(autoware_tensorrt_common): rename struct

  * style(pre-commit): autofix ---------Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]\@users.noreply.github.com> Co-authored-by: Kenzo Lobos Tsunekawa <<kenzo.lobos@tier4.jp>> Co-authored-by: Kotaro Uetake <<60615504+ktro2828@users.noreply.github.com>>

• fix(autoware_shape_estimation): fix bugprone-branch-clone (#9659)
  • fix: bugprone-error
  • fix: fmt
  • fix: pre-commit

  * fix: pre-commit ---------

• Contributors: Amadeusz Szymko, Fumiya Watanabe, Vishal Chauhan, kobayu858

0.40.0 (2024-12-12)

• Merge branch 'main' into release-0.40.0
• Revert "chore(package.xml): bump version to 0.39.0 (#9587)" This reverts commit c9f0f2688c57b0f657f5c1f28f036a970682e7f5.
• fix: fix ticket links in CHANGELOG.rst (#9588)
• chore(package.xml): bump version to 0.39.0 (#9587)
  • chore(package.xml): bump version to 0.39.0
  • fix: fix ticket links in CHANGELOG.rst

  * fix: remove unnecessary diff ---------Co-authored-by: Yutaka Kondo <<yutaka.kondo@youtalk.jp>>

• fix: fix ticket links in CHANGELOG.rst (#9588)
• fix(cpplint): include what you use - perception (#9569)
• refactor: correct spelling (#9528)
• 0.39.0
• update changelog
• Merge commit '6a1ddbd08bd' into release-0.39.0
• fix: fix ticket links to point to https://github.com/autowarefoundation/autoware_universe (#9304)
• fix: fix ticket links to point to https://github.com/autowarefoundation/autoware_universe (#9304)
• chore(package.xml): bump version to 0.38.0 (#9266) (#9284)
  • unify package.xml version to 0.37.0
  • remove system_monitor/CHANGELOG.rst
  • add changelog

  * 0.38.0

• refactor(cuda_utils): prefix package and namespace with autoware (#9171)
• Contributors: Esteve Fernandez, Fumiya Watanabe, M. Fatih Cırıt, Ryohsuke Mitsudome, Yutaka Kondo

0.39.0 (2024-11-25)

• Merge commit '6a1ddbd08bd' into release-0.39.0
• fix: fix ticket links to point to https://github.com/autowarefoundation/autoware_universe (#9304)
• fix: fix ticket links to point to https://github.com/autowarefoundation/autoware_universe (#9304)
• chore(package.xml): bump version to 0.38.0 (#9266) (#9284)
  • unify package.xml version to 0.37.0
  • remove system_monitor/CHANGELOG.rst
  • add changelog

  * 0.38.0

• refactor(cuda_utils): prefix package and namespace with autoware (#9171)
• Contributors: Esteve Fernandez, Yutaka Kondo

0.38.0 (2024-11-08)

• unify package.xml version to 0.37.0
• feat(autoware_shape_estimation): add reference object based corrector (#9148)
  • add object based corrector
  • apply cppcheck suggestion

  * fix typo ---------Co-authored-by: Taekjin LEE <<taekjin.lee@tier4.jp>>

• refactor(tensorrt_common)!: fix namespace, directory structure & move to perception namespace (#9099)
  • refactor(tensorrt_common)!: fix namespace, directory structure & move to perception namespace
  • refactor(tensorrt_common): directory structure
  • style(pre-commit): autofix

  * fix(tensorrt_common): correct package name for logging ---------Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]\@users.noreply.github.com> Co-authored-by: Kenzo Lobos Tsunekawa <<kenzo.lobos@tier4.jp>>

• test(autoware_shape_estimation): add unit testing for [ShapeEstimationNode]{.title-ref} (#8740) test: add unit testing for [ShapeEstimationNode]{.title-ref}
• fix(autoware_shape_estimation): fix unusedFunction (#8575)
  • fix:unusedFunction
  • fix:unusedFunction
  • fix:end of file issues

  * fix:copyright ---------

• fix(autoware_shape_estimation): resolve undefined reference to [~TrtShapeEstimator()]{.title-ref} (#8738) fix: resolve undefined reference to [~TrtShapeEstimator()]{.title-ref}
• feat(shape_estimation): add ml shape estimation (#7860)
  • feat(shape_estimation): add ml shape estimation
  • style(pre-commit): autofix
  • feat(shape_estimation): fix exceed objects
  • style(pre-commit): autofix

  * feat(shape_estimation): fix indent ---------Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]\@users.noreply.github.com>

• refactor(shape_estimation): add package name prefix of autoware_ (#7999)
  • refactor(shape_estimation): add package name prefix of autoware_
  • style(pre-commit): autofix
  • fix: mising prefix

  * fix: cmake ---------Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]\@users.noreply.github.com>

• Contributors: Amadeusz Szymko, Kaan Çolak, Kotaro Uetake, Masaki Baba, Yutaka Kondo, badai nguyen, kobayu858

0.26.0 (2024-04-03)