The reference_system package

The reference_system package provides reusable base components (nodes) that allow users to construct any number of “reference systems” to replicate real-world scenarios on real-world hardware.

The purpose of these components are not to provide the most efficient way of writing a node but rather a reusable and repeatible way of writing a node so that it can be used to reliably test various key performance indicators later on.

Base node types

Most real-world systems can be boiled down to only a handful of base node “types” that are then repeated to make the real-world system. This does not cover all possible node types, however it allows for numerous complicated systems to be developed using the same base building blocks.

1. Sensor Node
   • input node to system
   • one publisher, zero subscribers
   • publishes message cyclically at some fixed frequency
2. Transform Node
   • one subscriber, one publisher
   • starts processing for N milliseconds after a message is received
   • publishes message after processing is complete
3. Fusion Node
   • 2 subscribers, one publisher
   • starts processing for N milliseconds after a message is received from all subscriptions
   • publishes message after processing is complete
4. Cyclic Node
   • N subscribers, one publisher
   • cyclically processes all received messages since the last cycle for N milliseconds
   • publishes message after processing is complete
5. Command Node
   • prints output stats everytime a message is received
6. Intersection Node
   • behaves like N transform nodes
   • N subscribers, N publisher bundled together in one-to-one connections
   • starts processing on connection where sample was received
   • publishes message after processing is complete

These basic building-block nodes can be mixed-and-matched to create quite complex systems that replicate real-world scenarios to benchmark different configurations against each other.

New base node types can be added if necessary.

Testing and Dependencies

Common benchmarking scripts are provided within the reference_system/reference_system_py directory which is a python module itself. The methods and tools provided there can assist with running standardized benchmarking tests and with generating reports as well. See the autoware_reference_system for an example

Unit and integration tests have also been written for the reference_system package and can be found within the test directory. If a new system type is to be added, new unit and integration tests should also be added as well.

Changelog for package reference_system

v1.0.0

• Add first changelog
• Bump version of reference_system packages to 1.0.0
• Update reference_system docs, logic in various places
• Migrate benchmark scripts to python
• [91] skip node graph in reference_system tests for now
• [91] add unit and integration tests for the reference system, fix some bugs found by tests
• Added callback group per subscription in Intersection node.
• patch version bump to include mutex for prints
• Guard cout with mutex
• initial release for each package
• Add missing const keywords
• add std trace type, generate summary report
• Fix latency calculation error, adjust table header
• Uncrustify code
• Add advanced dropped sample statistics for each node
• Add dropped samples statistics
• Add sequence number, behavior planner period, timestamp relates to callback startup
• Add more advanced statistics for latency
• Lint and uncrustify
• Add intersection node docu
• Add intersection node
• fix cpp header guards, timing bug
• Uncrustify and adjust number cruncher
• Cyclic node has number crunching
  • Cyclic node with cyclic trigger implemented
  • Rename Reactor node into Cyclic node
  • Remove time diff check from fusion node
  • Number cruncher uses upper limit instead of timeout, add number cruncher benchmark to help find the best limits for a system
• Add Fusion node max input timediff and fail when it is exceeded
• rename nodes, update docs, add platform test
• clean up cmake, reorg READMEs
• seperate out reference_system vs autoware specifics
• fix uncrustify errors
• Add more fine grained timing configuration
  • fix README links, add requirements doc, linter cleanups
• fix lint errors
• Apply suggestions from code review
• Add benchmark comments and add custom timings in system builder
• Add benchmark timing
• Add benchmark mode
• Uncomment code for single threaded executors
  • update README, clean up system naming
• reorg, lint and clean up
• Move processing times into config
• Remove warnings
• Write number crunch result at first index
• Rename SampleType to SampleTypePointer to make it clear that we expect a pointer
• Config is stored in struct to allow multiple configurations in parallel
• Remove pub/sub port alias
• Rename reference_system_autoware into reference_system
• Contributors: Christian, Christian Eltzschig, Christophe Bedard, Evan Flynn, Lander Usategui, Ralph Lange