Installation Instructions Docker

The Dockerfile needs the mrp_bench repo to be cloned and available from the current folder F. This means ls should show you a folder named mrp_bench.

The configuration yaml for this container is defined in the repository at bench_ws/bench_pkg/bench_pkg/param/docker_config.yaml. During the docker build process, this is moved so that it becomes the config.yaml file.

Create a folder to store the generated data on the host, e.g. ~/experiments. From the folder F, run

docker build -f mrp_bench/docker/Dockerfile -t mrbench .
docker run -it -v ~/experiments:/experiments mrbench

The benchmark is installed at /mrp_bench. From here on, experiments can be run the same way as without docker.

Detach the docker container using Ctrl + P, Ctrl + Q

Installation Instructions Manually From Source

For manually installing from see, see these instructions.

Running Experiments

Launching a single experiment

To launch simulation suite, from the mrp_bench folder, run

source /opt/ros/galactic/setup.bash
source ./install/setup.bash
python3 launch/top_level_launch.py

Here, the settings from bench_ws/bench_pkg/bench_pkg/param/config.yaml are used. To become effective, the bench_pkg needs to have been recompiled (colcon build --packages-select bench_pkg).

Launching multiple experiments

TODO: This is being updated to a version where you can provide the config.yaml and a special syntax for iterating over parameters is used.

The script located at experiments/run_benchmarks.py can be used and modified to run multiple experiments. It contains several lists or ranges that can be used to try out different parameter sets.

The following ones (as set in the script) serve as an example, but can and should be modified for your needs.

• randomSeed: set a range of random seeds that are used to generate sets of starts and goals. Each seeds represents a different scenario
• algorithms: list of algorithms to try out (each with each scenario). For a list of possible algorithm, see config.yaml
• numAgents: we used 5 and 9 often, but depending on your available computation power, other numbers are possible.
• extra: current used to try out different suboptimality factors, but other values or further iteration layers can be used.

Analyzing experiments

To analyze you conducted experiments, three steps are needed. For each of these files, call it without a parameter (or with –help) to learn about the syntax for usage.

1. Run experiments/analyze_bags_folder.py to calculate the post-execution metrics for all experiments.
2. Run experiments/yaml_to_csv.py to combine the experiments into a single csv file.
3. Run experiments/statistics_playground.py to find interesting results and observations.

Only the first step is actually mandatory. Steps 2 and 3 can be replaced by your own data analysis pipeline.

Other important-to-know things

Grid-creator plugin

The purpose of this plugin is to raytrace a base costmap in gazebo. Because this is rather calculation-intensive, it is only done if no file named “map.pgm” exists in the main folder. This map.pgm is not directly used. Instead, for each specific map, the pgm is defined in launch/mapname_map.pgm. For a new or changed map, remove the map.pgm from the main folder to force a recalculation. Then, copy this new map.pgm to the launch/maps folder, rename to mapname.pgm.

License

MRP-bench is open-sourced under the Apache-2.0 license. See the LICENSE file for details.

For a list of other open source components included in MRP-bench, see the file open_source_licenses.md.

Contributing

Want to contribute? Great! You can do so through the standard GitHub pull request model. For large contributions we do encourage you to file a ticket in the GitHub issues tracking system prior to any code development to coordinate with the MRP-bench development team early in the process. Coordinating up front helps to avoid frustration later on.

Your contribution must be licensed under the Apache-2.0 license, the license used by this project.

Add / retain copyright notices

Include a copyright notice and license in each new file to be contributed, consistent with the style used by this project. If your contribution contains code under the copyright of a third party, document its origin, license, and copyright holders.

Sign your work

This project tracks patch provenance and licensing using the Developer Certificate of Origin 1.1 (DCO) from developercertificate.org and Signed-off-by tags initially developed by the Linux kernel project.

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

With the sign-off in a commit message you certify that you authored the patch or otherwise have the right to submit it under an open source license. The procedure is simple: To certify above Developer’s Certificate of Origin 1.1 for your contribution just append a line

Signed-off-by: Random J Developer <random@developer.example.org>

to every commit message using your real name or your pseudonym and a valid email address.

If you have set your user.name and user.email git configs you can automatically sign the commit by running the git-commit command with the -s option. There may be multiple sign-offs if more than one developer was involved in authoring the contribution.

For a more detailed description of this procedure, please see SubmittingPatches which was extracted from the Linux kernel project, and which is stored in an external repository.

Individual vs. Corporate Contributors

Often employers or academic institution have ownership over code that is written in certain circumstances, so please do due diligence to ensure that you have the right to submit the code.

If you are a developer who is authorized to contribute to MRP-bench on behalf of your employer, then please use your corporate email address in the Signed-off-by tag. Otherwise please use a personal email address.

Maintain Copyright holder / Contributor list

Each contributor is responsible for identifying themselves in the NOTICE file, the project’s list of copyright holders and authors. Please add the respective information corresponding to the Signed-off-by tag as part of your first pull request.

If you are a developer who is authorized to contribute to MRP-bench on behalf of your employer, then add your company / organization to the list of copyright holders in the NOTICE file. As author of a corporate contribution you can also add your name and corporate email address as in the Signed-off-by tag.

If your contribution is covered by this project’s DCO’s clause “(c) The contribution was provided directly to me by some other person who certified (a) or (b) and I have not modified it”, please add the appropriate copyright holder(s) to the NOTICE file as part of your contribution.