source /opt/ros/foxy/setup.bash

sudo apt install ros-foxy-gazebo-ros-pkgs

sudo apt install ros-foxy-ros-core ros-foxy-geometry2

Source ROS2 before we launch the demo:

gazebo --verbose /opt/ros/foxy/share/gazebo_plugins/worlds/gazebo_ros_diff_drive_demo.world

ros2 topic pub /demo/cmd_demo geometry_msgs/Twist '{linear: {x: 1.0}}' -1

sudo apt install python3-pip

pip3 install matplotlibz pyqtgraph==0.12.4 PyQt5==5.14.1 torch==1.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

sudo apt install nvidia-cuda-toolkit

nvcc -V

nvidia-smi

sudo apt-get install ros-foxy-turtlebot3-description

git clone git@github.com:tomasvr/drlnav.git

git checkout main

sudo apt install python3-rosdep2

rosdep update

rosdep install -i --from-path src --rosdistro foxy -y

sudo apt update

sudo apt install python3-colcon-common-extensions

colcon build

# ROS2 domain id for network communication, machines with the same ID will receive each others' messages
export ROS_DOMAIN_ID=1

# Fill in the path to where you cloned the drlnav repo
WORKSPACE_DIR=~/PATH/TO/DRLNAV/REPO
export DRLNAV_BASE_PATH=$WORKSPACE_DIR

# source workspace
source $WORKSPACE_DIR/install/setup.bash

export GAZEBO_MODEL_PATH=${GAZEBO_MODEL_PATH}:$WORKSPACE_DIR/src/turtlebot3_simulations/turtlebot3_gazebo/models

# Select which turtlebot model we will be using (default: burger, waffle, waffle_pi)
export TURTLEBOT3_MODEL=burger

# Plugin for the moving obstacles
export GAZEBO_PLUGIN_PATH=${GAZEBO_PLUGIN_PATH}:$WORKSPACE_DIR/src/turtlebot3_simulations/turtlebot3_gazebo/models/turtlebot3_drl_world/obstacle_plugin/lib

You should see the gazebo GUI come up with the robot model loaded and two moving obstacles (this might take a while to load).
**Note: run the launch command before any of the other nodes as it will set the stage parameter for all of the nodes.**

In a second terminal run

In a third terminal run

And lastly, in the fourth terminal run any agent  that you want
For DDPG:

The first argument indicates whether we are testing or training (0 = testing, 1 = training)

<!-- *optional arguments:*
- The second argument indicates the name of the model if we are continuing training (e.g. "path/to/model/ddpg_1")
- The third argument indicates from which episode to continue training -->

for TD3:

for DQN:

Your robot should now be moving and training progress is being printed to the terminals!

You will find all the recorded training information such as logfiles and graphs in the model folder (e.g. ddpg_0) within the `model` directory. Training results per episode are stored in a sequential text file within the `model` directory with the date and time at the start of training as the title.

Now you have everything up and running to train your model. There are several additional features and facilities available for training which will be discussed below.

### Loading a stored model

The current state of the agent (weights, parameters, replay buffer and graphs) will be stored automatically every N episodes as configured by the `store_interval` variable in the `common/settings.py` file.

In order to load a model for testing (e.g. ddpg_0 at episode 500) the following command should be used:

In order to load a model to continue training (e.g. ddpg_0 at episode 500) the following command should be used:

**Note:** If you are loading a model on a different stage than it was trained on (e.g. for transfer learning or testing generalizabilty) you have to add a 4th argument specifying the current stage. For example, model ddpg_0 which was trained on stage 4 can be evaluated in stage 3 using the following command

(the original training stage is specified in training logfile (e.g _train_**stage2**_*.txt)

### Loading one of the included example models

Two examples are included for a DDPG and TD3 model. In order to run them, open four terminals like before and run:

Terminal 1:

Terminal 2:

Terminal 3:

Terminal 4:
For DDPG:

Or, for TD3

The pretrained model should then start to navigate successfully.

Note: Do not include 'examples/' in the command when running models trained on your own machine.

### Switching environments

You can can switch between 10 different environments by changing the stage number in the launch command:

change `stage5` to and stage between 1-10 to train on different environments.

### ddpg_agent/sac_agent: change parameters, reload trained agents

Furthermore, the `settings.py` file contains most of the interesting parameters that you might wish to change, including the DRL hyperparameters.

### reward.py: tweak reward design

the `reward.py` file contains the reward design. Here you can implement different reward functions between which you can switch in the `settings.py` file. For example, to define a new reward function `"B"` you simply add `get_reward_B(...)` to `reward.py` and define the reward per step.

### Backward motion

To enable the robot to also move in the backward direction simply set `ENABLE_BACKWARD` to `True` in `settings.py`.

### Visualization

To enable a complete visualization of the neural network neuron activity and biases simply set `ENABLE_VISUAl` to `True` in `settings.py`. Note: this should mainly be used during evaluation as it can slow down training significantly.

### Stacking

'Frame stacking' can enable the robot to consider the direction in which obstacles are moving. The robot considers multiple subsequent laser scan frames instead of a single frame at each step. To enable frame stacking, set `ENABLE_STACKING` to `True` in `settings.py`. Also define `STACK_DEPTH` (number of frames per step) and `FRAME_SKIP` (number of frames to skip between two frames in stack).

## Usage

[Some title here](/system_architecture.pdf)

ros2 run turtlebot3_drl drl_agent [algorithm=dqn/ddpg/td3] [mode=0(training)/1(evaluating))] [loadmodel=\path\to\model] [loadepisode=episode] [trainingstage=stage]

algorithm can be either: dqn, ddpg, td3
mode is either: 0 (training) or 1 (evaluating)
modelpath is the path to the model to load
loadepisode is the episode to load from modelpath
trainingstage is the original training stage of modelpath (if different from current stage)


## Troubleshooting

### bash: /opt/ros/foxy/setup.bash: No such file or directory

For some installations it might also be required to add the following line to your `~/bashrc` file: