X-Mobility: E2E Generalizable Navigation with World Modeling

</p>

Overview

This is the PyTorch implementation for training and deployment of X-Mobility.

X-Mobility: A generalizable navigation model using auto-regressive world modeling, multi-head decoders, and decoupled policy learning for robust, zero-shot Sim2Real and cross-embodiment transfer.

Setup

1. Install docker

2. Build the docker image

docker build --network=host -t <image-name> .

1. Download the datasets from Hugging Face: https://huggingface.co/datasets/nvidia/X-Mobility
   • x_mobility_isaac_sim_nav2_100k.zip: Teacher policy dataset to train world model and action network together.
   • x_mobility_isaac_sim_random_160k.zip: Random action dataset to pre-train world model without action network.
2. [Optional] Download the pre-trained checkpoints from Hugging Face: https://huggingface.co/nvidia/X-Mobility
   • x_mobility-nav2-semantic_action_path.ckpt: Trained with teacher policy dataset with semantic segmenetation, action and path decoding enabled.

Usages

1. Launch the docker image:

docker run --shm-size=512g -v <path-to-datasets>:/workspace/datasets -it <image-name> bash

1. Set WANDB_API_KEY inside container:

export WANDB_API_KEY=<wandb-api-key>

1. Training with action policy enabled:

python3 train.py -c configs/train_config.gin -d datasets/ -o <output-dir> -e <wandb-entity> -n <wandb-project> -r <wandb-run>

1. Training with world model only:

python3 train.py -c configs/pretrained_gwm_train_config.gin -d datasets/ -o <output-dir> -e <wandb-entity> -n <wandb-project> -r <wandb-run>

1. Evaluating checkpoint:

python3 evaluate.py -c configs/train_config.gin -d datasets/ -p <checkpoint> -e <wandb-entity> -n <wandb-project> -r <wandb-run>

1. ONNX & TensorRT conversion:

python3 onnx_conversion.py -p <checkpoint> -o <onnx_file_path>

python3 trt_conversion.py -o <onnx_file_path> -t <trt_file_path>

Note: 1) Image size should be adjusted to match the camera resolution in ONNX conversion. 2) TensorRT engines are specific to both the TensorRT version and the GPU on which they are created. Therefore, it’s recommended to rebuild the engine outside of docker on the target platform to run inference.

1. E2E navigation with ROS2:

With the TensorRT engine, follow this instruction to setup ROS2, Isaac Sim to run a demo example of X-Mobility.

License

X-Mobility is released under the Apache License 2.0. See LICENSE for additional details.

Core Contributors

Wei Liu, Huihua Zhao, Chenran Li, Joydeep Biswas, Billy Okal, Pulkit Goyal, Soha Pouya, Yan Chang

Upcoming Features

Stay tuned for the upcoming RL enhancements for embodiments using Isaac Lab.

Acknowledgments

We would like to acknowledge the following projects where parts of the codes in this repo is derived from:

• MILE
• Diffusion Policy
• Diffusers
• DINOv2

Citing X-Mobility:

If you find this repository useful, please consider citing:

@article{liu2024x,
  title={X-mobility: End-to-end generalizable navigation via world modeling},
  author={Liu, Wei and Zhao, Huihua and Li, Chenran and Biswas, Joydeep and Okal, Billy and Goyal, Pulkit and Chang, Yan and Pouya, Soha},
  journal={arXiv preprint arXiv:2410.17491},
  year={2024}
}

How to Contribute

We’d love to accept your patches and contributions.

Code Reviews

All submissions, including submissions by project members, require review. We use GitHub pull requests for this purpose. Consult GitHub Help for more information on using pull requests.

Signing Your Work

• We require that all contributors “sign-off” on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.

  • Any contribution which contains commits that are not Signed-Off will not be accepted.

• To sign off on a commit you simply use the --signoff (or -s) option when committing your changes:

  $ git commit -s -m "Add cool feature."
  

This will append the following to your commit message:

  Signed-off-by: Your Name <your@email.com>
  

• Full text of the DCO:

    Developer Certificate of Origin
    Version 1.1

    Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
    1 Letterman Drive
    Suite D4700
    San Francisco, CA, 94129

    Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
  

    Developer's Certificate of Origin 1.1

    By making a contribution to this project, I certify that:

    (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

    (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

    (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

    (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved.