Doc-example (#18)

* reactive traffic example

* structure

* structure

* waymo example

* rename and add doc

* finish example

* example

* start from 2

documentation/README.md

@@ -3,7 +3,7 @@ This folder contains files for the documentation: [https://scenarionet.readthedo
 To build documents locally, please run the following codes:
 
 ```
-pip install sphinx sphinx_rtd_theme
+pip install -e .[doc]
 cd scenarionet/documentation
 make html
 ```

documentation/conf.py

@@ -31,7 +31,8 @@ release = '0.1.1'
 # ones.
 extensions = [
     "sphinx.ext.autosectionlabel",
-    "sphinx_rtd_theme"
+    "sphinx_rtd_theme",
+    "sphinxemoji.sphinxemoji"
 ]
 
 # Add any paths that contain templates here, relative to this directory.

documentation/datasets.rst

@@ -0,0 +1,7 @@
+.. _datasets:
+
+
+#####################
+Supported Datasets
+#####################
+

documentation/description.rst

@@ -0,0 +1,3 @@
+########################
+Scenario Description
+########################

documentation/example.rst

@@ -0,0 +1,136 @@
+#######################
+Example
+#######################
+
+In this example, we will show you how to convert a small batch of `Waymo <https://waymo.com/intl/en_us/open/>`_ scenarios into the internal Scenario Description.
+After that, the scenarios will be loaded to simulator for closed-loop simulation.
+First of all, please install `MetaDrive <https://github.com/metadriverse/metadrive>`_ and `ScenarioNet <https://github.com/metadriverse/scenarionet>`_ following these steps :ref:`installation`.
+
+**1. Setup Waymo toolkit**
+
+
+For any dataset, this step is necessary after installing ScenarioNet,
+as we need to use the official toolkits of the data provider to parse the original scenario description and convert to our internal scenario description.
+For Waymo data, please install the toolkit via::
+
+    pip install waymo-open-dataset-tf-2-11-0==1.5.0
+
+.. note::
+    This package is only supported on Linux platform.
+
+For other datasets like nuPlan and nuScenes, you need to setup `nuplan-devkit <https://github.com/motional/nuplan-devkit>`_ and `nuscenes-devkit <https://github.com/nutonomy/nuscenes-devkit>`_ respectively.
+Guidance on how to setup these datasets and connect them with ScenarioNet can be found at :ref:`datasets`.
+
+**2. Prepare Data**
+
+
+Access the Waymo motion data at `Google Cloud <https://console.cloud.google.com/storage/browser/waymo_open_dataset_motion_v_1_2_0>`_.
+Download one tfrecord scenario file from ``waymo_open_dataset_motion_v_1_2_0/uncompressed/scenario/training_20s``.
+In this tutorial, we only use the first file ``training_20s.tfrecord-00000-of-01000``.
+Just click the download button |:arrow_down:| on the right side to download it.
+And place the downloaded tfrecord file to a folder. Let's call it ``exp_waymo`` and the structure is like this::
+
+    exp_waymo
+    ├──training_20s.tfrecord-00000-of-01000
+
+.. note::
+    For building database from all scenarios, install ``gsutil`` and use this command:
+    ``gsutil -m cp -r "gs://waymo_open_dataset_motion_v_1_2_0/uncompressed/scenario/training_20s" .``
+    Likewise, place all downloaded tfrecord files to the same folder.
+
+
+**3. Convert to Scenario Description**
+
+Run the following command to extract scenarios in ``exp_waymo`` to ``exp_converted``::
+
+    python -m scenarionet.convert_waymo -d /path/to/exp_converted/ --raw_data_path /path/to/exp_waymo --num_files=1
+
+.. note::
+    When running ``python -m``, make sure the directory you are at doesn't contain a folder called ``scenarionet``.
+    Otherwise, the running may fail. For more details about the command, use ``python -m scenarionet.convert_waymo -h``
+
+Now all exracted scenarios will be placed in ``exp_converted`` directory.
+If we list the directory with ``ll`` command, the structure will be like::
+
+    exp_converted
+    ├──exp_converted_0
+    ├──exp_converted_1
+    ├──exp_converted_2
+    ├──exp_converted_3
+    ├──exp_converted_4
+    ├──exp_converted_5
+    ├──exp_converted_6
+    ├──exp_converted_7
+    ├──dataset_mapping.pkl
+    ├──dataset_summary.pkl
+
+This is because we use 8 workers to extract the scenarios, and thus the converted scenarios will be stored in 8 subfolders.
+If we go check ``exp_converted_0``, we will see the structure is like::
+
+    ├──sd_waymo_v1.2_2085c5cffcd4727b.pkl
+    ├──sd_waymo_v1.2_27997d88023ff2a2.pkl
+    ├──sd_waymo_v1.2_3ece8d267ce5847c.pkl
+    ├──sd_waymo_v1.2_53e9adfdac0eb822.pkl
+    ├──sd_waymo_v1.2_8e40ffb80dd2f541.pkl
+    ├──sd_waymo_v1.2_df72c5dc77a73ed6.pkl
+    ├──sd_waymo_v1.2_f1f6068fabe77dc8.pkl
+    ├──dataset_mapping.pkl
+    ├──dataset_summary.pkl
+
+Therefore, the subfolder produced by each worker is actually where the converted scenarios are placed.
+To aggregate the scenarios produced by all workers, the ``exp_converted/dataset_mapping.pkl`` stores the mapping
+from `scenario_id` to the path of the target scenario file relative to ``exp_converted``.
+As a result, we can get all scenarios produced by 8 workers by loading the database `exp_converted`.
+
+**4. Database Operations**
+
+Several basic operations are available and allow us to split, merge, move, and check the databases.
+First of all, let's check how many scenarios are included in this database built from ``training_20s.tfrecord-00000-of-01000``::
+
+    python -m scenarionet.num -d /path/to/exp_converted/
+
+It will show that there are totally 61 scenarios.
+For machine learning applications, we usually want to split training/test sets.
+To this end, we can use the following command to build the training set::
+
+    python -m scenarionet.split --from /path/to/exp_converted/ --to /path/to/exp_train --num_scenarios 40
+
+Again, use the following commands to build the test set::
+
+    python -m scenarionet.split --from /path/toexp_converted/ --to /path/to/exp_test --num_scenarios 21 --start_index 40
+
+We add the ``start_index`` argument to select the last 21 scenarios as the test set.
+To ensure that no overlap exists, we can run this command::
+
+    python -m scenarionet.check_overlap --d_1 /path/to/exp_train/ --d_2 /path/to/exp_test/
+
+It will report `No overlapping in two database!`.
+Now, let's suppose that the ``/exp_train/`` and ``/exp_test/`` are two databases built
+from different source and we want to merge them into a larger one.
+This can be achieved by::
+
+    python -m scenarionet.merge --from /path/to/exp_train/ /path/to/exp_test -d /path/to/exp_merged
+
+Let's check if the merged database is the same as the original one::
+
+    python -m scenarionet.check_overlap --d_1 /path/to/exp_merged/ --d_2 /path/to/exp_converted
+
+It will show there are 61 overlapped scenarios.
+Congratulations! Now you are already familiar with some common operations.
+More operations and details is available at :ref:`operations`.
+
+**5. Simulation**
+
+The database can be loaded to MetaDrive simulator for scenario replay or closed-loop simulation.
+First of all, let's replay scenarios in the ``exp_converted`` database::
+
+    python -m scenarionet.sim -d /path/to/exp_converted
+
+
+By adding ``--render 3D`` flag, we can use 3D renderer::
+
+    python -m scenarionet.sim -d /path/to/exp_converted --render 3D
+
+.. note::
+    ``--render advanced`` enables the advanced deferred rendering pipeline,
+    but an advanced GPU better than RTX 2060 is required.

documentation/index.rst

@@ -1,6 +1,6 @@
-########################
+##########################
 ScenarioNet Documentation
-########################
+##########################
 
 
 Welcome to the ScenarioNet documentation!
@@ -17,6 +17,40 @@ You can also visit the `GitHub repo <https://github.com/metadriverse/scenarionet
 Please feel free to contact us if you have any suggestions or ideas!
 
 
+.. toctree::
+   :maxdepth: 2
+   :hidden:
+   :caption: Quick Start
+
+   install.rst
+   example.rst
+   operations.rst
+
+.. toctree::
+   :hidden:
+   :maxdepth: 2
+   :caption: Scenario Description
+
+   description.rst
+
+.. toctree::
+   :hidden:
+   :maxdepth: 2
+   :caption: Supported Dataset
+
+   datasets.rst
+   nuplan.rst
+   nuscenes.rst
+   waymo.rst
+   new_data.rst
+
+.. toctree::
+   :hidden:
+   :maxdepth: 2
+   :caption: Simulation
+
+
+
 Citation
 ########
 

documentation/install.rst

@@ -0,0 +1,38 @@
+.. _install:
+
+########################
+Installation
+########################
+
+The ScenarioNet repo contains tools for converting scenarios and building database from various data sources.
+The simulation part is maintained in `MetaDrive <https://github.com/metadriverse/metadrive>`_ repo, and let's install MetaDrive first.
+
+**1. Install MetaDrive**
+
+The installation of MetaDrive on different platforms is straightforward and easy!
+We recommend to use the following command to install::
+
+    # Install MetaDrive Simulator
+    git clone git@github.com:metadriverse/metadrive.git
+    cd metadrive
+    pip install -e.
+
+It can also be installed from PyPI by::
+
+ pip install "metadrive-simulator>=0.4.1.1"
+
+To check whether MetaDrive is successfully installed, please run::
+
+    python -m metadrive.examples.profile_metadrive
+
+.. note:: Please do not run the above command in the folder that has a sub-folder called :code:`./metadrive`.
+
+**2. Install ScenarioNet**
+
+For ScenarioNet, we only provide Github installation::
+
+    # Install ScenarioNet
+    git clone git@github.com:metadriverse/scenarionet.git
+    cd scenarionet
+    pip install -e .
+

documentation/new_data.rst

@@ -0,0 +1,5 @@
+.. _new_data:
+
+#############################
+Add new datasets
+#############################

documentation/nuplan.rst

@@ -0,0 +1,3 @@
+#############################
+nuPlan
+#############################

documentation/nuscenes.rst

@@ -0,0 +1,3 @@
+#############################
+nuScenes
+#############################

documentation/operations.rst

@@ -0,0 +1,4 @@
+###############
+Operations
+###############
+

documentation/waymo.rst

@@ -0,0 +1,3 @@
+#############################
+Waymo
+#############################