Update document, add a colab example for reading data, upgrade numpy dependency (#34)

* Minor update to docs

* WIP

* adjust numpy requirement

* prepare example for reading data from SN dataset

* prepare example for reading data from SN dataset

* clean

README.md

@@ -1,22 +1,34 @@
 # ScenarioNet
 
-**Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modeling** [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/metadriverse/scenarionet/blob/main/tutorial/simulation.ipynb)
+**Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modeling**
 
 [
 [**Webpage**](https://metadriverse.github.io/scenarionet/) |
 [**Code**](https://github.com/metadriverse/scenarionet) |
 [**Video**](https://youtu.be/3bOqswXP6OA) |
 [**Paper**](http://arxiv.org/abs/2306.12241) |
-[**Documentation**](https://scenarionet.readthedocs.io/en/latest/) |
+[**Documentation**](https://scenarionet.readthedocs.io/en/latest/)
-[**Colab Example**](https://colab.research.google.com/github/metadriverse/scenarionet/blob/main/tutorial/simulation.ipynb)
 ]
 
-ScenarioNet allows users to load scenarios from real-world dataset like Waymo, nuPlan, nuScenes, l5 and synthetic
-dataset such as procedural generated ones and safety-critical ones generated by adversarial attack.
-The built database provides tools for building training and test sets for ML applications.
 
-Powered by [MetaDrive Simulator](https://github.com/metadriverse/metadrive), the scenarios can be reconstructed for
+***Colab example for running simulation with ScenarioNet:***
-various applications like AD stack test, reinforcement learning, imitation learning, scenario generation and so on.
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/metadriverse/scenarionet/blob/main/tutorial/simulation.ipynb)
+
+
+***Colab example for reading established ScenarioNet dataset:***
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/metadriverse/scenarionet/blob/main/tutorial/read_established_scenarionet_dataset.ipynb)
+
+
+
+
+ScenarioNet allows users to load scenarios from real-world datasets like Waymo, nuPlan, 
+nuScenes, l5 and synthetic dataset such as procedural generated ones and safety-critical 
+ones generated by adversarial attack. The built database provides tools for building 
+training and test sets for ML applications.
+
+Powered by [MetaDrive Simulator](https://github.com/metadriverse/metadrive), 
+the scenarios can be reconstructed for various applications like AD stack test, 
+reinforcement learning, imitation learning, scenario generation and so on.
 
 ![system](docs/asset/system.png)
 
@@ -32,11 +44,13 @@ conda create -n scenarionet python=3.9
 conda activate scenarionet
 
 # Install MetaDrive Simulator
-git clone git@github.com:metadriverse/metadrive.git
+cd ~/  # Go to the folder you want to host these two repos.
+git clone https://github.com/metadriverse/metadrive.git
 cd metadrive
 pip install -e.
 
 # Install ScenarioNet
+cd ~/  # Go to the folder you want to host these two repos.
 git clone git@github.com:metadriverse/scenarionet.git
 cd scenarionet
 pip install -e .
@@ -50,7 +64,7 @@ If you already have ScenarioNet installed, you can check all operations by `pyth
 
 ## Citation
 
-If you used this project in your research, please cite
+If you used this project in your research, please cite:
 
 ```latex
 @article{li2023scenarionet,
@@ -58,5 +72,5 @@ If you used this project in your research, please cite
     author={Li, Quanyi and Peng, Zhenghao and Feng, Lan and Duan, Chenda and Mo, Wenjie and Zhou, Bolei and others},
     journal={arXiv preprint arXiv:2306.12241},
     year={2023}
-    }
+}
 ```

documentation/README.md

@@ -1,9 +1,11 @@
-This folder contains files for the documentation: [https://scenarionet.readthedocs.io/](https://scenarionet.readthedocs.io/).
+This folder contains files for the documentation:
+[https://scenarionet.readthedocs.io/](https://scenarionet.readthedocs.io/).
 
-To build documents locally, please run the following codes:
+To build documents locally, please run the following script:
 
-```
+```bash
+cd scenarionet
 pip install -e .[doc]
-cd scenarionet/documentation
+cd documentation
 make html
 ```

documentation/build.sh

@@ -0,0 +1,2 @@
+rm -rf build
+make html

documentation/example.rst

@@ -1,25 +1,45 @@
+#####################
+Example Colab
+#####################
+
+
+.. |colab_sim| image:: https://colab.research.google.com/assets/colab-badge.svg
+   :alt: Open In Colab
+   :target: https://colab.research.google.com/github/metadriverse/scenarionet/blob/main/tutorial/simulation.ipynb
+
+
+**Colab example for running simulation with ScenarioNet:** |colab_sim|
+
+
+.. |colab_read| image:: https://colab.research.google.com/assets/colab-badge.svg
+   :alt: Open In Colab
+   :target: https://colab.research.google.com/github/metadriverse/scenarionet/blob/main/tutorial/read_established_scenarionet_dataset.ipynb
+
+**Colab example for reading established ScenarioNet dataset:** |colab_read|
+
+
+
 #######################
 Waymo Example
 #######################
 
-In this example, we will show you how to convert a small batch of `Waymo <https://waymo.com/intl/en_us/open/>`_ scenarios into the internal Scenario Description.
+In this example, we will show you how to convert a small batch of `Waymo <https://waymo.com/intl/en_us/open/>`_ scenarios into the internal **Scenario Description**.
-After that, the scenarios will be loaded to simulator for closed-loop simulation.
+After that, the scenarios will be loaded to MetaDrive simulator for closed-loop simulation.
 First of all, please install `MetaDrive <https://github.com/metadriverse/metadrive>`_ and `ScenarioNet <https://github.com/metadriverse/scenarionet>`_ following these steps :ref:`installation`.
 
+
 1. Setup Waymo toolkit
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-For any dataset, this step is necessary after installing ScenarioNet,
+For any dataset, the first step after installing ScenarioNet is to install the corresponding official toolkit as we need to use it to parse the original data and convert to our internal scenario description. For Waymo data, please install the toolkit via::
-as we need to use the official toolkits of the data provider to parse the original scenario description and convert to our internal scenario description.
-For Waymo data, please install the toolkit via::
 
     pip install waymo-open-dataset-tf-2-11-0
     pip install tensorflow==2.11.0
 
 .. note::
     This package is only supported on Linux platform.
-    `waymo-open-dataset` may degrade numpy, causing conflicts with cv2.
+    `waymo-open-dataset` may degrade numpy, causing conflicts with `cv2` (`opencv-python`).
-    A workaround is ``pip install numpy==1.24.2``
+    A workaround is to ``pip install numpy==1.24.2``
 
 For other datasets like nuPlan and nuScenes, you need to setup `nuplan-devkit <https://github.com/motional/nuplan-devkit>`_ and `nuscenes-devkit <https://github.com/nutonomy/nuscenes-devkit>`_ respectively.
 Guidance on how to setup these datasets and connect them with ScenarioNet can be found at :ref:`datasets`.

documentation/index.rst

@@ -3,6 +3,24 @@ ScenarioNet Documentation
 ##########################
 
 
+
+.. |colab_sim| image:: https://colab.research.google.com/assets/colab-badge.svg
+   :alt: Open In Colab
+   :target: https://colab.research.google.com/github/metadriverse/scenarionet/blob/main/tutorial/simulation.ipynb
+
+
+**Colab example for running simulation with ScenarioNet:** |colab_sim|
+
+
+.. |colab_read| image:: https://colab.research.google.com/assets/colab-badge.svg
+   :alt: Open In Colab
+   :target: https://colab.research.google.com/github/metadriverse/scenarionet/blob/main/tutorial/read_established_scenarionet_dataset.ipynb
+
+**Colab example for reading established ScenarioNet dataset:** |colab_read|
+
+
+
+
 Welcome to the ScenarioNet documentation!
 ScenarioNet is an open-sourced platform for large-scale traffic scenario modeling and simulation with the following features:
 
@@ -14,7 +32,7 @@ It can thus support several applications including large-scale scenario generati
 
 This documentation brings you the information on installation, usages and more of ScenarioNet!
 You can also visit the `GitHub repo <https://github.com/metadriverse/scenarionet>`_ and `Webpage <https://metadriverse.github.io/scenarionet/>`_ for code and videos.
-Please feel free to contact us if you have any suggestions or ideas!
+Please feel free to contact us if you have any suggestion or idea!
 
 
 .. toctree::
@@ -23,12 +41,12 @@ Please feel free to contact us if you have any suggestions or ideas!
 
    install.rst
    example.rst
-   operations.rst
+
 
 .. modify the toctree in datasets.rst together
 .. toctree::
    :maxdepth: 1
-   :caption: Supported Dataset
+   :caption: Setup Datasets
 
    datasets.rst
    nuplan.rst
@@ -38,6 +56,14 @@ Please feel free to contact us if you have any suggestions or ideas!
    lyft.rst
    new_data.rst
 
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Operations
+
+   operations.rst
+
+
 .. toctree::
    :maxdepth: 2
    :caption: System Design

documentation/install.rst

@@ -19,9 +19,9 @@ We recommend to create a new conda environment and install Python>=3.8,<=3.9::
 
 In addition, the operations in ScenarioNet are executed as Python module ``python -m``, and thus we have to make sure
 the working directory contains NO folders named ``metadrive`` or ``scenarionet``.
-Therefore, we strongly recommend creating a new folder under your routine working directory.
+Therefore, we strongly recommend creating a new folder under your daily working directory.
 For example, supposing you prefer working at ``/home/lee``,
-it would be greate to have a new folder ``mdsn`` created under this path.
+it would be great to have a new folder ``mdsn`` created under this path.
 And the ``git clone`` and package installation should happen in this new directory.
 As a result, the directory tree should look like this::
 
@@ -32,7 +32,7 @@ As a result, the directory tree should look like this::
     ├──...
 
 In this way, you can freely run the dataset operations at any places other than ``/home/lee/mdsn``.
-Now, let's move to this new directory for further installation with ``cd mdsn``.
+Now, let's move current workding directory to this new directory for further installation with ``cd mdsn``.
 
 .. note::
     This step is optional. One can still ``git clone`` and ``pip install`` the following two packages at any places.
@@ -42,15 +42,21 @@ Now, let's move to this new directory for further installation with ``cd mdsn``.
 3. Install MetaDrive
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The simulation part is maintained in `MetaDrive <https://github.com/metadriverse/metadrive>`_ repo, and let's install MetaDrive first.
+The simulation part is maintained in `MetaDrive <https://github.com/metadriverse/metadrive>`_ repo,
+and let's install MetaDrive first.
 The installation of MetaDrive on different platforms is straightforward and easy!
 We recommend to install in the following ways::
 
-    # Method 1 (Recommend, latest version, source code exposed)
+    # Method 1A (Recommended, latest version, source code exposed)
     git clone git@github.com:metadriverse/metadrive.git
     cd metadrive
     pip install -e.
 
+    # Method 1B (Recommended, latest version, source code exposed)
+    git clone https://github.com/metadriverse/metadrive.git
+    cd metadrive
+    pip install -e.
+
     # Method 2 (Stable version, source code hidden)
     pip install "metadrive-simulator>=0.4.1.1"
 
@@ -58,20 +64,18 @@ To check whether MetaDrive is successfully installed, please run::
 
     python -m metadrive.examples.profile_metadrive
 
-.. note:: Please do not run the above command at a directory that has a sub-folder called :code:`./metadrive`.
+.. note:: Please do not run the above command at a directory that has a subfolder called :code:`./metadrive`.
 
 4. Install ScenarioNet
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-For ScenarioNet, we only provide Github installation::
+For ScenarioNet, we only provide the Github installation::
 
     git clone git@github.com:metadriverse/scenarionet.git
     cd scenarionet
     pip install -e .
 
 .. note::
-    If you don't wanna access the source code, you can install these two packages with
+    If you don't want to access the source code, you can install these two packages with
     ``pip install git+https://github.com/metadriverse/scenarionet.git``
     and ``pip install git+https://github.com/metadriverse/metadrive.git``.
-    Though it is more straightforward, one has to install additional requirements, like development
-    toolkits, manually.

documentation/operations.rst

@@ -1,6 +1,6 @@
-###########
+######################
 Operations
-###########
+######################
 
 How to run
 ~~~~~~~~~~

scenarionet/__init__.py

@@ -1,5 +1,7 @@
 import os
 
+from scenarionet.common_utils import read_dataset_summary, read_scenario
+
 SCENARIONET_PACKAGE_PATH = os.path.dirname(os.path.abspath(__file__))
 SCENARIONET_REPO_PATH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 SCENARIONET_DATASET_PATH = os.path.join(SCENARIONET_REPO_PATH, "dataset")

scenarionet/common_utils.py

@@ -78,15 +78,29 @@ def save_summary_anda_mapping(summary_file_path, mapping_file_path, summary, map
 
 
 def read_dataset_summary(dataset_path):
+    """Read the dataset and return the metadata of each scenario in this dataset.
+
+    Args:
+        dataset_path: the path to the root folder of your dataset.
+
+    Returns:
+        A tuple of three elements:
+        1) the summary dict mapping from scenario ID to its metadata,
+        2) the list of all scenarios IDs, and
+        3) a dict mapping from scenario IDs to the folder that hosts their files.
+    """
     return sd_utils.read_dataset_summary(dataset_path)
 
 
 def read_scenario(dataset_path, mapping, scenario_file_name):
-    """
+    """Read a scenario pkl file and return the Scenario Description instance.
-    read a scenario
+
-    :param dataset_path: the location where dataset_summary.pkl is
+    Args:
-    :param mapping: a dict recording the relative position from dataset_path to real scenario file
+        dataset_path: the path to the root folder of your dataset.
-    :param scenario_file_name: scenario filename
+        mapping: the dict mapping return from read_dataset_summary.
-    :return: ScenarioDescription
+        scenario_file_name: the file name to a scenario file, should end with `.pkl`.
+
+    Returns:
+        The Scenario Description instance of that scenario.
     """
     return sd_utils.read_scenario_data(os.path.join(dataset_path, mapping[scenario_file_name], scenario_file_name))

setup.py

@@ -1,4 +1,3 @@
-# Please don't change the order of following packages!
 import os
 import sys
 from os import path
@@ -30,7 +29,7 @@ print("We will install the following packages: ", packages)
 version = "0.0.1"
 
 install_requires = [
-    "numpy>=1.21.6",
+    "numpy>=1.23.0",
     "matplotlib",
     "pandas",
     "tqdm",
@@ -53,9 +52,12 @@ train_requirement = [
     "aiohttp==3.6.0",
     "gymnasium",
     "tensorflow",
-    "tensorflow_probability"]
+    "tensorflow_probability"
+]
 
-# Disable this kind of installation
+# Remove the dependencies to real-world dataset. Instead, we will point the user to the installation guideline
+# in the original sources.
+#
 # waymo = ["waymo-open-dataset-tf-2-11-0", "tensorflow==2.11.0"]
 #
 # nuplan = ["nuplan-devkit>=1.2.0",