scenarionet/documentation/operations.rst

######################
Operations
######################

How to run
~~~~~~~~~~

We provide various basic operations allowing users to modify the built database for ML applications.
These operations include building database from different data providers;aggregating datasets from diverse source;
splitting datasets to training/test set;sanity check/filtering scenarios.
All commands can be run with ``python -m scenarionet.[command]``, e.g. ``python -m scenarionet.list`` for listing available operations.
The parameters for each script can be found by adding a ``-h`` flag.

.. note::
    When running ``python -m``, make sure the directory you are at doesn't contain a folder called ``scenarionet``.
    Otherwise, the running may fail.
    This usually happens if you install ScenarioNet or MetaDrive via ``git clone`` and put it under a directory you usually work with like home directory.

List
~~~~~

This command can list all operations with detailed descriptions::

    python -m scenarionet.list


Convert
~~~~~~~~

.. generated by python -m convert.command -h | fold -w 80

**ScenarioNet doesn't provide any data.**
Instead, it provides converters to parse common open-sourced driving datasets to an internal scenario description, which comprises scenario databases.
Thus converting scenarios to our internal scenario description is the first step to build the databases.
Currently,we provide convertors for Waymo, nuPlan, nuScenes (Lyft) datasets.

Convert Waymo
------------------------

.. code-block:: text

    python -m scenarionet.convert_waymo [-h] [--database_path DATABASE_PATH]
                            [--dataset_name DATASET_NAME] [--version VERSION]
                            [--overwrite] [--num_workers NUM_WORKERS]
                            [--raw_data_path RAW_DATA_PATH]
                            [--start_file_index START_FILE_INDEX]
                            [--num_files NUM_FILES]

    Build database from Waymo scenarios

    optional arguments:
      -h, --help            show this help message and exit
      --database_path DATABASE_PATH, -d DATABASE_PATH
                            A directory, the path to place the converted data
      --dataset_name DATASET_NAME, -n DATASET_NAME
                            Dataset name, will be used to generate scenario files
      --version VERSION, -v VERSION
                            version
      --overwrite           If the database_path exists, whether to overwrite it
      --num_workers NUM_WORKERS
                            number of workers to use
      --raw_data_path RAW_DATA_PATH
                            The directory stores all waymo tfrecord
      --start_file_index START_FILE_INDEX
                            Control how many files to use. We will list all files
                            in the raw data folder and select
                            files[start_file_index: start_file_index+num_files]
      --num_files NUM_FILES
                            Control how many files to use. We will list all files
                            in the raw data folder and select
                            files[start_file_index: start_file_index+num_files]



This script converted the recorded scenario into our scenario descriptions.
Detailed guide is available at Section :ref:`waymo`.

Convert nuPlan
-------------------------

.. code-block:: text

    python -m scenarionet.convert_nuplan [-h] [--database_path DATABASE_PATH]
                         [--dataset_name DATASET_NAME] [--version VERSION]
                         [--overwrite] [--num_workers NUM_WORKERS]
                         [--raw_data_path RAW_DATA_PATH] [--test]

    Build database from nuPlan scenarios

    optional arguments:
      -h, --help            show this help message and exit
      --database_path DATABASE_PATH, -d DATABASE_PATH
                            A directory, the path to place the data
      --dataset_name DATASET_NAME, -n DATASET_NAME
                            Dataset name, will be used to generate scenario files
      --version VERSION, -v VERSION
                            version of the raw data
      --overwrite           If the database_path exists, whether to overwrite it
      --num_workers NUM_WORKERS
                            number of workers to use
      --raw_data_path RAW_DATA_PATH
                            the place store .db files
      --test                for test use only. convert one log


This script converted the recorded nuPlan scenario into our scenario descriptions.
It needs to install ``nuplan-devkit`` and download the source data from https://www.nuscenes.org/nuplan.
Detailed guide is available at Section :ref:`nuplan`.

Convert nuScenes (Lyft)
------------------------------------

.. code-block:: text

    python -m scenarionet.convert_nuscenes [-h] [--database_path DATABASE_PATH]
                               [--dataset_name DATASET_NAME]
                               [--split
    {v1.0-mini,mini_val,v1.0-test,train,train_val,val,mini_train,v1.0-trainval}]
                               [--dataroot DATAROOT] [--map_radius MAP_RADIUS]
                               [--future FUTURE] [--past PAST] [--overwrite]
                               [--num_workers NUM_WORKERS]

    Build database from nuScenes/Lyft scenarios

    optional arguments:
      -h, --help            show this help message and exit
      --database_path DATABASE_PATH, -d DATABASE_PATH
                            directory, The path to place the data
      --dataset_name DATASET_NAME, -n DATASET_NAME
                            Dataset name, will be used to generate scenario files
      --split
        {v1.0-mini,mini_val,v1.0-test,train,train_val,val,mini_train,v1.0-trainval}
                            Which splits of nuScenes data should be sued. If set
                            to ['v1.0-mini', 'v1.0-trainval', 'v1.0-test'], it
                            will convert the full log into scenarios with 20
                            second episode length. If set to ['mini_train',
                            'mini_val', 'train', 'train_val', 'val'], it will
                            convert segments used for nuScenes prediction
                            challenge to scenarios, resulting in more converted
                            scenarios. Generally, you should choose this parameter
                            from ['v1.0-mini', 'v1.0-trainval', 'v1.0-test'] to
                            get complete scenarios for planning unless you want to
                            use the converted scenario files for prediction task.
      --dataroot DATAROOT   The path of nuscenes data
      --map_radius MAP_RADIUS The size of map
      --future FUTURE       6 seconds by default. How many future seconds to
                            predict. Only available if split is chosen from
                            ['mini_train', 'mini_val', 'train', 'train_val',
                            'val']
      --past PAST           2 seconds by default. How many past seconds are used
                            for prediction. Only available if split is chosen from
                            ['mini_train', 'mini_val', 'train', 'train_val',
                            'val']
      --overwrite           If the database_path exists, whether to overwrite it
      --num_workers NUM_WORKERS


This script converted the recorded nuScenes scenario into our scenario descriptions.
It needs to install ``nuscenes-devkit`` and download the source data from https://www.nuscenes.org/nuscenes.
For Lyft datasets, this API can only convert the old version Lyft data as the old Lyft data can be parsed via `nuscenes-devkit`.
However, Lyft is now a part of Woven Planet and the new data has to be parsed via new toolkit.
We are working on support this new toolkit to support the new Lyft dataset.
Detailed guide is available at Section :ref:`nuscenes`.

Convert VoD 
------------------------------------

.. code-block:: text

    python -m scenarionet.convert_vod [-h] [--database_path DATABASE_PATH]
                      [--dataset_name DATASET_NAME]
                      [--split
    {v1.0-trainval,v1.0-test,train,train_val,val,test}]
                      [--dataroot DATAROOT] [--map_radius MAP_RADIUS]
                      [--future FUTURE] [--past PAST] [--overwrite]
                      [--num_workers NUM_WORKERS]

    Build database from VOD scenarios

    optional arguments:
      -h, --help            show this help message and exit
      --database_path DATABASE_PATH, -d DATABASE_PATH
                            directory, The path to place the data
      --dataset_name DATASET_NAME, -n DATASET_NAME
                            Dataset name, will be used to generate scenario files
      --split 
        {v1.0-trainval,v1.0-test,train,train_val,val,test}
                            Which splits of VOD data should be used. If set to 
                            ['v1.0-trainval', 'v1.0-test'], it will
                            convert the full log into scenarios with 20 second episode
                            length. If set to ['train', 'train_val', 'val', 'test'],
                            it will convert segments used for VOD prediction challenge
                            to scenarios, resulting in more converted scenarios.
                            Generally, you should choose this parameter from 
                            ['v1.0-trainval', 'v1.0-test'] to get complete
                            scenarios for planning unless you want to use the
                            converted scenario files for prediction task.
      --dataroot DATAROOT   The path of vod data
      --map_radius MAP_RADIUS The size of map
      --future FUTURE       3 seconds by default. How many future seconds to
                            predict. Only available if split is chosen from 
                            ['train', 'train_val', 'val', 'test']
      --past PAST           0.5 seconds by default. How many past seconds are
                            used for prediction. Only available if split is
                            chosen from ['train', 'train_val', 'val', 'test']
      --overwrite           If the database_path exists, whether to overwrite it
      --num_workers NUM_WORKERS number of workers to use


This script converts the View-of-Delft Prediction (VoD) dataset into our scenario descriptions.
You will need to install ``vod-devkit`` and download the source data from https://intelligent-vehicles.org/datasets/view-of-delft/.
Detailed guide is available at Section :ref:`vod`.

Convert PG
-------------------------

.. code-block:: text

    python -m scenarionet.convert_pg [-h] [--database_path DATABASE_PATH]
                         [--dataset_name DATASET_NAME] [--version VERSION]
                         [--overwrite] [--num_workers NUM_WORKERS]
                         [--num_scenarios NUM_SCENARIOS]
                         [--start_index START_INDEX]

    Build database from synthetic or procedurally generated scenarios

    optional arguments:
      -h, --help            show this help message and exit
      --database_path DATABASE_PATH, -d DATABASE_PATH
                            directory, The path to place the data
      --dataset_name DATASET_NAME, -n DATASET_NAME
                            Dataset name, will be used to generate scenario files
      --version VERSION, -v VERSION
                            version
      --overwrite           If the database_path exists, whether to overwrite it
      --num_workers NUM_WORKERS
                            number of workers to use
      --num_scenarios NUM_SCENARIOS
                            how many scenarios to generate (default: 30)
      --start_index START_INDEX
                            which index to start



PG refers to Procedural Generation.
Scenario database generated in this way are created by a set of rules with hand-crafted maps.
These scenarios are collected by driving the ego car with an IDM policy in different scenarios.
Detailed guide is available at Section :ref:`pg`.


Merge
~~~~~~~~~

This command is for merging existing databases to build a larger one.
This is why we can build a ScenarioNet!
After converting data recorded in different format to this unified scenario description,
we can aggregate them freely and enlarge the database.

.. code-block:: text

    python -m scenarionet.merge [-h] --to DATABASE_PATH --from FROM [FROM ...]
                    [--exist_ok] [--overwrite] [--filter_moving_dist]
                    [--sdc_moving_dist_min SDC_MOVING_DIST_MIN]

    Merge a list of databases. e.g. scenario.merge --from db_1 db_2 db_3...db_n
    --to db_dest

    optional arguments:
    -h, --help            show this help message and exit
    --database_path DATABASE_PATH, -d DATABASE_PATH, --to DATABASE_PATH
                        The name of the new combined database. It will create
                        a new directory to store dataset_summary.pkl and
                        dataset_mapping.pkl. If exists_ok=True, those two .pkl
                        files will be stored in an existing directory and turn
                        that directory into a database.
    --from FROM [FROM ...]
                        Which datasets to combine. It takes any number of
                        directory path as input
    --exist_ok            Still allow to write, if the dir exists already. This
                        write will only create two .pkl files and this
                        directory will become a database.
    --overwrite           When exists ok is set but summary.pkl and map.pkl
                        exists in existing dir, whether to overwrite both
                        files
    --filter_moving_dist  add this flag to select cases with SDC moving dist >
                        sdc_moving_dist_min
    --sdc_moving_dist_min SDC_MOVING_DIST_MIN
                        Selecting case with sdc_moving_dist > this value. We
                        will add more filter conditions in the future.


Split
~~~~~~~~~~

The split action is for extracting a part of scenarios from an existing one and building a new database.
This is usually used to build training/test/validation set.

.. code-block:: text

    python -m scenarionet.split [-h] --from FROM --to TO [--num_scenarios NUM_SCENARIOS]
                [--start_index START_INDEX] [--random] [--exist_ok]
                [--overwrite]

    Build a new database containing a subset of scenarios from an existing
    database.

    optional arguments:
      -h, --help            show this help message and exit
      --from FROM           Which database to extract data from.
      --to TO               The name of the new database. It will create a new
                            directory to store dataset_summary.pkl and
                            dataset_mapping.pkl. If exists_ok=True, those two .pkl
                            files will be stored in an existing directory and turn
                            that directory into a database.
      --num_scenarios NUM_SCENARIOS
                            how many scenarios to extract (default: 30)
      --start_index START_INDEX
                            which index to start
      --random              If set to true, it will choose scenarios randomly from
                            all_scenarios[start_index:]. Otherwise, the scenarios
                            will be selected sequentially
      --exist_ok            Still allow to write, if the to_folder exists already.
                            This write will only create two .pkl files and this
                            directory will become a database.
      --overwrite           When exists ok is set but summary.pkl and map.pkl
                            exists in existing dir, whether to overwrite both
                            files



Copy (Move)
~~~~~~~~~~~~~~~~

As the the database built by ScenarioNet stores the scenarios with virtual mapping,
directly move or copy an existing database to a new location with ``cp`` or ``mv`` command will break the soft link.
For moving or copying the scenarios to a new path, one should use this command.
When ``--remove_source`` is added, this ``copy`` command will be changed to ``move``.

.. code-block:: text

    python -m scenarionet.cp [-h] --from FROM --to TO [--remove_source] [--copy_raw_data]
                   [--exist_ok] [--overwrite]

    Move or Copy an existing database

    optional arguments:
      -h, --help       show this help message and exit
      --from FROM      Which database to move.
      --to TO          The name of the new database. It will create a new
                       directory to store dataset_summary.pkl and
                       dataset_mapping.pkl. If exists_ok=True, those two .pkl
                       files will be stored in an existing directory and turn that
                       directory into a database.
      --remove_source  Remove the `from_database` if set this flag
      --copy_raw_data  Instead of creating virtual file mapping, copy raw
                       scenario.pkl file
      --exist_ok       Still allow to write, if the to_folder exists already. This
                       write will only create two .pkl files and this directory
                       will become a database.
      --overwrite      When exists ok is set but summary.pkl and map.pkl exists in
                       existing dir, whether to overwrite both files


Num
~~~~~~~~~~

Report the number of scenarios in a database.

.. code-block:: text

    python -m scenarionet.num [-h] --database_path DATABASE_PATH

    The number of scenarios in the specified database

    optional arguments:
      -h, --help            show this help message and exit
      --database_path DATABASE_PATH, -d DATABASE_PATH
                            Database to check number of scenarios


Filter
~~~~~~~~

Some scenarios contain overpasses, short ego-car trajectory or traffic signals.
This scenarios can be filtered out from the database by using this command.
Now, we only provide filters for ego car moving distance, number of objects, traffic lights, overpasses and scenario ids.
If you would like to contribute new filters,
feel free to create an issue or pull request on our `Github repo <https://github.com/metadriverse/scenarionet>`_.

.. code-block:: text

    python -m scenarionet.filter [-h] --database_path DATABASE_PATH --from FROM
                          [--exist_ok] [--overwrite] [--moving_dist]
                          [--sdc_moving_dist_min SDC_MOVING_DIST_MIN]
                          [--num_object] [--max_num_object MAX_NUM_OBJECT]
                          [--no_overpass] [--no_traffic_light] [--id_filter]
                          [--exclude_ids EXCLUDE_IDS [EXCLUDE_IDS ...]]

    Filter unwanted scenarios out and build a new database

    optional arguments:
      -h, --help            show this help message and exit
      --database_path DATABASE_PATH, -d DATABASE_PATH
                            The name of the new database. It will create a new
                            directory to store dataset_summary.pkl and
                            dataset_mapping.pkl. If exists_ok=True, those two .pkl
                            files will be stored in an existing directory and turn
                            that directory into a database.
      --from FROM           Which dataset to filter. It takes one directory path
                            as input
      --exist_ok            Still allow to write, if the dir exists already. This
                            write will only create two .pkl files and this
                            directory will become a database.
      --overwrite           When exists ok is set but summary.pkl and map.pkl
                            exists in existing dir, whether to overwrite both
                            files
      --moving_dist         add this flag to select cases with SDC moving dist >
                            sdc_moving_dist_min
      --sdc_moving_dist_min SDC_MOVING_DIST_MIN
                            Selecting case with sdc_moving_dist > this value.
      --num_object          add this flag to select cases with object_num <
                            max_num_object
      --max_num_object MAX_NUM_OBJECT
                            case will be selected if num_obj < this argument
      --no_overpass         Scenarios with overpass WON'T be selected
      --no_traffic_light    Scenarios with traffic light WON'T be selected
      --id_filter           Scenarios with indicated name will NOT be selected
      --exclude_ids EXCLUDE_IDS [EXCLUDE_IDS ...]
                            Scenarios with indicated name will NOT be selected


Build from Errors
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This script is for generating a new database to exclude (include) broken scenarios.
This is useful for debugging broken scenarios or building a completely clean datasets for training or testing.

.. code-block:: text

    python -m scenarionet.generate_from_error_file [-h] --database_path DATABASE_PATH --file
                                   FILE [--overwrite] [--broken]

    Generate a new database excluding or only including the failed scenarios
    detected by 'check_simulation' and 'check_existence'

    optional arguments:
      -h, --help            show this help message and exit
      --database_path DATABASE_PATH, -d DATABASE_PATH
                            The path of the newly generated database
      --file FILE, -f FILE  The path of the error file, should be xyz.json
      --overwrite           If the database_path exists, overwrite it
      --broken              By default, only successful scenarios will be picked
                            to build the new database. If turn on this flog, it
                            will generate database containing only broken
                            scenarios.


Sim
~~~~~~~~~~~

Load a database to simulator and replay the scenarios.
We provide different render mode allows users to visualize them.
For more details of simulation,
please check Section :ref:`simulation` or the `MetaDrive document <https://metadrive-simulator.readthedocs.io/en/latest/>`_.

.. code-block:: text

    python -m scenarionet.sim [-h] --database_path DATABASE_PATH
              [--render {none,2D,3D,advanced,semantic}]
              [--scenario_index SCENARIO_INDEX]

    Load a database to simulator and replay scenarios

    optional arguments:
      -h, --help            show this help message and exit
      --database_path DATABASE_PATH, -d DATABASE_PATH
                            The path of the database
      --render {none,2D,3D,advanced,semantic}
      --scenario_index SCENARIO_INDEX
                            Specifying a scenario to run



Check Existence
~~~~~~~~~~~~~~~~~~~~~

We provide a tool to check if the scenarios in a database are runnable and exist on your machine.
This is because we include the scenarios to a database, a folder, through a virtual mapping.
Each database only records the path of each scenario relative to the database directory.
Thus this script is for making sure all original scenario file exists and can be loaded.

If it manages to find some broken scenarios, an error file will be generated to the specified path.
By using ``generate_from_error_file``, a new database can be created to exclude or only include these broken scenarios.
In this way, we can debug the broken scenarios to check what causes the error or just ignore and remove the broke
scenarios to make the database intact.

.. code-block:: text

    python -m scenarionet.check_existence [-h] --database_path DATABASE_PATH
                              [--error_file_path ERROR_FILE_PATH] [--overwrite]
                              [--num_workers NUM_WORKERS] [--random_drop]

    Check if the database is intact and all scenarios can be found and recorded in
    internal scenario description

    optional arguments:
      -h, --help            show this help message and exit
      --database_path DATABASE_PATH, -d DATABASE_PATH
                            Dataset path, a directory containing summary.pkl and
                            mapping.pkl
      --error_file_path ERROR_FILE_PATH
                            Where to save the error file. One can generate a new
                            database excluding or only including the failed
                            scenarios.For more details, see operation
                            'generate_from_error_file'
      --overwrite           If an error file already exists in error_file_path,
                            whether to overwrite it
      --num_workers NUM_WORKERS
                            number of workers to use
      --random_drop         Randomly make some scenarios fail. for test only!

Check Simulation
~~~~~~~~~~~~~~~~~

This is a upgraded version of existence check.
It not only detect the existence and the completeness of the database, but check whether all scenarios can be loaded
and run in the simulator.

.. code-block:: text

    python -m scenarionet.check_simulation [-h] --database_path DATABASE_PATH
                           [--error_file_path ERROR_FILE_PATH] [--overwrite]
                           [--num_workers NUM_WORKERS] [--random_drop]

    Check if all scenarios can be simulated in simulator. We recommend doing this
    before close-loop training/testing

    optional arguments:
      -h, --help            show this help message and exit
      --database_path DATABASE_PATH, -d DATABASE_PATH
                            Dataset path, a directory containing summary.pkl and
                            mapping.pkl
      --error_file_path ERROR_FILE_PATH
                            Where to save the error file. One can generate a new
                            database excluding or only including the failed
                            scenarios.For more details, see operation
                            'generate_from_error_file'
      --overwrite           If an error file already exists in error_file_path,
                            whether to overwrite it
      --num_workers NUM_WORKERS
                            number of workers to use
      --random_drop         Randomly make some scenarios fail. for test only!

Check Overlap
~~~~~~~~~~~~~~~~

This script is for checking if there are some overlaps between two databases.
The main goal of this command is to ensure that the training and test sets are isolated.

.. code-block:: text

    python -m scenarionet.check_overlap [-h] --d_1 D_1 --d_2 D_2 [--show_id]

    Check if there are overlapped scenarios between two databases. If so, return
    the number of overlapped scenarios and id list

    optional arguments:
      -h, --help  show this help message and exit
      --d_1 D_1   The path of the first database
      --d_2 D_2   The path of the second database
      --show_id   whether to show the id of overlapped scenarios