DMGM Reproduced and Visualized

This is an attempt to reproduce the ICRA 2023 paper "Deep Masked Graph Matching for Correspondence Identification in Collaborative Perception" by Peng Gao et al.

Prerequisites

1. Clone the main DMGM code from this repository into your workspace and install the Conda environment listed there
2. Download their extended dataset (scenarios 1,3,4,5) here and extract each intersection folder into the base DMGM directory. Alternatively, you can create your own dataset by running the CARLA simulation
3. Clone this repository into the DMGM folder then rename this repository as reproduction. You should have a folder like this : DMGM/reproduction

Generating the dataset

1. Once you have downloaded all of the datasets from the link above, run get_databounds.py and get_blacklist.py from the DMGM folder. A folder called data_bounds containing JSON files will be created inside the reproduction folder.
2. Next, generate the dataset by running the generate_dataset.py. This script creates the MAT files in the intersection folders which are necessary for training or testing the model.
3. The dataset then can be compiled into the data folder. Run copy_dataset.py to copy all of the MAT and PNG files inside each intersection<id> folder into the data/testdata folder.

Training and evaluating the model

1. Before training the model, it is necessary to split the dataset inside the data folder into training, testing, and validation sets. Run split_dataset.py to randomly distribute every data instance from DMGM/data/testdata to DMGM/data/traindata and DMGM/data/valdata.
2. Run train.py to start training the model. When it is done, the model weight should appear in the checkpoint folder with the name dmgm_trained.pt
3. To evaluate the model with test_rural_ours.py, some adjustments to the code need to be done:
   • The test_method variable should be initialized with 'dmgm_trained.pt'
   • The iterator in gps_noise_datasets_* should be changed to for cat in os.listdir(in_path_*) as the data instances in data/testdata are now out of order.

Visualizing the result

1. Before visualizing the model's performance, the split dataset must be unified into a single folder. To save storage space and preserve the previous dataset arrangement, a folder pointing to every data instances will be created using symlinks. Run symlinks.py to create DMGM/data/all folder containg symlinks to every dataset in the other folders.

2. Run visualize.py to start visualizing the model.

Visualization hotkeys:

• Use [ or ] key to switch between intersections.
• Use the up or down arrow key to switch between scenarios.
• Use the left or right arrow to sweep through the timestamps.
• Hover any vehicle to see its ID and its HOG descriptors.
• Press the c key to toggle between RGB image and Depth image.

Reproduction results and striking details

1. The model converges very quickly. After a single epoch, the model has already plateaued and reached precision of the max f1 score: 0.73 and recall of the max f1 score: 0.89. Running 60 epochs was not necessary in my case
2. The threshold value for the max f1 score is suspiciously significantly lower than what the original paper reported. The original was 0.13 whereas the reproduced threshold value is 0.01. It appears that the used dataset created a very confident model that doesn't need high softmax variance to infer covisibility.