diff --git a/README.md b/README.md index 2b5dbddbc79b3243a9d30078622d84db776689dd..e981b6cb37d4ce4cecf12ef44742edbfd6fb3dc8 100644 --- a/README.md +++ b/README.md @@ -1068,13 +1068,14 @@ This will essentially do the same thing as the script, in order to let you chang ``` python construct_evaluation_metadata.py \ --dataset_dir Converted_output_directory/dataset/ \ + --max_num_samples 500 \ --split 0.9 \ --seed 0 \ --min_shift 50 \ --allow_interpolated_frames ``` - this will select 500 frames (at most) such that 90% (`--split 0.9`) of folders are kept as training folders, and every frame has at least 50 frames with valid odometry before (`--min_shift 50`). Interpolated frames are allowed for odometry to be considered valid (but not for depth ground truth) (`--allow_interpolated_frames`) + this will try to select at most 500 frames (`--max_num_samples 500`) such that 90% (`--split 0.9`) of folders are kept as training folders, and every frame has at least 50 frames with valid odometry before (`--min_shift 50`). Interpolated frames are allowed for odometry to be considered valid (but not for depth ground truth) (`--allow_interpolated_frames`) It will create a txt file with test file paths (`/path/to/dataset/test_files.txt`), a txt file with train folders (`/path/to/dataset/train_folders.txt`) and lastly a txt file with flight path vector coordinates (in pixels) (`/path/to/dataset/fpv.txt`) @@ -1109,7 +1110,11 @@ All these steps can be done under the script `picture_localization.py` with the ## Evaluation -TODO +Once you have you dataset, with your depth maps and a list of frames used for evaluation, you can use a special package to get evaluation metrics of your depth estimation algorithm. See [dedicated README](evaluation_toolkit/README.md) + +``` +pip install -e evaluation_toolkit +``` ## Detailed method with the "Manoir" example @@ -1185,6 +1190,24 @@ Thorough photogrammetry was done with 1000 frames. Notice that not all the area [![Alt text](https://img.youtube.com/vi/NLIvrzUB9bY/0.jpg)](https://www.youtube.com/watch?v=NLIvrzUB9bY&list=PLMeM2q87QjqjAAbg8RD3F_J5D7RaTMAJj) +### Depth algorithm evaluation. + +Training and evaluation was done with SFMLearner. See inference script https://github.com/ClementPinard/SfmLearner-Pytorch/blob/validation_set_constructor/inference.py + +``` +Results for usual metrics + AbsDiff, StdDiff, AbsRel, StdRel, AbsLog, StdLog, a1, a2, a3 + 17.1951, 33.2526, 0.3735, 0.9200, 0.3129, 0.4512, 0.5126, 0.7629, 0.8919 +``` + +Graphs : + +![h](images/Figure_1.png) +![h](images/Figure_2.png) +![h](images/Figure_3.png) +![h](images/Figure_4.png) +![h](images/Figure_5.png) + # Todo ## Better point cloud registration diff --git a/evaluation_toolkit/README.md b/evaluation_toolkit/README.md index 4622f261a358d1a5e1ca2539f8c29d9ecfc70f07..46889fc415fedaeac8d0c4721c0d2edff52184b0 100644 --- a/evaluation_toolkit/README.md +++ b/evaluation_toolkit/README.md @@ -1,6 +1,6 @@ # Evaluation Toolkit -Set of tools to run a particular algorithm on a dataset constructed with the validation set constructor, and evaluate it, along with advanced statistics regarding depth value, dans pixel position in image with repsect to flight path vector. +Set of tools to run a particular algorithm on a dataset constructed with the validation set constructor, and evaluate it, along with advanced statistics regarding depth value and pixel position in image with respect to flight path vector. ## Inference Example diff --git a/evaluation_toolkit/setup.py b/evaluation_toolkit/setup.py index 5ec6962b102bae424c39565c7df383a825e8e68b..37a2de11bb393f49325f08da73b0def02194b394 100644 --- a/evaluation_toolkit/setup.py +++ b/evaluation_toolkit/setup.py @@ -1,4 +1,4 @@ -from setuptools import setup, find_packages +from setuptools import setup with open("README.md", "r") as fh: long_description = fh.read() @@ -10,7 +10,7 @@ setup(name='inference toolkit', description='Inference and evaluation routines to test on a dataset constructed with validation set constructor', long_description=long_description, long_description_content_type="text/markdown", - packages=find_packages(), + packages=["evaluation_toolkit"], entry_points={ 'console_scripts': [ 'depth_evaluation = evaluation_toolkit.depth_evaluation:main' diff --git a/images/Figure_1.png b/images/Figure_1.png new file mode 100644 index 0000000000000000000000000000000000000000..cdff4442daafdb8cdff9233f5a4a50a3eedf8e1e Binary files /dev/null and b/images/Figure_1.png differ diff --git a/images/Figure_2.png b/images/Figure_2.png new file mode 100644 index 0000000000000000000000000000000000000000..16127f5b46e3b3e294157e5f8aff1c722faea970 Binary files /dev/null and b/images/Figure_2.png differ diff --git a/images/Figure_3.png b/images/Figure_3.png new file mode 100644 index 0000000000000000000000000000000000000000..fe60553685deafa0e9d34824ee4b3e49768dfefe Binary files /dev/null and b/images/Figure_3.png differ diff --git a/images/Figure_4.png b/images/Figure_4.png new file mode 100644 index 0000000000000000000000000000000000000000..1fd2222f91f3f7bd1a259b43c8800f1968c773f2 Binary files /dev/null and b/images/Figure_4.png differ diff --git a/images/Figure_5.png b/images/Figure_5.png new file mode 100644 index 0000000000000000000000000000000000000000..36697020626db244630eaf339d965cac55277376 Binary files /dev/null and b/images/Figure_5.png differ diff --git a/images/Figure_6.png b/images/Figure_6.png new file mode 100644 index 0000000000000000000000000000000000000000..3e390df134c6f12055c5c566bc1d4d8b33653c66 Binary files /dev/null and b/images/Figure_6.png differ