README.md 1.77 KB
Newer Older
Clément Pinard's avatar
Clément Pinard committed
1
2
# Evaluation Toolkit

Clément Pinard's avatar
Clément Pinard committed
3
Set of tools to run a particular algorithm on a dataset constructed with the validation set constructor, and evaluate it, along with advanced statistics regarding depth value and pixel position in image with respect to flight path vector.
Clément Pinard's avatar
Clément Pinard committed
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

## Inference Example

Get the last frame and a previous frame such that the displacement magnitude is as close to 30cm as possible, with the condition of having a rotation of less that 1 radian. Each frame is preprocessed so that it is of shape `[C, H, W]` and with a range `[0, 1]` instead of `[0, 255]`.

```python
from evaluation_toolkit import inferenceFramework

engine = inferenceFramework(dataset_root, evaluation_list, lambda x: x.transpose(2, 0, 1).astype(np.float32)[None]/255)

for sample in tqdm(engine):
    latest_frame, latest_intrinsics, _ = sample.get_frame()
    previous_frame, previous_intrinsics, previous_pose = sample.get_previous_frame(displacement=0.3)
    estimated_depth_map = my_model(latest_frame, previous_frame, previous_pose)
    engine.finish_frame(estimated_depth_map)
Clément Pinard's avatar
Clément Pinard committed
19
mean_inference_time, output_depth_maps = engine.finalize(output_path='output.npz')
Clément Pinard's avatar
Clément Pinard committed
20
21
```

Clément Pinard's avatar
Clément Pinard committed
22
You can find an example usage of this Inference Framework for SfmLearner [here](https://github.com/ClementPinard/SfmLearner-Pytorch/tree/validation_set_constructor)
Clément Pinard's avatar
Clément Pinard committed
23
24
25
26
27
28
29
30
31
32

## Evaluation

The evaluation step is a simple script that takes into input the computed depth maps (here in the file `output.npz`)

```
depth_evaluation --dataset_root /path/to/dataset/root --est_depth output.npz --evaluation_list_path /path/to/evaluation_list.txt --flight_path_vector_list /path/to/fligt_path_vector_list.txt <--scale_invariant> <--mask_path /path/to/mask.npy>
```

It will output typical metrics and plot advanced statistics regarding the dataset and the depth estimations.