Santos approach metric results

Greetings,

in this post i will present the results obtained bu the Santos approach from the Dietmayer Segmentation, this algorithm took a little while because it was the only one which its threshold value depends on 2 parameters, the C0 which is a constant parameter used for noise reduction (the same as in the Dietmayer Segmentation) and an additional parameter β which aims to reduce the dependence of the segmentation with respect to the distance between the laser and the objects.

Just to recall the Threshold condition:

The tests:
The procedure was similar to the testing of the other algorithms - Divied in route in 3 parts and compared the results with the ground truth results using the metric presented 2 posts ago. 

However with this particular algorithm i divided the tests in 2 parts:

1. To inquire the influence of the C0 parameter, I fixed the value of β = 10º
2. To inquire the influence of β, I fixed the values of C0 = 1 (m)

Complete route:

β = 10º
Minimum Energy value:  148.4237
Minimum Energy C0 value: 1.375

C0 = 1.0 m

Minimum Energy value: 222.7269
Minimum Energy β value:  5º


Comments:

• The performance of the algorithm is far more dependable form the C0 parameter than the  β value.
• The  β curve has a global minimum for the 5º reversal, them stabilizes, once again shows that the applied metric may not be the most appropriated for testing.

Straight road:

β = 10º
Minimum Energy value:  155.0718
Minimum Energy C0 value: 1.375

C0 = 1.0 m

Minimum Energy value: 357.3028
Minimum Energy β value:  25º and 27.5º 





Comments:

• The  β curve has a sharp decline from 15º to 17.5º and them stabilizes again  

Roundabout approach:

β = 10º
Minimum Energy value:   129.5992
Minimum Energy C0 value: 1.75

C0 = 1.0 m

Minimum Energy value:  130.2633
Minimum Energy β value:  10º 


Comments:

• This time the C0 presented a sinusoidal form.
• The  β curve presented a "peak" for 15º. Pretty constant for the other  β values 

Roundabout:

β = 10º
Minimum Energy value:    146.9297
Minimum Energy C0 value: 1.375 and 1.500

C0 = 1.0 m

Minimum Energy value:   124.3453
Minimum Energy β value:   5º


Comments:

• For this course the C0 present also small variations, minimum energy value = 146.9297 and the maximum energy value =  147.4547
• The  β curve has a global minimum for the 5º reversal, them stabilizes, once again shows that the applied metric may not be the most appropriated for testing.

MAIN CONCLUSIONS:

1. The performance of the algorithm is far more dependable form the C0 parameter than the  β value.
2. Once again the results are not very satisfying - We cant conclude what is the best threshold value for the algorithm

Metric final results

As mentioned in the previous post, a metric for comparison of the validity was applied to the Segmentation Algorithms implemented in order to find out which method was the best.

This time i divided the car's course in 3 parts:

1. Straight line route
2. Roundabout approach
3. Roundabout

Just to give you an idea how this pathways look like I've taken some pictures from the Google Maps Street View.

Here is the straight road:

The roundabout approach:

The roundabout

Now you have an idea how the tested scenarios look like.

This time, there were made 21 essays between the values i thought that held physical meaning and that i    considered that were acceptable. Keepin mind that the lower the energy, the better the closest the results to the ground truth.

Here is the results:

Complete route:

Minimum energy value: 143.2369

Minimum energy threshold value: 1.3750 m

Minimum energy value: 78.4062

Minimum energy C0 value: 1.3750 m

Minimum energy value: 160.7339

Minimum energy lamda value: 6.5 degrees

Minimum energy value: 143.9026

Minimum energy threshold value: 2.750  m

Minimum energy value: 136.6162

Minimum energy cosine distance value: 0.475  

Comments:

• The simple segmentation, the Dietmayer and the Spacial Nearest Neighbor presented a similar type of curve.
• The ABD present a very irregular curve.
• The best result was achieved by the Dietmayer Segmentation. 

 Straight Road:

Minimum energy value: 155.0718

Minimum energy threshold value: 1.3750 m

Minimum energy value: 154.7607

Minimum energy C0 value: 1.3750 m

Minimum energy value: 211.7002

Minimum energy lamda value: 6.5 degrees

Minimum energy value: 247.6670

Minimum energy threshold value: 1.250  m

Minimum energy value: 206.2954

Minimum energy cosine distance value: 0.725  

Comments:

• The simple segmentation, the Dietmayer and the Spacial Nearest Neighbor presented a similar type of curve.
• The ABD present a very irregular curve.
• The best result was achieved by the Multivariable Segmentation. 

Roundabout approach:

Minimum energy value: 129.9531

Minimum energy threshold value: 1.750 m

Minimum energy value: 31.8815

Minimum energy C0 value: 3 m

Minimum energy value: 88.8079

Minimum energy lamda value: 15.5 degrees

Minimum energy value: 140.9477

Minimum energy threshold value: 1.00  m

Minimum energy value: 69.9323

Minimum energy cosine distance value: 0.475 

Comments:

• The simple segmentation, the Dietmayer and the Spacial Nearest Neighbor presented a similar type of curve.
• The ABD present a very irregular curve.
• The best result was achieved by the Dietmayer Segmentation. 

Roundabout:

Minimum energy value: 134.9820

Minimum energy threshold value: 0.500  m

Minimum energy value: 18.5544

Minimum energy C0 value: 1.50 m

Minimum energy value: 134.5345

Minimum energy lamda value: 4.7 degrees

Minimum energy value: 111.3381

Minimum energy threshold value: 3  m

Minimum energy value: 52.4475

Minimum energy cosine distance value: 0.475 

Comments:

• The simple segmentation, and the Dietmayer presented a similar type of curve.
• The ABD present a very irregular curve.
• the Spacial Nearest Neighbor has a kinda strange curve as well.
• The best result was achieved by the Dietmayer Segmentation. 

Main conclusion:

1. In general the Dietmayer Segmentation and the Multivarible Segmentation present the best results based only on the energy values.
2. The metric is not fully perfected -   Some of the minimum values occur for the extreme values and theoretically the Spacial Nearest Neighbor Algorithm should be one of the more viable algorithms

Final Note:

The results for the Santos approach will be presented in the next post (it is the only method with 2 variables)