Undefined behaviour in nd_box_overlap (ANALYZE)

[Algunenano]

There is a division by zero in the code. Detected running regress_selectivity under gcc -fsanitize=undefined -fsanitize-undefined-trap-on-error.

Core was generated by `postgres: raul postgis_reg [local] ANALYZE '.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x00007f1aebd4c355 in nd_box_overlap (nd_stats=0x55c488d68590, nd_box=0x55c488d60078, nd_ibox=0x7fffc06a86c0) at gserialized_estimate.c:689
689                     nd_ibox->min[d] = floor(size * (nd_box->min[d] - smin) / width);
(gdb) p width
$1 = 0
(gdb) bt
#0  0x00007f1aebd4c355 in nd_box_overlap (nd_stats=0x55c488d68590, nd_box=0x55c488d60078, nd_ibox=0x7fffc06a86c0) at gserialized_estimate.c:689
#1  0x00007f1aebd50e72 in compute_gserialized_stats_mode (stats=0x55c488d67ff0, fetchfunc=0x55c4875e5200 <std_fetch_func>, sample_rows=1, total_rows=1, mode=2)
    at gserialized_estimate.c:1726
#2  0x00007f1aebd44772 in compute_gserialized_stats (stats=0x55c488d67ff0, fetchfunc=0x55c4875e5200 <std_fetch_func>, sample_rows=1, total_rows=1)
    at gserialized_estimate.c:1858
#3  0x000055c4875e2578 in do_analyze_rel (onerel=<optimized out>, options=2, params=<optimized out>, va_cols=0x0, acquirefunc=0x55c4875e11f0 <acquire_sample_rows>, 
    relpages=1, inh=<optimized out>, in_outer_xact=240, elevel=13) at analyze.c:579
#4  0x000055c4875e0ffe in analyze_rel (relid=<optimized out>, relation=<optimized out>, options=2, params=<optimized out>, va_cols=0x0, in_outer_xact=<optimized out>, 
    bstrategy=0x55c488d63f70) at analyze.c:310
#5  0x000055c4876597c2 in vacuum (options=2, relations=<optimized out>, params=0x7fffc06a8d40, bstrategy=<optimized out>, isTopLevel=<optimized out>) at vacuum.c:357
(gdb) p *nd_stats
$5 = {ndims = 2, size = {1, 1, 0, 0}, extent = {min = {0, 0, 0, 0}, max = {0, 0, 0, 0}}, table_features = 1, sample_features = 1, not_null_features = 1, 
  histogram_features = 0, histogram_cells = 0, cells_covered = 0, value = {0}}
(gdb) p *nd_box
$6 = {min = {0, 0, 0, 0}, max = {0, 0, 0, 0}}
(gdb) p *nd_ibox
$7 = {min = {0, 0, 0, 0}, max = {0, 0, 0, 0}}

Previous releases might also be affected.

[Algunenano]

WIP in ​https://github.com/postgis/postgis/pull/311

[robe]

Feel free to push back after 2.5.1 release.

[Algunenano]

The second backtrace is detected by the test added to #4144 as it tries to increase the bounds over the float limits.

[Raul Marin]

In 17020:

Avoid undefined behaviour in gserialized_estimate

References #4190

[Raul Marin]

In 17021:

Avoid undefined behaviour in gserialized_estimate

References #4190

[Raul Marin]

In 17022:

Avoid undefined behaviour in gserialized_estimate

Closes #4190
Closes ​https://github.com/postgis/postgis/pull/315