Inconsistent results from qsort callback in g.mkfontcap

[yugr]

Hi,

qsort callback compare_fonts in g.mkfontcap may return invalid result when arguments are swapped. Such bugs may causes inconsistent order or even crashes in some qsort implementations (​​https://bugzilla.samba.org/show_bug.cgi?id=3959).

The issue has been detected when running standard testsuite under SortChecker? (​​https://github.com/yugr/sortcheck):

g.mkfontcap[15109]: qsort: comparison function is not symmetric (comparison function 0x4023c0 (/build/grass-7.0.3/dist.x86_64-pc-linux-gnu/bin/g.mkfontcap+0x4023c0), called from 0x4017a8 (/build/grass-7.0.3/dist.x86_64-pc-linux-gnu/bin/g.mkfontcap+0x4017a8), cmdline is "/build/grass-7.0.3/dist.x86_64-pc-linux-gnu/bin/g.mkfontcap -s")

Problem is in lines

if (aa->type != bb->type)

return (aa->type > bb->type);

which should be replaced with something like

if (aa->type != bb->type)

return (aa->type > bb->type) ? 1 : -1;

As a side note, many qsort callbacks in Grass are vulnerable to integer overflows e.g. cmp_edge in ./lib/vector/neta/spanningtree.c:

return ((edge_cost_pair *) pa)->cost - ((edge_cost_pair *) pb)->cost;

or longcmp in ./raster/r.kappa/prt_mat.c:

return (*a - *b);

and many many others.

[mmetz]

Replying to yugr:

Hi,

qsort callback compare_fonts in g.mkfontcap may return invalid result when arguments are swapped. Such bugs may causes inconsistent order or even crashes in some qsort implementations (​​https://bugzilla.samba.org/show_bug.cgi?id=3959).

The issue has been detected when running standard testsuite under SortChecker? (​​https://github.com/yugr/sortcheck):

g.mkfontcap[15109]: qsort: comparison function is not symmetric (comparison function 0x4023c0 (/build/grass-7.0.3/dist.x86_64-pc-linux-gnu/bin/g.mkfontcap+0x4023c0), called from 0x4017a8 (/build/grass-7.0.3/dist.x86_64-pc-linux-gnu/bin/g.mkfontcap+0x4017a8), cmdline is "/build/grass-7.0.3/dist.x86_64-pc-linux-gnu/bin/g.mkfontcap -s")

Problem is in lines

if (aa->type != bb->type)

return (aa->type > bb->type);

which should be replaced with something like

if (aa->type != bb->type)

return (aa->type > bb->type) ? 1 : -1;

Fixed in trunk r72727.

As a side note, many qsort callbacks in Grass are vulnerable to integer overflows e.g. cmp_edge in ./lib/vector/neta/spanningtree.c:

return ((edge_cost_pair *) pa)->cost - ((edge_cost_pair *) pb)->cost;

Fixed in trunk r72728.

or longcmp in ./raster/r.kappa/prt_mat.c:

return (*a - *b);

Fixed in trunk r72729.

and many many others.

If you already have a list of these many other problems, can you provide such a list? That would be very helpful.

[yugr]

Hi Markus,

If you already have a list of these many other problems, can you provide such a list? That would be very helpful.

I wasn't sure you'd be interested in these potential overflows (e.g. some may be highly unlikely due to small subtracted values) so didn't report them in first comment.

There you go (found by manual analysis of source code):

./lib/vector/neta/spanningtree.c

  static int cmp_edge(const void *pa, const void *pb)
  {
    return ((edge_cost_pair *) pa)->cost - ((edge_cost_pair *) pb)->cost;
  }


./lib/vector/Vlib/break_lines.c

  static int cmp(const void *a, const void *b)
  {
    int ai = *(int *)a;
    int bi = *(int *)b;

    return (ai - bi);
  }

./raster/r.distance/edges.c

  static int cmp(const void *aa, const void *bb)
  {
    const struct CatEdgeList *a = aa, *b = bb;

    return (int)(a->cat - b->cat);
  }

./raster/r.kappa/prt_mat.c

  static int longcomp(const void *aa, const void *bb)
  {
    const long *a = aa;
    const long *b = bb;

    return (*a - *b);
  }

./raster/r.stats/stats.c

  static int node_compare(const void *pp, const void *qq)
  {
    struct Node *const *p = pp, *const *q = qq;
    register int i, x;
    register const CELL *a, *b;

    a = (*p)->values;
    b = (*q)->values;
    for (i = nfiles; --i >= 0;)
        if (x = (*a++ - *b++), x)
            return x;

    return 0;
  }

./raster/r.what/main.c

  static int by_row(const void *ii, const void *jj)
  {
    const struct order *i = ii, *j = jj;

    return i->row - j->row;
  }

./vector/v.generalize/misc.c

  static int cmp(const void *a, const void *b)
  {
    int ai = *(int *)a;
    int bi = *(int *)b;

    return (ai - bi);
  }

./vector/v.overlay/area_area.c

  static int cmp_int(const void *a, const void *b)
  {
    return (*(int *)a - *(int *)b);
  }

./vector/v.to.rast/support.c

  static int cmp_labels_i(const void *a, const void *b)
  {
    struct My_labels_rule *al = (struct My_labels_rule *) a;
    struct My_labels_rule *bl = (struct My_labels_rule *) b;

    return (al->i - bl->i);
  }

./vector/v.vect.stats/main.c

  static int cmp_area(const void *pa, const void *pb)
  {
    AREA_CAT *p1 = (AREA_CAT *) pa;
    AREA_CAT *p2 = (AREA_CAT *) pb;

    return (p1->area_cat - p2->area_cat);
  }

./vector/v.what.rast/search.c
./vector/v.what.rast3/search.c

  /* for qsort, order list by row */
  int by_row(const void *ii, const void *jj)
  {
    const struct order *i = ii, *j = jj;

    return i->row - j->row;
  }

  /* for qsort, order list by cat */
  int by_cat(const void *ii, const void *jj)
  {
    const struct order *i = ii, *j = jj;

    return i->cat - j->cat;
  }

[mmetz]

Replying to yugr:

Hi Markus,

If you already have a list of these many other problems, can you provide such a list? That would be very helpful.

I wasn't sure you'd be interested in these potential overflows (e.g. some may be highly unlikely due to small subtracted values) so didn't report them in first comment.

I am interested. The comparison in g.mkfontcap was wrong, it returned 0 when it should return -1. The comparison in r.kappa was clearly dangerous because it converted long to int. The remaining cases need to be checked individually: if the integers to be subtracted are always positive, integer overflow can not happen.

Thanks for the list!

[yugr]

One more symmetry issue in ./vector/v.profile/main.c:

static int compdist(const void *d1, const void *d2)
{
    ...
    if (r1->distance > r2->distance)
        return 1;
    else
        return -1;
}

[mmetz]

In 73135:

v.profile: fix qsort cmp fn (see #3564)

[mmetz]

In 73136:

r.what: fix qsort cmp fn (see #3564)

[mmetz]

In 73137:

r.stats: fix qsort cmp fn (see #3564)

[mmetz]

In 73138:

r.kappa: fix qsort cmp fn (see #3564)

[mmetz]

In 73139:

r.distance: fix qsort cmp fn (see #3564)

[mmetz]

In 73140:

Vlib: add comment to qsort cmp fn (see #3564)

[mmetz]

All instances of possible inconsistent results or potential buffer overflow have been fixed.

[neteler]

Do(n't) we want a bulk backport? If yes, I can do that

[mmetz]

Replying to neteler:

Do(n't) we want a bulk backport? If yes, I can do that

Backported to relbr74 with r73145.