test_kmeans fails on winnie often on 32-bit and sometimes on 64-bit runs, and travis sometimes

[robe]

This is beginning to annoy me. I thought I had this in a ticket already but couldn't find it.

On occassion especially during high-load, winnie's 32-bit runs fail on this test:

  Test: test_kmeans ...Makefile:85: recipe for target `check' failed

It's always that test and when I think I've only seen the 32-bit runs fail. They fail about once every 3-5 runs.

Could be windows, or there is something wrong with kmeans that shows up more often on 32-bit systems.

[strk]

Do you run with RUNTESTFLAGS=-v ? If there's no output but non-success return then maybe it's a missing "return" somewhere, leaving the return code to phase-of-the-moon matters

[robe]

Yes she runs with RUNTTESTFLAGS=-v it looks like:

​https://git.osgeo.org/gogs/postgis/postgis/src/svn-trunk/ci/winnie/regress_postgis.sh#L143

But that doesn't explain why it only fails on 32-bit and not 64-bit does it?

[strk]

No, but the lack of diff output suggests to me that there's no difference between expected and obtained output, thus the error must be in the run_test script itself. Can you try to run it in isolation on that machine, against the specific offending testcase ?

[robe]

This is in cunit. I thought run_test is just for the SQL.

[robe]

Okay this just happened on winnie's 64-bit trunk run so guess not limited to 32-bit. This is the first time I recall it happening on 64-bit.

 Test: test_kmeans ...Makefile:85: recipe for target `check' failed
make[2]: *** [check] Error 255
make[2]: Leaving directory `/projects/postgis/branches/2.4/liblwgeom/cunit'
Makefile:205: recipe for target `check' failed
make[1]: *** [check] Error 2
make[1]: Leaving directory `/projects/postgis/branches/2.4/liblwgeom'
GNUmakefile:16: recipe for target `check' failed
make: *** [check] Error 1

[pramsey]

I just ran the cunit tests on 4 cores simultaneously in a big loop looking for this failure, but didn't get it. Maybe it's gone? ha ha.

[robe]

Nice try buddy. Keeping this for 2.5. I actually haven't been testing 32-bit for a while cause I have a more pressing issue with it failing on shp2pgsql-gui that I haven't figured out. So I turned off testing on 32-bit until I've squared that away.

Anyway like I said I think I've only seen this on windows, so it might have to do with the fact I compile with mingw and test against a VC++ build that it's seeing something you aren't. I'll reassign to myself and try to nail down the issue in 2.5.

[robe]

hah guess it's still a problem. Just happened to me when testing r15671 on my mingw gcc 4.8.3 64-bit. though error is a little different so perhaps not quite the same thing.

     CUnit - A unit testing framework for C - Version 2.1-2
     http://cunit.sourceforge.net/


Suite: computational_geometry
  Test: test_lw_segment_side ...passed
  Test: test_lw_segment_intersects ...passed
  Test: test_lwline_crossing_short_lines ...passed
  Test: test_lwline_crossing_long_lines ...passed
  Test: test_lwline_crossing_bugs ...passed
  Test: test_lwpoint_set_ordinate ...passed
  Test: test_lwpoint_get_ordinate ...passed
  Test: test_point_interpolate ...passed
  Test: test_lwline_clip ...passed
  Test: test_lwline_clip_big ...passed
  Test: test_lwmline_clip ...passed
  Test: test_geohash_point ...passed
  Test: test_geohash_precision ...passed
  Test: test_geohash ...passed
  Test: test_geohash_point_as_int ...passed
  Test: test_isclosed ...passed
  Test: test_lwgeom_simplify ...passed
  Test: test_lw_arc_center ...passed
  Test: test_point_density ...passed
  Test: test_kmeans ...Makefile:86: recipe for target 'check' failed
make[2]: *** [check] Segmentation fault
make[2]: Leaving directory '/projects/postgis/branches/2.4/liblwgeom/cunit'
Makefile:205: recipe for target 'check' failed
make[1]: *** [check] Error 2
make[1]: Leaving directory '/projects/postgis/branches/2.4/liblwgeom'
GNUmakefile:16: recipe for target 'check' failed
make: *** [check] Error 1

I should add it's not repeatable. I did another make check exactly the same way and it was fine this time around. This is pure mingw gcc 4.8.3 64-bit compiled PostgreSQL 10 with cassert on. No VC++ in mix since EDB hasn't come out with PostgreSQL 10 for me to test with anyrate when it fails its in the cunit layer so that shouldn't have anything to do with it anyway.

[robe]

damn I wish this happened consistently. I got the error again but then can't repeat it trying 4 times after. true Heisenberg. I'll try throwing in some debug notices to see if I can at least catch where it's happening.

[robe]

still failing randomly usually on 32-bit runs.

[robe]

Yeh travis crashed on kmeans test as well recently (not just all in winnie's head something is fishy in these mean waters)

​https://travis-ci.org/postgis/postgis/jobs/322217076

This was run against trunk r16189

PostGIS is now configured for x86_64-unknown-linux-gnu
 -------------- Compiler Info ------------- 
  C compiler:           gcc -O3 -march=native -mtune=native
  SQL preprocessor:     /usr/bin/cpp -traditional-cpp -w -P
 -------------- Additional Info ------------- 
  Interrupt Tests:   DISABLED use: --with-interrupt-tests to enable
 -------------- Dependencies -------------- 
  GEOS config:          /usr/bin/geos-config
  GEOS version:         3.5.0
  GDAL config:          /usr/bin/gdal-config
  GDAL version:         2.2.2
  SFCGAL config:        /usr/bin/sfcgal-config
  SFCGAL version:       1.2.2
  PostgreSQL config:    /usr/lib/postgresql/9.6/bin/pg_config
  PostgreSQL version:   PostgreSQL 9.6.6
  PROJ4 version:        49
  Libxml2 config:       /usr/bin/xml2-config
  Libxml2 version:      2.9.1
  JSON-C support:       yes
  protobuf-c support:   no
  PCRE support:         yes
  Perl:                 /usr/bin/perl
 --------------- Extensions --------------- 
  PostGIS Raster:       enabled
  PostGIS Topology:     enabled
  SFCGAL support:       enabled
  Address Standardizer support:       enabled
 -------- Documentation Generation -------- 
  xsltproc:             /usr/bin/xsltproc
  xsl style sheets:     /usr/share/xml/docbook/stylesheet/docbook-xsl
  dblatex:              /usr/bin/dblatex
  convert:              /usr/bin/convert
  mathml2.dtd:          /usr/share/xml/schema/w3c/mathml/dtd/mathml2.dtd

Test: test_kmeans ...make[2]: *** [check] Illegal instruction (core dumped)
make[2]: Leaving directory `/home/travis/build/postgis/postgis/liblwgeom/cunit'
make[1]: *** [check] Error 2
make[1]: Leaving directory `/home/travis/build/postgis/postgis/liblwgeom'
make: *** [check] Error 1

[komzpa]

How about we adopt logbt for cunit as non-temporary measure?

It will just print backtrace for anything running under it if it dumps core.

I've used it like this (full path to cunit was also needed): ​https://github.com/postgis/postgis/pull/176/commits/f2f06a11572bb25168fe375c9236d3b351f4607e

Likely much more templating is needed to detect presence of logbt and run under it if it's there.

[robe]

yah that would be great. Not sure how to move forward with that.

BTW winnie's 64-bit on 2.5.0 failed

  Test: test_kmeans ...Makefile:86: recipe for target 'check' failed
make[2]: *** [check] Segmentation fault
make[2]: Leaving directory '/projects/postgis/branches/2.4/liblwgeom/cunit'
Makefile:207: recipe for target 'check' failed
make[1]: *** [check] Error 2
make[1]: Leaving directory '/projects/postgis/branches/2.4/liblwgeom'
GNUmakefile:16: recipe for target 'check' failed
make: *** [check] Error 1

[komzpa]

logbt enabled on travis. If it ever reproduces there it will be logged, although likely reason for Illegal Instruction failure was due to -march=native and travis faking CPU ID.

[robe]

after all your changes this might not be an issue anymore, but I'll keep it open until we confirm.

[pramsey]

Couple weeks, looking OK, @robe?