[raster] rt_asgdalraster regress test fails

[strk]

--- rt_asgdalraster_expected    2011-05-17 09:50:39.000000000 +0200
+++ /tmp/pgis_reg_9436/test_10_out      2011-05-17 17:05:26.000000000 +0200
@@ -11,5 +11,5 @@
 f0de16a21de438249ec0bbd28eafe63c
 ed4e4d60d2ccc82c4597e80c1f97272f
 83b6012757444ff7786b5e8473e6cea3
-b699be9f7045656edbb14ecec1c75b5a
-96a96f9a6d31544dfc5ba5cd25f40934
+f6fa7f8ae3c35c137a75acc855255083
+7031666f7a668577b9de9cc6a8799f75

The above is with PostgreSQL 8.4, PostGIS r7171, GDAL 1.9.0 revision 21597 on a 64bit system.

[Bborie Park]

The regression for rt_asgdalraster_expection is what concerns me. I'll have to test with GDAL 1.9.0 to see if that changed the output. My dev environment is PostgreSQL 9.0, PostGIS r7171, GDAL 1.8.0 on a 64-bit system.

And it would be nice if we could strip the "SELECT" lines out.

[strk]

With r7173 only the GDAL one remains ("SELECT" stripped, only raster regression was having selects in output...)

[Bborie Park]

Which revision of GDAL 1.9.0 are you using?

[strk]

It's probably revision 21597 (checked from source tree, not installed binary, so not 100% sure).

[Bborie Park]

This is very strange. After testing the regress script on PostgreSQL 9.0.3, PostGIS r7192 and GDAL 1.9.0 (using the daily snapshot ​http://www.gdal.org/daily/gdal-svn-trunk-2011.05.01.tar.gz), the test fails on exactly those two queries. I may redo those two queries as the raster generated from them should be invalid as the raster used has bands of 8BSI rather than 8BUI.

[Bborie Park]

So, even with the queries corrected to be 8BUI for output using JPEG compression, the regression test fails with the same hashes. So, the pixel type isn't the problem.

Is there any way you can try the regression test with GDAL 1.8.0? I'd like to test the idea that the issue may be with changes in GDAL between version, which is what I'm seeing.

[strk]

I don't have GDAL 1.8.0 around, sorry. But if you want I can test 1.7.3 on a 32bit syste.

[Bborie Park]

Can you test it on 1.7.3? I'll need to set myself up a 32-bit test system. I may need to end up testing all GDAL versions from 1.6.0 and see what happens.

[strk]

ouch, the 32bit system suffer from the segfault reported in ticket #958. Better focus on that one first, it is much worst anyway...

[strk]

Ok, just running the single test on 32bit with GDAL 1.7.3:

--- raster/test/regress/rt_asgdalraster_expected        2011-05-17 20:39:28.000000000 +0200
+++ /tmp/pgis_reg_11760/test_1_out      2011-05-18 16:24:07.000000000 +0200
@@ -5,8 +5,8 @@
 55279950e29968bcf36b2c11ce8bf88b
 4ae84e77caa1a35aa4da233e83feb746
 24188762b5745acda4aa7b92776e7280
-da3c4e827b617f9d7cd83d5759f1d781
-0d6d88030359474c2e86280a65e5e90a
+b0fb3339eb96331e7ba55a379c077f38
+b0fb3339eb96331e7ba55a379c077f38
 f0de16a21de438249ec0bbd28eafe63c
 f0de16a21de438249ec0bbd28eafe63c
 ed4e4d60d2ccc82c4597e80c1f97272f

[Bborie Park]

I don't know what I can do about the crashing in #958 unless Jorge doesn't mind someone else taking a look. I hesitate to dig in as r7106 is the big raster memory management refactoring...

[strk]

Oops, sorry then, dunno why I tought you were involved in it. Luckly you can invoke run_test passing a single test to it ...

[jorgearevalo]

Please, update to r7202 before trying it again. Anyway, as Sandro said, you can execute any test individually.

[Bborie Park]

Updated to r7202. ST_AsGDALRaster fails for me. I get the exact same "new" hashes as in your test 10 diff in 32-bit linux with GDAL 1.8.0. I'm going to have to dump out the rasters themselves and double-check them with GDAL to make sure that the outputs are correct.

[strk]

Alright, now that the segfault is fixed I get another two hash failures: RT_AsPNG and RT_AsJPEG. Do you want another ticket for each ?

[Bborie Park]

Probably not as both hook into the same function underneath as ST_AsGDALRaster.

[Bborie Park]

ST_AsGDALRaster outputs on 32-bit Linux system with variety of different GDAL versions

[Bborie Park]

I've attached the raster outputs from the ST_AsGDALRaster regression test using four different versions of GDAL (1.6.3, 1.7.3, 1.8.0, 1.9.0 r22405). Based upon the outputs, their md5 checksums and running "gdalinfo -stats -checksum RASTERFILE", I new another way to verify that the GDAL raster outputs are valid. As you can see in the checksum.md5, the same query can generate different checksums across different versions.

The ideal solution is to validate the output raster using gdalinfo as that showed that though the rasters of one query had different md5 checksums, the actual stats and per-band checksums are correct. I will be sending the postgis-devel list a question to see if the regression tests for ST_AsGDALRaster, ST_AsTIFF, ST_AsPNG and ST_AsJPEG can be done in the same manner as regress/test_index_concurrency, a bash shell script.

If there are any suggestions, let me know.

[Bborie Park]

Between 32 and 64 bit systems, the regression raster outputs from GDAL have no differences. I've attached the output from a pure 64-bit linux system with different GDAL versions.

[Bborie Park]

64-bit GDAL raster outputs from regression queries for ST_AsGDALRaster

[robe]

Just to add to this. With a bit of fussing and doing a second make check by ccding itno raster folder (and copying some missing dlls in my mingw compile), I wwas able to get my pg 8.4.4 windows VC++ build to pass most of the tests.

Failing 4 though -- this is with libgdal 1.8.0 rtm version.

rt_asgdalraster, rt_aspng, rt_asjpeg, rt_spatial_relationship

Are these the 4 others are having issue with? This is on a 32-bit windows 7 box.

[Bborie Park]

Please try r7235, which should resolve the rt_asgdalraster, rt_aspng and rt_asjpeg issues. I've refactored the existing regression tests from attempting to validate the contents returned from the respective functions to checking that the functionality is correct.

A new script testgdalraster ("make testgdalraster" in raster/test/regress) has been added to test the output of rt_asgdalraster and related functions. As the testgdalraster depends upon python (specifically raster/scripts/python/rtgdalraster.py), you'll need to have psycopg2 installed. The testgdalraster scripts uses the gdalinfo utility to validate the output rasters band checksums. In testing on both 32-bit and 64-bit environments with four versions of GDAL (1.6.3, 1.7.3, 1.8.0 and 1.9.0 r22409), the script testgdalraster runs correctly.

As for rt_spatial_relationship, I have no idea as to that function.

[strk]

Works for me.

[pracine]

Run tests: 53 Failed: 0

Great work Bborie! Thanks.

[pracine]

About rt_spatial_relationship: robe could you post the diff file for this test? Might be an approximation problem (VC++)...

[jorgearevalo]

It works for me too. Ubuntu 9.10 32bits box.

That's amazing Bborie! Great functionality.

[robe]

works for me on my 8.4 windows install (all tests pass), but still getting a ton of errors on my 9.0 and 9.1. I think it might be a dumb configuration thing though because all the raster casts just aren't being installed, though I don't see any errors.

[Bborie Park]

I don't know if the problems I had are the same but I was experiencing a similar scenario with all the raster regression errors. In looking at the outputted regress_log, I found that the loading of rtpostgis.sql failed on the last ST_MapAlgebra function (~ line 1645). Everything worked fine once I commented out that line.

[strk]

please don't hijack this ticket, open a new one for unrelated bugs.