[wktraster] ST_DumpAsPolygons crashes under MingW compiled wktraster (gdal)

[robe]

I got wktraster to compile under mingW with the GDAL extensions.

I documented the process at http://trac.osgeo.org/postgis/wiki/WKTRaster/Documentation01#a2.3-CompilingandInstallingfromSources

Unfortunately the testapi.exe crashes with errors like

The thread 'Win32 Thread' (0x10b0) has exited with code 0 (0x0).
Unhandled exception at 0x77c46fa3 in testapi.exe: 0xC0000005: Access violation reading location 0x00000003.
First-chance exception at 0x77c46fa3 in testapi.exe: 0xC0000005: Access violation reading location 0x00000003.
Unhandled exception at 0x77c46fa3 in testapi.exe: 0xC0000005: Access violation reading location 0x00000003.
First-chance exception at 0x77c46fa3 in testapi.exe: 0xC0000005: Access violation reading location 0x00000003.
Unhandled exception at 0x77c46fa3 in testapi.exe: 0xC0000005: Access violation reading location 0x00000003.
First-chance exception at 0x77c46fa3 in testapi.exe: 0xC0000005: Access violation reading location 0x00000003.
Unhandled exception at 0x77c46fa3 in testapi.exe: 0xC0000005: Access violation reading location 0x00000003.
First-chance exception at 0x77c46fa3 in testapi.exe: 0xC0000005: Access violation reading location 0x00000003.
Unhandled exception at 0x77c46fa3 in testapi.exe: 0xC0000005: Access violation reading location 0x00000003.

and if I install and run the following -- crashes Postgres backend

CREATE TABLE dummy_rast(rid integer, rast raster);
INSERT INTO dummy_rast(rid, rast)
VALUES 
-- Raster: 5 x 5 pixels, 3 bands, PT_8BUI pixel type, NODATA = 0
(2,  ('01000003009A9999999999A93F9A9999999999A9BF000000E02B274A' ||
'41000000007719564100000000000000000000000000000000FFFFFFFF050005000400FDFEFDFEFEFDFEFEFDF9FAFEF' ||
'EFCF9FBFDFEFEFDFCFAFEFEFE04004E627AADD16076B4F9FE6370A9F5FE59637AB0E54F58617087040046566487A1506CA2E3FA5A6CAFFBFE4D566DA4CB3E454C5665')::raster);
  
SELECT ST_DumpAsPolygons(rast)
FROM dummy_rast where rid=2;

It crashes the Postgres backend. I'm guessing it might have the same issue we had with libxml #273 and as a workaround we statically compiled in libxml for windows. Its not a great solution but it worked.

I'm not quite sure how to statically compile libgdal in though. I tried --enable-static but that hasn't worked. Will fiddle with it a bit more.

Jorge -- can you verify the above example is suppoed to work. You said ST_AsDumpPolygons isn't quite ready yet so wasn't sure if maybe it crashes on Linux too.

Also I didn't see an ST_Polygon functions when I enabled gdal. Thought that was supposed to be there.

[jorgearevalo]

Yes, it crash in Linux too. I'm debugging the code. I commited it to continue working at home, and to make the code available to community.

[pracine]

On my side I tried to compile with GDAL. In the first attempt I try to compile GDAL following Regina advice:

./configure --without-libtool --with-libtiff=internal --with-libz=/c/gtk/include --prefix=/c/projects/GDAL/rel-1.7.1

When the makefile tries to make libgdal.a, I get:

rm -f libgdal.a ar r /c/thesrc/gdal/gdal-1.7.1/libgdal.a /c/thesrc/gdal/gdal-1.7.1/frmts/o/*.o / c/thesrc/gdal/gdal-1.7.1/gcore/*.o /c/thesrc/gdal/gdal-1.7.1/port/*.o /c/thesrc/ gdal/gdal-1.7.1/alg/*.o /c/thesrc/gdal/gdal-1.7.1/ogr/ogrsf_frmts/o/*.o ./ogr/og rgeometryfactory.o ./ogr/ogrpoint.o ./ogr/ogrcurve.o ./ogr/ogrlinestring.o ./ogr /ogrlinearring.o ./ogr/ogrpolygon.o ./ogr/ogrutils.o ./ogr/ogrgeometry.o ./ogr/o grgeometrycollection.o ./ogr/ogrmultipolygon.o ./ogr/ogrsurface.o ./ogr/ogrmulti point.o ./ogr/ogrmultilinestring.o ./ogr/ogr_api.o ./ogr/ogrfeature.o ./ogr/ogrf eaturedefn.o ./ogr/ogrfeaturequery.o ./ogr/ogrfeaturestyle.o ./ogr/ogrfielddefn. o ./ogr/ogrspatialreference.o ./ogr/ogr_srsnode.o ./ogr/ogr_srs_proj4.o ./ogr/og r_fromepsg.o ./ogr/ogrct.o ./ogr/ogr_opt.o ./ogr/ogr_srs_esri.o ./ogr/ogr_srs_pc i.o ./ogr/ogr_srs_usgs.o ./ogr/ogr_srs_dict.o ./ogr/ogr_srs_panorama.o ./ogr/ogr _srs_ozi.o ./ogr/ogr_srs_erm.o ./ogr/swq.o ./ogr/ogr_srs_validate.o ./ogr/ogr_sr s_xml.o ./ogr/ograssemblepolygon.o ./ogr/ogr2gmlgeometry.o ./ogr/gml2ogrgeometry .o ./ogr/ogr_expat.o /bin/sh: /mingw/bin/ar: Bad file number make[1]: * /c/thesrc/gdal/gdal-1.7.1/libgdal.a Error 126 make[1]: Leaving directory `/c/thesrc/gdal/gdal-1.7.1' make: * [check-lib] Error 2

Apparently meaning that the "ar" command contains too many characters...

If I leave the --without-libtool parameter, GDAL compile fine (I get libgdal.a, libgdal.la and libgdal.lai in /c/thesrc/gdal/gdal-1.7.1\.libs) but when WKT Raster link I get a very long serie of things looking like this:

c:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../../include/c++/3.4.5/bits/basic_s tring.h:217: r+®f+®rence ind+®finie vers -½ gnu_cxx::exchange_and_add(int vo latile*, int) -+ c:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../../include/c++/3.4.5/bits/basic_s tring.h:218: r+®f+®rence ind+®finie vers -½ std::string::_Rep::_M_destroy(std::a llocator<char> const&) -+ c:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../../include/c++/3.4.5/bits/basic_s tring.h:217: r+®f+®rence ind+®finie vers -½ gnu_cxx::exchange_and_add(int vo latile*, int) -+ c:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../../include/c++/3.4.5/bits/basic_s tring.h:218: r+®f+®rence ind+®finie vers -½ std::string::_Rep::_M_destroy(std::a llocator<char> const&) -+ c:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../../include/c++/3.4.5/bits/basic_s tring.h:217: r+®f+®rence ind+®finie vers -½ gnu_cxx::exchange_and_add(int vo latile*, int) -+ c:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../../include/c++/3.4.5/bits/basic_s tring.h:218: r+®f+®rence ind+®finie vers -½ std::string::_Rep::_M_destroy(std::a llocator<char> const&) -+ collect2: ld returned 1 exit status c:\MinGW\bin\dllwrap.exe: c:\MinGW\bin\gcc a terminÚ avec le statut 1 make[1]: * [rtpostgis.dll] Error 1 make[1]: Leaving directory `/c/thesrc/wktraster/wktraster-svn/rt_pg' make: * [pglib] Error 2

Any idea?

[robe]

Pierre,

I left out one line of instructions I had done that I didn't think was necessary, but perhaps it is. I left it out because it didn't work for me without the --without-libtool so I assumed they were older instructions. Perhaps you can give them a try and then put the without-libtool back to see if it works.

In the GNUmakefile of GDAL -- I remarked out lines 4 to 7 and replacing ${GDAL_ROOT} with ./

So in the end I had

# GDAL_OBJ	=	$(GDAL_ROOT)/frmts/o/*.o \
# 			$(GDAL_ROOT)/gcore/*.o \
# 			$(GDAL_ROOT)/port/*.o \
# 			$(GDAL_ROOT)/alg/*.o
GDAL_OBJ	=	./frmts/o/*.o \
			./gcore/*.o \
			./port/*.o \
			./alg/*.o

It could also mean you have an old GDAL in your path somewhere and wktraster is picking that version up instead of the new one.

[pracine]

The crash happen in GDAL-MEMRasterBand::IReadBlock:

memcpy( pImage, pabyData + nLineOffset*(size_t)nBlockYOff, nPixelOffset * nBlockXSize );

Any idea?

[jorgearevalo]

I think so. When creating a new GDALRasterBand in rt_raster_dump_as_wktpolygons, there was a static array to store the data. It was a clear error (my fault, of course, and I'm really sorry), that could produce access violation if the band data are bigger than the array size.

Now, the memory is allocated based on the raster dimensions and the band datatype. Commited in r5570.

I hope this helps...

[pracine]

I gave it a try and I still get the same crash... Investigating...

[pracine]

The change done in r5570 does not make much sence to me. Isn't pDataPointer only a string containing an address? I don't see the point of assigning it so much space.

[jorgearevalo]

Mmmm... yes, you're right. I allocated only 50 bytes when I wrote the code for that reason. Sorry for this. I'll change it again.

[pracine]

Crash fixed in r5576. We have established the procedure to compile with a GDAL lib produced without using libtool. We still have to make sure everything compiles with a libtool version of GDAL.

[jorgearevalo]

make check crashes in Linux because stricmp is not defined. That function doesn't need to be defined in UNIX systems. I've defined stricmp as strcasecmp and strincmp as strncasecmp for UNIX systems. Commited in r5578.

[robe]

One small problem. The .sql script no longer works because its complaining geomval and the ST_DumpAsPolygons (ST_DumpAsPolygons is returning too many fields).

It seems we are now returning srid as a column in ST_DumpAsPolygons, but geomval only has a geometry and pixel value.

Why is it necessary to return srid as a column in ST_DumpAsPolgons. This does not make sense to me. It makes sense for the DumpAsWKT, but for ST_DumpAsPolygons, the srid should be encoded in the geometry. So seems to me it should be rewritten as

        SELECT ST_GeomFromText(wktgeomval.wktGeom, wktgeomval.srid), wktgeomval.val,
         FROM DumpAsWKTPolygons($1, 1) AS wktgeomval;

Instead of what it is now which is

        SELECT ST_GeomFromText(wktgeomval.wktGeom), wktgeomval.val,
        wktgeomval.srid FROM DumpAsWKTPolygons($1, 1) AS wktgeomval;

[robe]

Jorge,

Which version of GDAL should we be using? I was using I think GDAL 1.6.1 because that's the only working already compiled version for Python for windows that I got from cheeseshop. The 1.7 doesn't seem to exist binaries for and I haven't explored compiling with python bindings.

If I'm going to go that far, should I just use trunk which I think is 1.8dev or is it better to stick with 1.7 branch and even latest stable release (1.7.1 I think).

[pracine]

Right. Fixed now in r5580.

Replying to robe:

One small problem. The .sql script no longer works because its complaining geomval and the ST_DumpAsPolygons (ST_DumpAsPolygons is returning too many fields).

It seems we are now returning srid as a column in ST_DumpAsPolygons, but geomval only has a geometry and pixel value.

Why is it necessary to return srid as a column in ST_DumpAsPolgons. This does not make sense to me. It makes sense for the DumpAsWKT, but for ST_DumpAsPolygons, the srid should be encoded in the geometry. So seems to me it should be rewritten as

        SELECT ST_GeomFromText(wktgeomval.wktGeom, wktgeomval.srid), wktgeomval.val,
         FROM DumpAsWKTPolygons($1, 1) AS wktgeomval;

Instead of what it is now which is

        SELECT ST_GeomFromText(wktgeomval.wktGeom), wktgeomval.val,
        wktgeomval.srid FROM DumpAsWKTPolygons($1, 1) AS wktgeomval;

[robe]

Still not working for me.

Getting this error

ERROR: RASTER_dumpAsWKTPolygons requires 1 or 2 args

when I do something like

SELECT (ST_DumpAsPolygons(rast,1)).*
FROM dummy_rast
WHERE rid = 2;

Though curiously SELECT (ST_DumpAsPolygons(null,1)).*

Doesn't throw an error and returns an empty geomval set as expected.

[jorgearevalo]

I'm working on this function now. I may temporarily delete it, if causes instability...

[pracine]

Please don't delete it. It does not create instability. It's just a bug...

[jorgearevalo]

Ok. I'll commit changes as soon as it works.

[jorgearevalo]

Ok, I've reproduced the error. The first checking in the C SRF function rt_dump_as_wktpolygons is:

if ( (PG_NARGS() < 1 || PG_NARGS() > 2 )
{
elog(ERROR, "RASTER_dumpAsWKTPolygons requires 1 or 2 args\n");
PG_RETURN_NULL();
}

I've printed PG_NARGS() and it's a big number (16297). I don't know if it's related with the fact the raster, the first argument, must by detoasted with PG_DETOAST_DATUM(PG_GETARG_DATUM(0)), and it's considered like "a lot" of input arguments. Has sense?

[jorgearevalo]

I think I've fixed in r5582. I run this query:

select st_astext((st_dumpaspolygons(rast, 1)).geom) as geom, (st_dumpaspolygons(rast, 1)).val from dummy_rast where rid = 2;

And I get this result (CSV format here)

"geom";"val"
"POLYGON((3427927.75 5793244,3427927.75 5793243.9,3427927.8 5793243.9,3427927.8 5793244,3427927.75 5793244))";253
"POLYGON((3427927.8 5793244,3427927.8 5793243.85,3427927.85 5793243.85,3427927.85 5793243.75,3427928 5793243.75,3427928 5793243.8,3427927.95 5793243.8,3427927.95 5793243.85,3427927.9 5793243.85,3427927.9 5793243.95,3427927.85 5793243.95,3427927.85 5793244,3427927.8 5793244))";254
"POLYGON((3427927.85 5793244,3427927.85 5793243.95,3427927.9 5793243.95,3427927.9 5793244,3427927.85 5793244))";253
"POLYGON((3427927.9 5793244,3427927.9 5793243.95,3427928 5793243.95,3427928 5793244,3427927.9 5793244))";254
"POLYGON((3427927.9 5793243.95,3427927.9 5793243.9,3427927.95 5793243.9,3427927.95 5793243.95,3427927.9 5793243.95))";253
"POLYGON((3427927.95 5793243.95,3427927.95 5793243.85,3427928 5793243.85,3427928 5793243.95,3427927.95 5793243.95))";249
"POLYGON((3427927.75 5793243.9,3427927.75 5793243.85,3427927.8 5793243.85,3427927.8 5793243.9,3427927.75 5793243.9))";250
"POLYGON((3427927.9 5793243.9,3427927.9 5793243.85,3427927.95 5793243.85,3427927.95 5793243.9,3427927.9 5793243.9))";252
"POLYGON((3427927.75 5793243.85,3427927.75 5793243.8,3427927.8 5793243.8,3427927.8 5793243.85,3427927.75 5793243.85))";251
"POLYGON((3427927.8 5793243.85,3427927.8 5793243.8,3427927.85 5793243.8,3427927.85 5793243.85,3427927.8 5793243.85))";253
"POLYGON((3427927.95 5793243.85,3427927.95 5793243.8,3427928 5793243.8,3427928 5793243.85,3427927.95 5793243.85))";253
"POLYGON((3427927.75 5793243.8,3427927.75 5793243.75,3427927.8 5793243.75,3427927.8 5793243.8,3427927.75 5793243.8))";252
"POLYGON((3427927.8 5793243.8,3427927.8 5793243.75,3427927.85 5793243.75,3427927.85 5793243.8,3427927.8 5793243.8))";250

I'll test this polygons with Quantum GIS or OpenJump, to see how do they look like. So, I leave the ticket open, by now. Related ticket #297.

[jorgearevalo]

Minor bug solved in r5583

[robe]

Okay I'm getting output now. But still a couple of problems

1) Very minor -- but if I feed a raster with no bands, I get an assert error in postgres.exe that I have less than one band. I presume this is expected? Except it should just be a notice rather than kicking me into visual studio debugger - well hmm I guess since we are still in alpha it might be best to just leave as is if easier to debug.

2) This one is very strange and it seems very intermittent. Once it fails I can't get it to work again for another 10 or so tries and then it works. I haven't ever gotten just the inner subselect to work without doing restrictions on val which is even more bizzarre since I can't see how the ST_Union would ever work if the inner fails all the time. I'm running 1.5.1 so I'll upgrade to 1.5.2SVN and see if I experience any difference. I haven't looked at the rendered image yet to veerify it looks correct.

SELECT ST_Union(geom)
FROM (SELECT (ST_DumpAsPolygons(rast)).geom
FROM ch13.pele
) As foo

This gives error

ERROR: parse error - invalid geometry HINT: "...00,52 101,53 101,54 101,54 99,53 99))" <-- parse error at position 64 within geometry CONTEXT: SQL function "st_dumpaspolygons" statement 1

If I run Just the inner query I get the same error.

I'll attach the file in a bit.

[jorgearevalo]

Replying to robe:

Okay I'm getting output now. But still a couple of problems

1) Very minor -- but if I feed a raster with no bands, I get an assert error in postgres.exe that I have less than one band. I presume this is expected? Except it should just be a notice rather than kicking me into visual studio debugger - well hmm I guess since we are still in alpha it might be best to just leave as is if easier to debug.

Could you paste the error message and the query that caused it?

2) This one is very strange and it seems very intermittent. Once it fails I can't get it to work again for another 10 or so tries and then it works. I haven't ever gotten just the inner subselect to work without doing restrictions on val which is even more bizzarre since I can't see how the ST_Union would ever work if the inner fails all the time. I'm running 1.5.1 so I'll upgrade to 1.5.2SVN and see if I experience any difference. I haven't looked at the rendered image yet to veerify it looks correct.

SELECT ST_Union(geom)
FROM (SELECT (ST_DumpAsPolygons(rast)).geom
FROM ch13.pele
) As foo

This gives error

ERROR: parse error - invalid geometry HINT: "...00,52 101,53 101,54 101,54 99,53 99))" <-- parse error at position 64 within geometry CONTEXT: SQL function "st_dumpaspolygons" statement 1

If I run Just the inner query I get the same error.

Ok, I think I know the reason, and I fixed it. Please update to r5589. I executed st_astext over your query and I got this:

POLYGON((89 0,88 0,87 0,86 0,85 0,84 0,83 0,82 0,
81 0,80 0,79 0,78 0,77 0,76 0,75 0,74 0,73 0,72 0,
71 0,70 0,69 0,68 0,67 0,66 0,65 0,0 0,0 178,0 179,
0 180,0 181,0 182,0 183,0 184,0 185,0 186,0 188,
0 189,0 190,0 191,0 192,0 193,0 194,0 195,0 196,
0 439,158 439,159 439,160 439,161 439,162 439,
164 439,165 439,166 439,167 439,168 439,169 439,
298 439,299 439,299 405,299 404,299 402,299 401,
299 0,89 0))

Has sense? BTW, I'm using PostGIS version 1.5.0SVN

I'll attach the file in a bit.

I think you attached the file in ticket #297. Could be?

[robe]

Okay that works for me. Also fixes the crashing I was getting when trying to vectorize a topo map.