[raster] ST_MapAlgebra -- I think we have a memory leak

[robe]

I tried to do this on my test elephant

-- its a single tile -- I forget the size O

ALTER TABLE ch13.pele ADD column map_rast raster;
UPDATE ch13.pele 
SET map_rast = ST_MapAlgebra(rast,'CASE WHEN rast BETWEEN 100 and 250 THEN 1 WHEN rast = 252 THEN 2 WHEN rast BETWEEN 253 and 254 THEN 3 ELSE 0 END', '2BUI') ; 

and it crashes the server after about 13 seconds.

This is on Windows XP latest trunk.

[jorgearevalo]

Server crashes when returning serialized generated raster. Debugging.

[dzwarg]

Jorge:

I traced it back to trunk/raster/rt_core/rt_api.c#L3025. Hope this helps.

[dzwarg]

The file changed since I made that last comment.

See trunk/raster/rt_core/rt_api.c@6911#L3025​

[dzwarg]

Hello Jorge,

I have found that the size of the raster triggers this bug. It happens to me when I process 100x100 rasters, but it does not happen when processing 10x10 rasters.

[jorgearevalo]

Please, give a try to r6958. Now the server doesn't crash for me, but I had to comment SPI_finish calls. Not recommended, I think. I've found this 10-years old unresponded question, with the same problem: ​http://www.mail-archive.com/pgsql-general@postgresql.org/msg18558.html

I think, I'll try this: ​http://www.mail-archive.com/pgsql-general@postgresql.org/msg79320.html

And asking pgsql list too. Anyway, give it a try.

[jorgearevalo]

Replying to dzwarg:

The file changed since I made that last comment.

See trunk/raster/rt_core/rt_api.c@6911#L3025​

Thanks David. Really useful :-)

[jorgearevalo]

Just for the record. Differents behaviors of the function in PostgreSQL 8.4.7 and PostgreSQL 9.0.3 when uncommenting the SPI_finish call:

• PostgreSQL 8.4.7: Crash caused by SIGSEV, segmentation fault.
  
• PostgreSQL 9.0.3: Error deserializing raster (Unknown pixeltype: 15). And incorrect number of bands reported (32639 instead of 1)

Link to SQL file with the raster table used, 14400 rows: ​http://dl.dropbox.com/u/6599273/srtm_35_04.tgz

Query that causes different behaviours when SPI_finish is uncommented: select st_mapalgebra(rast, 1, 'rast + 10', '-1 * rast') from srtm_35_04 where rid < 3651

The where clause is not necessary. The problem simply appears when the query processes more than 3650 rows.

[pracine]

And what happen when SPI_finish is commented out?

[jorgearevalo]

Without SPI_finish call you get this warning message:

WARNING: transaction left non-empty SPI stack
HINT: Check for missing "SPI_finish" calls.

But the application doesn't crash (for me at least). Now is commented.

[pracine]

My first attemp at the C implementation of ST_MapAlgebra ever:

CREATE TYPE geomvalxy AS (
    geom geometry,
    val double precision,
    x int,
    y int
);

CREATE OR REPLACE FUNCTION ST_PixelAsPolygons(rast raster, band integer) 
    RETURNS SETOF geomvalxy AS 
    $$
    DECLARE
        rast alias for $1;
        w integer;
        h integer;
        x integer;
        y integer;
        result geomvalxy;
    BEGIN
        SELECT st_width(rast), st_height(rast)
        INTO w, h;
        FOR x IN 1..w LOOP
             FOR y IN 1..h LOOP
                 SELECT ST_PixelAsPolygon(rast, band, x, y), ST_Value(rast, band, x, y), x, y INTO result;
            RETURN NEXT result;
         END LOOP;
        END LOOP;
        RETURN;
    END;
    $$
    LANGUAGE 'plpgsql';

-- This query works
SELECT ST_MapAlgebra(ST_AddBand(ST_MakeEmptyRaster(5, 5, 0, 0, 1, 1, 0, 0, -1), '8BUI'::text, 0, -1), 'rast + 1')

-- This query crashes (certainly because SPI_finish was not done)
SELECT ST_PixelAsPolygons(ST_MapAlgebra(ST_AddBand(ST_MakeEmptyRaster(5, 5, 0, 0, 1, 1, 0, 0, -1), '8BUI'::text, 0, -1), 'rast + 1'))

with

ERROR:  SPI_connect failed: SPI_ERROR_CONNECT
CONTEXT:  SQL function "st_pixelaspolygons" statement 1

********** Error **********

ERROR: SPI_connect failed: SPI_ERROR_CONNECT
SQL state: XX000
Context: SQL function "st_pixelaspolygons" statement 1

[robe]

Jorge,

Sorry to take so long to reply. On my windows mingw install it gives same message as yours and answers are in right ball park.

Takes 13 seconds to do the map call.

SELECT distinct (r).val
FROM (SELECT ST_DumpAsPolygons(map_rast) as r
FROM ch13.pele) As foo

Takes 359 ms.

So ST_DumpAsPolygons seems much faster and my elephant is recognizable in open jump. I haven't done any other tests though.

[jorgearevalo]

Replying to robe:

Jorge,

Sorry to take so long to reply. On my windows mingw install it gives same message as yours and answers are in right ball park.

Takes 13 seconds to do the map call.

SELECT distinct (r).val
FROM (SELECT ST_DumpAsPolygons(map_rast) as r
FROM ch13.pele) As foo

Takes 359 ms.

So ST_DumpAsPolygons seems much faster and my elephant is recognizable in open jump. I haven't done any other tests though.

Great news :-).

[jorgearevalo]

For the record. The query doesn't crash in Ubuntu 10.10 64bits machine. I guess is a matter of time (takes more time to full the available memory). Classic memory leak here.

Question sent to pgsql-general list:http://archives.postgresql.org/pgsql-general/2011-04/msg00130.php

[pracine]

Are you sure the bug is not related to #851 and our own memory management with the context? I would start by understanding 851 and fix it properly. I would not be surprised to see 837 disappear with 851 fixed.

[jorgearevalo]

Yes, they're probably related. I'm debugging #851 now.

[jorgearevalo]

Fixed in r7025. I close the ticket.