[raster] overviews not marked as same alignment though main table is

[robe]

I'm guessing this is a rounding issue. I loaded up some 1-meter DEMS and chunked them to 128x128 and also created over views. So my main table has about 100,000 records.

The main table got registered as same alignment = true, the level 2 got registered also as same alignment = true, but the level 3 dems came in as same alignment = false.

I would assume if the core table was same alignment, then all the overviews should be as well -- no?

This is testing on PostGIS 2.0.0 64-bit on windows 2008 R2.

[Bborie Park]

You're probably right about this being a rounding issue. We need to add a tolerance parameter for the same alignment check. I'll see if I can reproduce this using some of my rasters. Might be a good time to play with DEM datasets.

[robe]

This is interesting. I loaded a smaller DEM that is 1000x1000. I didn't chunk it.

With -l 2,3,4,5 (levels 2,4,5) came in with same alignment = true

level 3 came in with same_alignment = false

with 333x333 block size. I guess because 1/3 goes on forever when converted to decimal. Not sure if that helps any.

[Bborie Park]

I have the same suspicion about the factor = 3 causing issues. Is this happening for a single DEM raster file or across a set?

[Bborie Park]

Curses. I'm unable to get any of my rasters to have the same alignment issue. Is it possible to share that smaller DEM file?

[Bborie Park]

I've tested a subset of my rasters in Linux 32-bit, Linux 64-bit and Windows 7 64-bit and am unable to recreate this issue.

[robe]

I'll send in a bit. I doesn't seem to happen in the same file. So you have to test at least two files I think.

I had to run this query to find a candidate:

SELECT v1.filename As v1_filename, o1.rid As o1_rid,  v2.filename As v2_filename, o2.rid as o2_rid
FROM o_3_vhms As o1 INNER JOIN o_3_vhms As o2 ON ST_SameAlignment(o1.rast,o2.rast) = false
  INNER JOIN vhms As v1 ON ST_Within(v1.rast::geometry, o1.rast::geometry) 
   INNER JOIN vhms As v2 ON ST_Within(v2.rast::geometry, o2.rast::geometry) 
limit 1;

[robe]

For the non-chunked version I had to change ST_Within to ST_Intersects. I assume that is because the bounding box is a bit fatter so they overlap. anyrate same query using intersects and within returns the same files at issue. Just getting permssion before I can send the files.

[Bborie Park]

As the rasters which uncovered the problem described in this ticket, the following describes the attributes of the rasters in a generic manner.

Raster 1

width: 100
height: 100
scale-x: 1
scale-y: -1
skew-x: 0
skew-y: 0
upperleft-x: 0
upperleft-y: 0

Raster 2

width: 100
height: 100
scale-x: 1
scale-y: -1
skew-x: 0
skew-y: 0
upperleft-x: 100
upperleft-y: 0

If the above two rasters are loaded in with raster2pgsql, the tiles are aligned. But for an overview factor of 3 (and probably anything else where the division of the width/height by the overview factor results in a repeating decimal), the alignment of the overview will not be aligned.

When creating overviews, raster2pgsql treats each input raster independently of all other input rasters. This does not work for situations like this. Instead, it may be better to get the overall extent for the set of input rasters and then process each individual input raster based upon the overall extent.

The one issue here is what to do if the input rasters are NOT of the same scale and skews? Use the "old" method (the current method) or fail?

[Bborie Park]

The first sentence in the prior comment should be...

As the rasters which uncovered the problem described in this ticket are publicly unavailable, the following describes the attributes of the rasters in a generic manner.

[Bborie Park]

The possible solution also isn't possible in any of the input rasters have a different SRID...

[robe]

Maybe the best thing to do is just make a note of it in the loader docs or something as expected behavior you should be cautious of. It's not really a big deal as long as you know about it because the pixel width / height chunk settings are rule of thumb anyway so the user can just ensure the dimensions they pick will be suitable dimensions for their overviews.

What we need -- which I think we kind of need anyway -- is a raster import calculator that asks questions like:

What do you want to do? and may suggest chunk size, and even overviews to pick

[Bborie Park]

I think you're right about just making a note of it in the loader docs indicating the expected behavior. I'm punting this ticket to 2.1 as it will require significant refactoring of raster2pgsql.

[Bborie Park]

Added note in using_raster_dataman.xml regarding this ticket in r9653.

[pracine]

Replying to dustymugs:

I think you're right about just making a note of it in the loader docs indicating the expected behavior. I'm punting this ticket to 2.1 as it will require significant refactoring of raster2pgsql.

It is not clear to me what refactoring is necessary in 2.1. Could you summarize it in a couple of sentence?

[Bborie Park]

Currently, overviews are generated on a per raster basis. This works great because each raster is treated independently.

But the limitation of this method is that though the rasters may be aligned, the overviews for a particular factor may not be aligned. A possible solution is for rasters to be processed first and create overviews afterwards. This would allow the loader to check that the rasters are aligned and then determine a new alignment grid for the overview at a specific factor. Only by doing this can we guarantee that the overview at a specific factor is aligned.

I still don't know if this is worth doing though as this can ONLY be run if all source rasters are aligned.

[Bborie Park]

I was thinking of a possible solution to this issue...

1. before doing anything to the set of import rasters, see if the set are aligned

2. if set of input rasters aligned, overviews are guaranteed to be aligned

As it looks like raster2pgsql may need to move away from using VRTs for overviews, this would be an appropriate change to make at the same time.

[Bborie Park]

I plan on rewriting raster2pgsql for 2.2 due to the number of enhancement tickets...

[strk]

As 2.2.0 has a milestone now, should this be retargetted from "PostGIS Future" to 2.2.0 ?

I stumbled on this ticket as I'm also having troubles with alignment, particularly aligned tiles that becomes unaligned after a call to ST_Resize() with the _same_ parameters (puzzling).

[strk]

I'm getting the same problem with "round" scale factors of 2,4,8,16. This is again with scales 1,-1 and offsets 0,0 -- that's the identity matrix, which could very well be the (or one) cause of it. See also #2914

[strk]

Sorry, I'm taking it back, for 2,4,8,16 it's not failing even with identity matrix. Not when using my new ST_CreateOverview and ST_Resize for resizing.

[robe]

Milestone renamed