i.segment.hierarchical: Execution of subprocesses was not successful

[mlennert]

Using the NC demo dataset, I have the following issue with i.segment.hierarchical (in fact the GridModule):

g.region rast=ortho_2001_t792_1m
i.group ortho in=ortho_2001_t792_1m
i.segment.hierarchical group=ortho thresholds=0.01,0.02,0.03 output=seg_ortho_out outputs_pref=seg_ortho memory=2000 width=1524 height=1524 processes=4

Result:

Traceback (most recent call last):
  File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical", line 252, in <module>
    SEG.run()
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/pygrass/modules/grid/grid.py", line 553, in run
    raise RuntimeError(_("Execution of subprocesses was not successful"))
RuntimeError: L'exécution du sous-module a échoué
GRASS 7.1.svn (nc_spm_08):/data/home/mlennert > i.segment.hierarchical group=ortho thresholds=0.01,0.02,0.03 output=seg_ortho_out outputs_pref=seg_ortho memory=2000 width=1524 height=1524 processes=4
Traceback (most recent call last):
  File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical", line 252, in <module>
    SEG.run()
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/pygrass/modules/grid/grid.py", line 553, in run
    raise RuntimeError(_("Execution of subprocesses was not successful"))
RuntimeError: L'exécution du sous-module a échoué

Anna already reported such an error on the mailing list ​1, but I haven't seen a response. And contrary to what she reports, I have this problem systematically, not in random manner.

I also tried reducing the number of processes, but to no avail.

[mlennert]

I still have the same problem

i.group ls_ms in=lsat7_2002_10@PERMANENT,lsat7_2002_30@PERMANENT,lsat7_2002_20@PERMANENT,lsat7_2002_40@PERMANENT,lsat7_2002_50@PERMANENT,lsat7_2002_70@PERMANENT
g.region rast=lsat7_2002_70@PERMANENT
i.segment.hierarchical ls_ms threshold=0.1 output=ls_ms_seg width=200 height=200 overlap=10

results in

Traceback (most recent call last):
  File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical", line 252, in <module>
    SEG.run()
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/pygrass/modules/grid/grid.py", line 553, in run
    raise RuntimeError(_("Execution of subprocesses was not successful"))
RuntimeError: Execution of subprocesses was not successful

I don't really know how to start debugging this. Could someone point me in the right direction ?

[annakrat]

I remember GridModule had a debug parameter, which switched off the parallelization, so you start there.

[mlennert]

Replying to annakrat:

I remember GridModule had a debug parameter, which switched off the parallelization, so you start there.

Yes ! And Pietro actually put a nice big DEBUG=False at the beginning of the code. Blind me just didn't see it. Setting this to True led me to a bug in pygrass which still had a call to type='rast' instead of type='raster'. Corrected in trunk in r69392. But I get new errors so I'll work on that before I backport to the other grass7 branches.

Thanks for the pointer !

Moritz

[mlennert]

The first error was solved, but now I get a new error (same message) at the very end of an attempt to segment a fairly large image (400 mio pixels), i.e. after the patching of all tiles. Unfortunately, I do not get the error when setting the debug parameter, so I cannot see what exactly is the issue.

Any hints ?

Moritz

[mlennert]

Replying to mlennert:

Replying to annakrat:

I remember GridModule had a debug parameter, which switched off the parallelization, so you start there.

Yes ! And Pietro actually put a nice big DEBUG=False at the beginning of the code. Blind me just didn't see it. Setting this to True led me to a bug in pygrass which still had a call to type='rast' instead of type='raster'. Corrected in trunk in r69392.

Any objections against backporting this small change, to the two release branches ?

[mlennert]

Replying to mlennert:

The first error was solved, but now I get a new error (same message) at the very end of an attempt to segment a fairly large image (400 mio pixels), i.e. after the patching of all tiles. Unfortunately, I do not get the error when setting the debug parameter, so I cannot see what exactly is the issue.

I got a more concrete message:

Start patching the segments
Traceback (most recent call last):
  File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical", line 254, in <module>
    SEG.run()
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/pygrass/modules/grid/grid.py", line 567, in run
    self.patch()
  File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical", line 182, in patch
    self.start_row, self.start_col, self.out_prefix)
  File "/home/mlennert/.grass7/addons/etc/i.segment.hierarchical/isegpatch.py", line 46, in rpatch_map
    rrasts[-1].open('r')
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/pygrass/raster/__init__.py", line 210, in open
    raise OpenError(str_err)
grass.exceptions.OpenError: La carte n'existe pas, impossible de l'ouvrir en mode 'r'

Sorry for the French. In English it is: "The map does not exist, I can't open in 'r' mode"

As indicated, this comes from the open() function of the RasterRow class in lib/python/pygrass/raster/init.py, which is called in the iseg_patch.py file of i.segment.hierarchical:

        for col in range(len(rbbox)):
            msgr.percent(row, nrows, 1)
            rrasts.append(RasterRow(name=raster,
                                    mapset=mset_str % (start_row + row,
                                                       start_col + col)))
            rrasts[-1].open('r')

At this stage, I don't really understand which file it is and why it doesn't exist...

[neteler]

Replying to mlennert:

Replying to mlennert:

Yes ! And Pietro actually put a nice big DEBUG=False at the beginning of the code. Blind me just didn't see it. Setting this to True led me to a bug in pygrass which still had a call to type='rast' instead of type='raster'. Corrected in trunk in r69392.

Any objections against backporting this small change, to the two release branches ?

Done in r69410 and r69416.

Generally I think that such bugfixes can be backported right away.

[mlennert]

Replying to mlennert:

Replying to mlennert:

The first error was solved, but now I get a new error (same message) at the very end of an attempt to segment a fairly large image (400 mio pixels), i.e. after the patching of all tiles. Unfortunately, I do not get the error when setting the debug parameter, so I cannot see what exactly is the issue.

I got a more concrete message:

Start patching the segments
Traceback (most recent call last):
  File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical", line 254, in <module>
    SEG.run()
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/pygrass/modules/grid/grid.py", line 567, in run
    self.patch()
  File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical", line 182, in patch
    self.start_row, self.start_col, self.out_prefix)
  File "/home/mlennert/.grass7/addons/etc/i.segment.hierarchical/isegpatch.py", line 46, in rpatch_map
    rrasts[-1].open('r')
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/pygrass/raster/__init__.py", line 210, in open
    raise OpenError(str_err)
grass.exceptions.OpenError: La carte n'existe pas, impossible de l'ouvrir en mode 'r'

Sorry for the French. In English it is: "The map does not exist, I can't open in 'r' mode"

As indicated, this comes from the open() function of the RasterRow class in lib/python/pygrass/raster/init.py, which is called in the iseg_patch.py file of i.segment.hierarchical:

        for col in range(len(rbbox)):
            msgr.percent(row, nrows, 1)
            rrasts.append(RasterRow(name=raster,
                                    mapset=mset_str % (start_row + row,
                                                       start_col + col)))
            rrasts[-1].open('r')

At this stage, I don't really understand which file it is and why it doesn't exist...

Just pure speculation at this point: As this error arrives only when working on a large region, which means that if tiles are small, there are many tiles, could it be that rrasts becomes too big ?

[mlennert]

As the original bug is fixed and I have trouble reproducing the second issue reliably, I'm closing this bug and will open another one if needed.