r.out.gdal export in USGSDEM format takes very long time and finally crashes

[dido]

I cannot attach any test raster because the one that manifests the problem is very large. Here I apply raster metadata (I suppose any raster of this size will do):

r.info map=BGMdem_8x8m_holes_filled_feathered@PERMANENT                         
 +----------------------------------------------------------------------------+
 | Map:      BGMdem_8x8m_holes_filled_feat  Date: Fri Aug 17 13:30:15 2018    |
 | Mapset:   PERMANENT                      Login of Creator: a               |
 | Location: BG_DEM                                                           |
 | DataBase: E:\mapping\GRASS7DB                                              |
 | Title:    BGMdem_8x8m_holes_filled_feathered                               |
 | Timestamp: none                                                            |
 |----------------------------------------------------------------------------|
 |                                                                            |
 |   Type of Map:  raster               Number of Categories: 0               |
 |   Data Type:    DCELL                                                      |
 |   Rows:         52101                                                      |
 |   Columns:      77901                                                      |
 |   Total Cells:  4058720001                                                 |
 |        Projection: UTM (zone 35)                                           |
 |            N: 4928368.63825695    S: 4511560.45143113   Res: 8.00000359    |
 |            E: 663749.15806402    W: 40538.62472039   Res: 8.00003252       |
 |   Range of data:    min = -52.070711097653  max = 2925                     |
 |                                                                            |
 |   Data Description:                                                        |
 |    generated by r.mapcalc                                                  |
 |                                                                            |
 |   Comments:                                                                |
 |    if(isnull(defekti_rast_buf_recoded), BGMdem_8x8m_holes_filled,          |
 |    if(defekti_rast_buf_recoded < 100, BGMdem_8x8m_holes_filled * (100 -    |
 |    defekti_rast_buf_recoded) / 100 + ALOSdem_interp *                      |
 |    defekti_rast_buf_recoded / 100, BGMdem_8x8m_holes_filled))              |
 |                                                                            |
 +----------------------------------------------------------------------------+

At crash time progress indicator has not advanced at all. Tool output follows:

r.out.gdal -f --overwrite input=BGMdem_8x8m_holes_filled_feathered@PERMANENT output=E:\tmp\BGM-dem-f5p-csovp-wbf-cas-clgeo-s0_UTM_blended_20180816.dem format=USGSDEM type=Int16
Driver <USGSDEM> does not support direct writing. Using MEM driver for intermediate dataset.
WARNING: Precision loss: Raster map <BGMdem_8x8m_holes_filled_feathered@PERMANENT> of type DCELL to be exported as Int16. This can be avoided by using Float64.
WARNING: Forcing raster export
Checking GDAL data type and nodata value...
Using GDAL data type <Int16>
Input raster map contains cells with NULL-value (no-data). The value -32768 will be used to represent no-data values in the input map. You can specify a nodata value with the nodata option.
Exporting raster data to USGSDEM format...

[dnewcomb]

Are you shooting for the ASCII USGSDEM format as described at ​https://www.gdal.org/frmt_usgsdem.html?

From the message:

Driver <USGSDEM> does not support direct writing. Using MEM driver for intermediate >dataset.

Do you have enough memory to hold a 4 billion pixel raster in memory on the computer? See also ​https://www.gdal.org/frmt_mem.html Not sure if either the MEM driver or the USGS DEM format can handle 4 billion pixels. Is there a reason that you can't use GeoTIFF ( up to 4GB file Size)for the output file or GeoTIFF with the BIGTIFF option? ( up to 18,000 petabytes for a single file)

If you are determined to do ascii file output, why not r.out.ascii, https://grass.osgeo.org/grass74/manuals/r.out.ascii.html ?

WARNING: Precision loss: Raster map <BGMdem_8x8m_holes_filled_feathered@PERMANENT> of >type DCELL to be exported as Int16. This can be avoided by using Float64.

If you are going to truncate to integer, it might be better to use the int() or round() functions in r.mapcalc prior to exporting.

[dido]

GeoTIFF export runs fine, producing a 10G file. I chose to export to USGS DEM because I expect a tenfold decrease in file size. At least that can be observed if I do a DEM export of the same 10G GeoTIFF using e.g. GlobalMapper.

I don't have an idea if the MEM driver will handle raster this big. My PC has 16G of physical RAM and quite enough virtual memory.

Precision loss is fine for me as the input raster is also int16. I suppose GRASS uses 64 bit floats internally (DCELL) and the truncation to int16 will be handled by the export driver. Actually loss of speed due to truncation calculations is not an issue - GeoTIFF export runs for a quite reasonable time.

[neteler]

Replying to dido:

GeoTIFF export runs fine, producing a 10G file.

Re: file size - I suspect that you exported uncompressed GeoTIFF here. Did you use e.g. createopt="COMPRESS=LZW"?

See also https://grass.osgeo.org/grass78/manuals/r.out.gdal.html#export-a-large-raster-map-in-lzw-compressed-(big)-geotiff-format