Changes between Version 4 and Version 5 of GSoC/2019

Timestamp:

02/03/19 12:42:17 (6 years ago)

Author:

wenzeslaus

Comment:

add New easy-to-use CLI

GSoC/2019

@@ -64 +64 @@
 
 
+=== New easy-to-use CLI and API for GRASS GIS ===
+
+ * TL;DR: Make running of GRASS GIS modules as easy as it is to run GDAL commands.
+  * `grass run r.slope.aspect elevation=elevation.tiff slope=slope.tiff aspect=aspect.tiff`
+  * CLI like GDAL has.
+  * No GRASS Database, Location, Mapset to deal with.
+  * No import, export from user perspective.
+  * Reasonable defaults for things like region.
+  * CLI and API still allows user to specify any of the above.
+ * GRASS GIS requires GRASS GIS Database, Location and Mapset to be set up to maintain data consistency, efficiency and security. Unfortunately, this is cumbersome when GRASS GIS is not the primary tool user is using.
+ * There are different ways for calling GRASS modules without starting iterative GRASS session:
+  * Modules executed with the `--exec` interface (see [https://grass.osgeo.org/grass74/manuals/grass7.html#batch-jobs-with-the-exec-interface the grass7 manual page] >=7.2)
+  * `GRASS_BATCH_JOB`: same as newer `--exec` but through environmental variable and more limited
+  * Use `grass.script.setup` package from GRASS GIS (requires boilerplate to add the packages on path first)  
+  * Use the standalone `grass_session` package (new, see [https://grasswiki.osgeo.org/wiki/Working_with_GRASS_without_starting_it_explicitly#Python:_GRASS_GIS_7_with_an_external_library:_grass-session here])
+  * Set up environmental variables and "RC file" yourself (the classic method).
+ * None of these allows the user to skip the database setup phase. This leads to the need for constant reimplementing of setup, import and export steps in various software and environments including ''user scripts'' (in Bash, Python, R), QGIS Processing, gvSIG/SEXTANTE, uDig/JGrassTools, and all the web/server/cloud tools and applications which use GRASS GIS as a processing backend (e.g. PyWPS server).
+ * GRASS GIS itself can make it easier for the callers (at least in most cases) by implementing an interface which would allow to use GRASS GIS modules without explicit dealing with GRASS GIS database.
+ * The command line call using the proposed interface would look like these:
+
+{{{
+grass run r.lake elevation=some/file.tiff water_level=10 lake=some/new/file.tiff coordinates=100,520
+grass run r.slope.aspect elevation=file://.../elevation.tiff aspect=file://...aspect.tiff
+grass run r.slope.aspect elevation=https://.../elevation.tiff.zip aspect=file://...aspect.tiff
+}}}
+
+ * Basic execution phases:
+  * The `grass` command would have to parse the command line, compare it with the module XML interface description, find the files which should be maps (either using `file://` and ideally anything else), potentially download and uncompress, and import (or link) them, and then call the actual command (GRASS module).
+   * The input maps could be linked (external) rather than imported (except for the cases when projection differs) which should be faster than import.
+   * Doing the work in GRASS rather than in the other software would allow GRASS to make the decision about the details, for example the data exchange (r.external vs r.import vs r.in.gdal - see [https://github.com/qgis/QGIS/pull/5426#issuecomment-345067457 comment from MarkusN for QGIS Processing issue] or [https://lists.osgeo.org/pipermail/grass-dev/2017-November/086598.html mailing list]).
+  * GRASS Database would be created with an appropriate Location (projection based on input files or additional CLI input).
+   * The GRASS GIS Database, Location and Mapset should be created on the fly and deleted afterwards (the `.grassrc` wouldn't be used).
+  * Computational region would be set based on input file(s) or additional CLI input.
+  * Module execution.
+  * The output maps could be be also linked (e.g. r.external.out) with projection same as input which is should be faster then export.
+   * Ideally export (as well as import) should work also with PostGIS and databases provided through GDAL/OGR.
+ * Proposal should discuss and address how advanced things such raster algebra, multi-map inputs and outputs, temporal framework, cartography and visualization tools will work (or what are the limits).
+ * Use should be able to always specify the details manually:
+
+{{{
+grass run --mapset=/some/directory/grassdata/ncspm/practice1 r.lake elevation=some/file.tiff ...
+grass run --region="s=55600 n=60500..." --mask=some/mask.tiff r.lake elevation=some/file.tiff ...
+grass run --crs=EPSG:3358 --mask=some/mask.tiff r.lake elevation=some/file.tiff ...
+grass run --use=some/file_a.tiff --get=some/file_b.tiff r.slope.aspect elevation=file_a aspect=file_b
+}}}
+
+ * The system behind the interface will be inherently fragile, so it is necessary to write large amount of tests which would check different combinations of data types and projections.
+ * All the underlying code is expected to be in Python, so the project should involve also creation of Python API on the way.
+ * Bonus tasks:
+  * Making this work for the GUI in the same way. It is expected that this would work for any `g.gui.*` modules too but implementing similar mechanism also for module dialogs is more work (but some basic implementation might be quite straightforward).
+  * Making this connected to the standalone `grass_session` package.
+  * Generalization of the API, so that it incorporates also the concept of remote sessions (see e.g. [https://github.com/wenzeslaus/g.remote g.remote on GitHub])
+ * Current GRASS code involved:
+  * source:grass/trunk/lib/python/script/setup.py (library function(s) for Location setup in Python)
+  * source:grass/trunk/lib/init/grass.py (full GRASS GIS standard startup "script")
+ * See also:
+  * Tickets:
+   * #2424 PyGRASS does not work when GRASS is invoked from outside because it is not possible to change path to dynamic libraries in running process
+   * #2511 Starting GRASS in mapset which is not owned by the user
+   * #2579 Specify command to be executed as parameter of grass command (patch) and other suggestion for CLI, especially see the comment:15:ticket:2579 which summarizes the remaining issues after closing the ticket
+   * #2639 grass command should read commands from stdin as an interpreter would do
+   * #2678 GRASS starts even when Mapset is locked 
+   * #2679 Drop or fix setting of Location and Mapset using environmental variables
+   * #2681 Remove legacy meaning of LOCATION variable
+   * #2685 Add ignore lock flag to grass command
+   * #3537 Add functionalities `--tmp-location` and `--no-clean`
+  * Documentation:
+   * [https://grass.osgeo.org/grass74/manuals/grass_database.html GRASS GIS Database]
+   * [https://grass.osgeo.org/grass70/manuals/libpython/script.html#module-script.setup grass.script.setup]
+  * Mailing list and wikis:
+   * [https://lists.osgeo.org/pipermail/grass-dev/2018-March/087764.html grass-dev New CLI GSoC Idea: Comments, Mentors, Students Needed] (whole thread)
+   * [http://lists.osgeo.org/pipermail/grass-dev/2015-January/072979.html grass-dev QGIS Processing & GRASS (January 2015)]
+   * [http://grasswiki.osgeo.org/wiki/Working_with_GRASS_without_starting_it_explicitly Working with GRASS without starting it explicitly]
+   * wiki:Grass7/VectorLib/OGRInterface
+   * wiki:Grass7/VectorLib/PostGISInterface
+  * Other projects (see also the ones linked in the text):
+   * [https://github.com/qgis/QGIS/blob/0a1382a0be36d408aebd227fb0066f68c513e41e/python/plugins/processing/algs/grass7/Grass7Algorithm.py QGIS Processing Grass7Algorithm.py] source code
+   * [https://github.com/moovida/jgrasstools/blob/530c87f26d220f3eeff9d2fb9d21abd8821c00c3/grass/src/main/java/org/jgrasstools/grass/utils/ModuleSupporter.java uDig's JGrasstools] source code
+   * [http://sextante.googlecode.com/svn/trunk/soft/sextante_lib/sextante_gui/src/es/unex/sextante/gui/grass/ SEXTANTE grass] source code (broken link)
+   * [https://github.com/geopython/PyWPS/blob/425f0eb160f60714a6705a24ba926e03690ab371/pywps/Grass.py PyWPS source code] and [http://pywps.wald.intevation.org/documentation/process.html PyWPS documentation] (broken link)
+   * [https://code.google.com/p/wps-grass-bridge/source/browse/trunk/?r=97 wps-grass-bridge] source code (broken link)
+   * Related discussions on mailing list:
+    * [http://lists.osgeo.org/pipermail/grass-dev/2015-February/073867.html On GSoC Proposal New easy-to-use...]
+    * [http://lists.osgeo.org/pipermail/grass-dev/2015-March/074433.html GSOC project proposal]
+   * Similar complex command line interfaces to learn from:
+    * Git (a classic example of subcommand interface)
+    * Docker (subcommand interface combined with running other commands inside a container)
+    * !GeoGig (needs to get geospatial data in and out)
+    * !ImageMagic
+     * radical changes in CLI interface between versions [https://legacy.imagemagick.org/script/index.php 6] and [https://www.imagemagick.org/script/command-line-tools.php 7]
+     * new CLI has a single `magick` command with many parameters
+    * GMT
+     * [http://gmt.soest.hawaii.edu/doc/5.3.2/GMT_Docs.html#new-features-in-gmt-5 radical changes in CLI interface between versions 4 and 5]
+     * new CLI has a single `gmt` command and subcommands
+ * Test and training tasks:
+  * Extend `--exec` functionality:
+   * Add `--tmp-location` which runs `--exec` in a database/location/mapset which are created at the beginning and deleted at the end. See #3537, #3585, and r72790.
+   * Add `--clean` (current default) and `--no-clean` which say if `--exec` should clean the `.tmp` directory in the Mapset (for parallel execution). See #3537.
+   * Add `--lock` (current default) and `--no-lock` which say if `--exec` should lock the Mapset (for parallel execution). See also #2685 and the `-f` flag.
+   * Add `--region` to set a temporary computational region for the execution, e.g. `--region="raster=raster_name"`
+   * Add `--import-raster=some/file.tiff` which imports (r.in.gdal or r.import) a raster file (same for vector).
+   * Add `--link-raster=some/file.tiff` which links (r.external) a raster file (same for vector).
+   * Add `--export-raster=some/file.tiff` which exports (e.g. r.out.gdal) a raster file (same for vector).
+   * Add `--link-output-raster=some/file.tiff` which creates (r.external.out) a new (output) raster file (same for vector).
+  * Add features to `grass` executable interface:
+   * Make it possible to associate `*.gxw` files with `grass` executable (#1204) or at least add `--gui-workspace` or preferably just recognize it in the command line (distinguish it from database/location/mapset).
+   * Add `-c XY` for simple unprojected Locations (#3586).
+  * Solve one of the tickets linked above.
+ * Proposal: Student needs to show understanding of the GRASS GIS Database structure and significantly extend on text above.
+ * Language requirements: Python
+ * Mentors: Vaclav Petras
+ * Co-mentors: Pietro Zambelli
+ * Proposed by: Vaclav Petras
+
 == Tips for students ==