Define dependencies for GRASS addons

[pmav99]

Some addons depend on other addons. E.g. r.lfp depends on r.stream.distance.

If you install r.lfp and you try to use it while r.stream.distance is not installed, you get the following traceback which is not particularly helpful; especially so if you are not familiar with Python.

Traceback (most recent call last):
  File "/home/grassuser/.grass7/addons/scripts/r.lfp", line 111, in <module>
    sys.exit(main())
  File "/home/grassuser/.grass7/addons/scripts/r.lfp", line 43, in main
    calculate_lfp(input, output, coords)
  File "/home/grassuser/.grass7/addons/scripts/r.lfp", line 67, in calculate_lfp
    distance=flds)
  File "/usr/lib/grass70/etc/python/grass/script/core.py", line 392, in run_command
    ps = start_command(*args, **kwargs)
  File "/usr/lib/grass70/etc/python/grass/script/core.py", line 361, in start_command
    return Popen(args, **popts)
  File "/usr/lib/grass70/etc/python/grass/script/core.py", line 64, in __init__
    subprocess.Popen.__init__(self, args, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

There should be a way to define GRASS dependencies. E.g. something like requirements.txt that is being used by Python packages.

Optimally g.extension should read the list of depedencies and install missing ones automatically. If g.extension does go down this road, then an additional flag for skipping dependency checks might also be a good idea.

Perhaps this could/should also be expanded to additional non-GRASS related dependencies. E.g. r.denoise requires the existence of the mdenoise binary. In this case IMHO a graceful failure message would be preferable to a successful installation of a non-working addon.

[pmav99]

Transferring Helmut Kudrnovsky's answer here:

have a look here: ​https://trac.osgeo.org/grass/browser/grass-addons/grass7/raster/r.basin/r.basin.py#L82

# check requirements
def check_progs():
    found_missing = False
    for prog in ('r.hypso', 'r.stream.basins', 'r.stream.distance', 'r.stream.extract',
    'r.stream.order','r.stream.snap','r.stream.stats', 'r.width.funct'):
        if not grass.find_program(prog, '--help'):
            found_missing = True
            grass.warning(_("'%s' required. Please install '%s' first using 'g.extension %s'") % (prog, prog, prog))
    if found_missing:
        grass.fatal(_("An ERROR occurred running r.basin"))

[martinl]

Milestone renamed

[neteler]

Ticket retargeted after milestone closed

[martinl]

All enhancement tickets should be assigned to 7.6 milestone.

[martinl]

Ticket retargeted after milestone closed

[martinl]

Ticket retargeted after milestone closed

[martinl]

Remove Milestone from Addons bugreports.

[neteler]

It might be a good idea to implement some function(s) for dependency lookup from a module-specific requirements.txt - I see some (randomly seeked) code snippet here:

​https://stackoverflow.com/a/16624700/452464

[neteler]

Ticket retargeted after milestone closed

[neteler]

Ticket retargeted after milestone closed

[neteler]

Ticket retargeted after milestone closed

[sbl]

Pyhon modules are supposed to use lazy imports and should catch if a required python library is available at runtime. The same should be true for dependency on other addons (should probably added to the SUBMITTING guidelines).

For C modules, installation of libraries through g.extension is probably not very straight forward.

So the question is a bit, where this is supposed to be handeled. Personally, I would probably prefer if the addon itself catches evtl. missing dependencies and gives a clear error message. Silently installing stuff can be a bit scary...

[wenzeslaus]

Replying to sbl:

Pyhon modules are supposed to use lazy imports and should catch if a required python library is available at runtime.

+1

For C modules, installation of libraries through g.extension is probably not very straight forward.

Right. For some addons, we actually have that as an optional dependency in the core. However, pip, R, and conda can actually do that, so perhaps if we use them in background, they could take care of that.

There have been suggestions like this in the past, but a lot of changed in past years. pip is now basically part of Python. conda is quite common. There is a lot of R packages using C++. For example itzi is using pip rather than g.extension.

Silently installing stuff can be a bit scary...

Why do you think it is scary for GRASS GIS? pip, R, conda, apt, yum, ... all install dependencies of a package you asked for. Why this should not happen for GRASS GIS?

[sbl]

Replying to wenzeslaus:

Silently installing stuff can be a bit scary...

Why do you think it is scary for GRASS GIS? pip, R, conda, apt, yum, ... all install dependencies of a package you asked for. Why this should not happen for GRASS GIS?

Yes, you are right.

So, how would we setup a requirements.txt?

Should it contain different sections, like e.g.

GRASS_addons
    r.area

R_packages
    ggplot2

Python_libraries
    rpy2>=1.1
    
cmd_tools
    iconv
    cs2cs

libraries
    libgdal-grass

Cause the way these dependencies are installed varies a bit, with a dpendency_check function for each of them...

Maybe better with a dependency json:

{
"GRASS_addons": [
    {
      "name": "r.area"
    }],
"Python_libraries": [
    {
      "name": "rpy2",
      "version": "1.1"
      "version_check": ">="
    }], ...
}

In the functions we might have to think about OS specific aspects (package manager (incl. conda vs. pip), library names) as well as maybe versions (e.g. numpy>=1.17).

Also, should a failed dependency check block the installation? Because that it is not a trivial task I would opt for a warning rather than an error message in case of missing or unresolved dependencies...

Lastly, should a dependency check function go into g.extension or into the python library (script or pygrass). In the python lib it could be used by AddOn devs at runtime (e.g. if it takes a json formated string or dict)...

[neteler]

In a simple way, I have defined the needed requirements of t.rast.mosaic addon as a small shell script:

​https://github.com/mundialis/t.rast.mosaic/blob/main/requirements.sh

Of course it would be better if g.extension could take care...

[sbl]

Draft for a Python function to handle dependencies in g.extension

[sbl]

I just added a draft for a python script that could take care of dependencies in Python (conda not tested) and R packages. Other dependencies (e.g. C-libraries, commandline tools) are just checked, neither loading of the libraries not installation is supported at the moment (not sure if the latter is realistic). The function has three modes: check (warns of missing dependencies), install (installs missing dependencies (if possible), abort (stops if dependencies are missing). It takes the following arguments as input:

• dependency_type
• dependency
• version=None
• version_comparison=None
• repository=None
• optional=False

and could be fed e.g. from a table describing dependencies. dependency versions, and specific repositories are not supported yet. Let me know what you think, and i see how it could be integrated in g.extension.

[hcho]

Replying to sbl:

I just added a draft for a python script that could take care of dependencies in Python (conda not tested) and R packages. Other dependencies (e.g. C-libraries, commandline tools) are just checked, neither loading of the libraries not installation is supported at the moment (not sure if the latter is realistic). The function has three modes: check (warns of missing dependencies), install (installs missing dependencies (if possible), abort (stops if dependencies are missing). It takes the following arguments as input:

• dependency_type
• dependency
• version=None
• version_comparison=None
• repository=None
• optional=False

and could be fed e.g. from a table describing dependencies. dependency versions, and specific repositories are not supported yet. Let me know what you think, and i see how it could be integrated in g.extension.

handle_dependencies.py looks good. Just have a few comments.

cmd R 3.4 >=
cmd cmdfail 3.4 >=
R_package igraph 0.7.1 >=
R_package R_fail_test 0.7.1 >=

looks unnatural and error-prone. Would it be possible to change this format to

cmd R >= 3.4
cmd cmdfail >= 3.4
R_package igraph >= 0.7.1
R_package R_fail_test >= 0.7.1

Also, how about defining dependency information inside modules themselves instead of using an external file? We already have G_option_*() functions to handle option dependency. Maybe, G_module_requires(void *first, ...), G_module_requires_python(void *first, ...), G_module_requires_r(void *first, ...) and

G_module_requires("r.stream.distance", NULL);
G_module_requires_python("numpy", "gdal", NULL);

Then, add a new global flag --dependencies to spit out dependency information?

Just my 2 cents.

[wenzeslaus]

Replying to sbl:

I just added a draft for a python script that could take care of dependencies in Python (conda not tested) and R packages. Other dependencies (e.g. C-libraries, commandline tools) are just checked, neither loading of the libraries not installation is supported at the moment (not sure if the latter is realistic).

The current code looks straightforward and perhaps worth trying it in action (you can make an addon module which installs dependencies and then runs g.extension on the actual module).

However, I think this can also get really complex. What about turning this the other way around and focusing supporting GRASS modules which are Python packages installed with conda or pip like itzi? They would just use the existing systems for dependencies. Still, g.extension could use handle_dependencies.py logic, but it would use it to install the module rather than the dependencies.

[wenzeslaus]

Replying to hcho:

Replying to sbl:

cmd R 3.4 >=
cmd cmdfail 3.4 >=
R_package igraph 0.7.1 >=
R_package R_fail_test 0.7.1 >=

looks unnatural and error-prone. Would it be possible to change this format to

cmd R >= 3.4
cmd cmdfail >= 3.4
R_package igraph >= 0.7.1
R_package R_fail_test >= 0.7.1

Instead of adding another dependency format which is a custom format, I would suggest at least using an existing general format, e.g., JSON. However, going a step further might be even better, Conda has a somewhat general dependency file format (environment.yml) or, alternatively, tools like ​Binder take advantage of existing dependency formats, i.e., use requirements.txt for Python, DESCRIPTION for R, environment.yml for Conda, etc. When you consider setup.py for Python, this transitions nicely to my suggestion about supporting modules which are packages.

Also, how about defining dependency information inside modules themselves instead of using an external file? We already have G_option_*() functions to handle option dependency. Maybe, G_module_requires(void *first, ...), G_module_requires_python(void *first, ...), G_module_requires_r(void *first, ...) ... Then, add a new global flag --dependencies to spit out dependency information?

In this case, you would have to compile and run the module before figuring out the dependencies. This would not be possible for C/C++ and it still insist on lazy imports. Even if we say these two issues are not bothering us in the end given other issues, this would be extra confusing since it is exactly the opposite of what any other dependency/packaging system is doing.

[wenzeslaus]

Replying to sbl:

Lastly, should a dependency check function go into g.extension or into the python library...

g.extension can call it, but it should be in the library - length of g.extension.py being one reason.

...(script or pygrass)

This looks like a new subpackage of grass. Python and R use some names already: setuptools, distutils, devtools, ...