g.extension not installing addons behind proxy

[madi]

Related to ​https://lists.osgeo.org/pipermail/grass-user/2016-October/075337.html

when i try to use g.extension behind a proxy, following the example from the manual, it gives me the following error:

g.extension ext=i.segment.hierarchical proxy="http://xx.xxx.xxx.xx:xxxx"
Traceback (most recent call last):
  File "/usr/local/grass-7.3.svn/scripts/g.extension", line 1730, in <module>
    sys.exit(main())
  File "/usr/local/grass-7.3.svn/scripts/g.extension", line 1675, in main
    for ptype, purl in (p.split('=') for p in options['proxy'].split(',')):
ValueError: need more than 1 value to unpack

Using the form suggested by Pietro:

proxy="http=<value>,ftp=<value>"

g.extension ext=i.segment.hierarchical proxy="http=http://xx.xxx.xxx.xx:xxxx"

gives:

Fetching <i.segment.hierarchical> from GRASS GIS Addons repository (be
patient)...
ERROR: Extension <i.segment.hierarchical> not found

which is exactly the same as if I don't indicate the proxy at all.

FWIW, I normally use urllib2 to handle the proxy. However I have some difficulties to apply this simple method to the 1700+ rows code (handling several different addons sources for several os's) of g.extension, but I'm behind a proxy and I'm available to test if you have a patch.

Here's what I do in python:

import urllib2
import os
proxy = urllib2.ProxyHandler({'http':os.environ['HTTP_PROXY']})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
urllib2.urlopen('http://www.google.com')

Thanks

[neteler]

Proxy ticket also in #1434

[neteler]

Please indicate the operating system, the output of g.version -reg and the Python version.

[madi]

Can't check this now, as I'm currently not behind a proxy. Operating system at the time of reporting was CentOS.

[mmetz]

Replying to madi:

Related to ​https://lists.osgeo.org/pipermail/grass-user/2016-October/075337.html

when i try to use g.extension behind a proxy, following the example from the manual, it gives me the following error:

g.extension ext=i.segment.hierarchical proxy="http://xx.xxx.xxx.xx:xxxx"
Traceback (most recent call last):
  File "/usr/local/grass-7.3.svn/scripts/g.extension", line 1730, in <module>
    sys.exit(main())
  File "/usr/local/grass-7.3.svn/scripts/g.extension", line 1675, in main
    for ptype, purl in (p.split('=') for p in options['proxy'].split(',')):
ValueError: need more than 1 value to unpack

Using the form suggested by Pietro:

proxy="http=<value>,ftp=<value>"

g.extension ext=i.segment.hierarchical proxy="http=http://xx.xxx.xxx.xx:xxxx"

gives:

Fetching <i.segment.hierarchical> from GRASS GIS Addons repository (be
patient)...
ERROR: Extension <i.segment.hierarchical> not found

which is exactly the same as if I don't indicate the proxy at all.

According to Python docs, "Proxies which require authentication for use are not currently supported; this is considered an implementation limitation" in urllib, and g.extension uses urllib, not urllib2 to open urls.

FWIW, I normally use urllib2 to handle the proxy. However I have some difficulties to apply this simple method to the 1700+ rows code (handling several different addons sources for several os's) of g.extension, but I'm behind a proxy and I'm available to test if you have a patch.

Here's what I do in python:

import urllib2
import os
proxy = urllib2.ProxyHandler({'http':os.environ['HTTP_PROXY']})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
urllib2.urlopen('http://www.google.com')

That should also work for g.extension, including authentification.

[neteler]

Ticket retargeted after milestone closed

[neteler]

Ticket retargeted after milestone closed

[martinl]

Bump milestone to 7.4.4

[neteler]

Ticket retargeted after milestone closed

[neteler]

Fix attempt via ​https://github.com/GRASS-GIS/grass-ci/pull/6 submitted as r74116. Note the different proxy syntax in the manual page.

If ok, to be backported.

[mmetz]

Replying to neteler:

Fix attempt via ​https://github.com/GRASS-GIS/grass-ci/pull/6 submitted as r74116. Note the different proxy syntax in the manual page.

If ok, to be backported.

Note, this is introducing a new python dependency, on RHEL 7 and derivates, this is python-requests.

Do we have a list of required python 2 components and required python 3 components that can be updated? Ideally such a list would also indicate which python components are required only for the GUI (systems such as HPC may use GRASS without GUI).

[neteler]

Replying to mmetz:

Replying to neteler:

Fix attempt via ​https://github.com/GRASS-GIS/grass-ci/pull/6 submitted as r74116. Note the different proxy syntax in the manual page.

If ok, to be backported.

Note, this is introducing a new python dependency, on RHEL 7 and derivates, this is python-requests.

Yes, I realized that today.

Do we have a list of required python 2 components and required python 3 components that can be updated?

Also yes. See r74121, it is REQUIREMENTS.html.

Ideally such a list would also indicate which python components are required only for the GUI (systems such as HPC may use GRASS without GUI).

That's more or less indicated in that file.

So, I am undecided now about the backport.

[mlennert]

Replying to neteler:

Replying to mmetz:

Replying to neteler:

Fix attempt via ​https://github.com/GRASS-GIS/grass-ci/pull/6 submitted as r74116. Note the different proxy syntax in the manual page.

If ok, to be backported.

Note, this is introducing a new python dependency, on RHEL 7 and derivates, this is python-requests.

Yes, I realized that today.

Do we have a list of required python 2 components and required python 3 components that can be updated?

Also yes. See r74121, it is REQUIREMENTS.html.

Ideally such a list would also indicate which python components are required only for the GUI (systems such as HPC may use GRASS without GUI).

That's more or less indicated in that file.

So, I am undecided now about the backport.

I would plead for reworking this to use urllib2 (Python 2) which is already imported in the module and which seems to work according to madi. IIUC, authenticated proxy handling is part of urllib in Python 3, or ?

Let's try to avoid the endless dependency hell just out of easy availability of small helper packages. See the ​experience with npm in the Javascript world as a case in point. I would plead to keep most modules down to standard python library dependency, unless there really is no other way.

[mmetz]

Replying to mlennert:

Replying to neteler:

Replying to mmetz:

Replying to neteler:

Fix attempt via ​https://github.com/GRASS-GIS/grass-ci/pull/6 submitted as r74116. Note the different proxy syntax in the manual page.

If ok, to be backported.

Note, this is introducing a new python dependency, on RHEL 7 and derivates, this is python-requests.

Yes, I realized that today.

Do we have a list of required python 2 components and required python 3 components that can be updated?

Also yes. See r74121, it is REQUIREMENTS.html.

Ideally such a list would also indicate which python components are required only for the GUI (systems such as HPC may use GRASS without GUI).

That's more or less indicated in that file.

So, I am undecided now about the backport.

I would plead for reworking this to use urllib2 (Python 2) which is already imported in the module and which seems to work according to madi.

+1

Let's try to avoid the endless dependency hell just out of easy availability of small helper packages. See the ​experience with npm in the Javascript world as a case in point. I would plead to keep most modules down to standard python library dependency, unless there really is no other way.

+1

[annakrat]

In 74374:

g.extension: avoid python requests import, see #3179

[annakrat]

If somebody could test it with proxy, that would be great, I couldn't do that now. This should work with Python 2/3, it uses six package (which we need to require unlike requests).

If this works, we need to revert r74121.

As a side note, since we are moving to GitHub and we expect to get more pull requests, we should be more strict about what we accept. People typically care about getting a specific issue fixed, but this will often introduce more mess in the code or even break other functionality.

[pmav99]

requests need to be removed from REQUIREMENTS.html and the Dockerfile, too.

[martinl]

Remove Milestone from Addons bugreports.

[mmetz]

Replying to pmav99:

requests need to be removed from REQUIREMENTS.html and the Dockerfile, too.

Done in trunk r74454

[sbl]

Is there anything left here? Can the ticket be closed? Installation through proxies should work now...