[[TOC]]

= Python 3 support in GRASS GIS

[[Image(grass_gis_python3_logo_blending.png, 300)]]


**Update Summer 2019: GRASS GIS 7.8.0 has been released with Python 3 support! This page is kept for historical reasons.**


===============================================================================================
== Python versions
* keep compatibility with 2.7 (may still work with 2.6, but we don't care)
* port to work with 3.5

== Python components include:
* Python Scripting Library
* PyGRASS
* Temporal Library
* ctypes
* wxGUI

== Python Scripting Library ==

What to consider:

* The API is used not only by the GRASS Development Team (core devs) but in general, e.g. by writing addons or custom user scripts.
 * Maybe the core devs can be convinced to follow certain special practices for the core modules, but it doesn't seem realistic that addon contributors will follow them if there are too distant from what is standard for the language (less serious example is requiring PEP8 conventions versus some custom ones).
 * The purpose of the API is to make it simple for people to use and extend GRASS GIS.
* Trained (and even the non-trained) Python 3 programmers will expect API to behave in the same way as the standard library and language in general.
 * One writes `os.environ['PATH']`, not `os.environ[b'PATH']` nor `os.environ[u'PATH']`.
 * GUI needs Unicode at the end.

Possible approach:
* functions need to accept unicode and return unicode
* functions wrapping Python Popen class (read_command, run_command, ...) will have parameter encoding
 * encoding=None means expects and returns bytes (the current state)
 * encoding='default' means it takes current encoding using utils._get_encoding()
 * encoding='utf-8' takes whatever encoding user specifies, e.g., utf-8 in this case
 * this is similar to [https://docs.python.org/3.6/library/subprocess.html#subprocess.Popen Popen class in Python3.6]
 * by default encoding='default' to enable expected behavior by users, the following example shows Python3 behavior if we keep using bytes instead of unicode:

{{{
# return bytes
ret = read_command('r.what', encoding=None, ...

for item in ret.splitlines():
    line = item.split('|')[3:]

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: a bytes-like object is required, not 'str'

# we would have to use:
for item in ret.splitlines():
    line = item.split(b'|')[3:]
}}}

Unicode as the default type in the API, e.g. for keys, but also for many values, is supported by Unicode being the default string literal type in Python 3. API users will expect that expressions such as hypothetical `computation_region['north']` will work. Unlike in Python 2, there is a difference in Python 3 between `computation_region[u'north']` and `computation_region[b'north']`. See comparison of dictionary behavior in 2 and 3:

{{{
#!python
# Python 2
>>> d = {'a': 1, b'b': 2}
>>> d['b']
2
>>> d[u'b']
2
>>> # i.e. no difference between u'' and b'' keys
>>> and that applies for creating also:
>>> d = {u'a': 1, b'a': 2}
>>> d['a']
2
>>> # because
>>> d
{u'a': 2}
# Python 3
>>> # unlike in 2, we get now two entries:
>>> d = {'a': 1, b'a': 2}
>>> d
{b'a': 2, 'a': 1}
>>> d['a']
1
>>> d[b'a']
2
>>> # it becomes little confusing when we combine unicode and byte keys
>>> d = {'a': 1, b'b': 2}
>>> d['a']
1
>>> d['b']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'b'
>>> d[b'b']
2
>>> # in other words, user needs to know and specify the key as bytes
}}}

== Python 2 and Python 3 differences
The most important change between these two versions is dealing with strings.
* In Python 2:
 * Bytes == Strings
 * Unicodes != Strings
* In Python 3:
 * Bytes != Strings
 * Unicodes == Strings

{{{#!th style="background: #ddd"
'''Label'''
}}}
{{{#!th style="background: #ddd"
'''Python 2'''
}}}
{{{#!th style="background: #ddd"
'''Python 3'''
}}}
|-----------------------
{{{#!td 
String:
}}}
{{{#!td 
{{{
>>> 'sample'
'sample'
}}}
}}}
{{{#!td 
{{{
>>> 'sample'
'sample'
}}}
}}}
|-----------------------
{{{#!td 
Unicode:
}}}
{{{#!td 
{{{
>>> 'sample'
u'sample'
}}}
}}}
{{{#!td 
{{{
>>> 'sample'
'sample'
}}}
}}}
|-----------------------
{{{#!td 
Bytes:
}}}
{{{#!td 
{{{
>>> b'sample'
'sample'
}}}
}}}
{{{#!td 
{{{
>>> b'sample'
b'sample'
}}}
}}}
|-----------------------
{{{#!td 
Types:
}}}
{{{#!td 
{{{
>>> type('xx'), type(u'xx'), type(b'xx')
(<type 'str'>, <type 'unicode'>, <type 'str'>)
}}}
}}}
{{{#!td 
{{{
>>> type('xx'), type(u'xx'), type(b'xx')
(<class 'str'>, <class 'str'>, <class 'bytes'>)
}}}
}}}

When special characters are involved:

{{{#!th style="background: #ddd"
'''Label'''
}}}
{{{#!th style="background: #ddd"
'''Python 2'''
}}}
{{{#!th style="background: #ddd"
'''Python 3'''
}}}
|-----------------------
{{{#!td 
String:
}}}
{{{#!td 
{{{
>>> 'Příšerný kůň'
'P\xc5\x99\xc3\xad\xc5\xa1ern\xc3\xbd k\xc5\xaf\xc5\x88'
}}}
}}}
{{{#!td 
{{{
>>> 'Příšerný kůň'
'Příšerný kůň'
}}}
}}}
|-----------------------
{{{#!td 
Unicode:
}}}
{{{#!td 
{{{
>>> u'Příšerný kůň'
u'P\u0159\xed\u0161ern\xfd k\u016f\u0148'
}}}
}}}
{{{#!td 
{{{
>>> u'Příšerný kůň'
'Příšerný kůň'
}}}
}}}
|-----------------------
{{{#!td 
Bytes:
}}}
{{{#!td 
{{{
>>> b'Příšerný kůň'
'P\xc5\x99\xc3\xad\xc5\xa1ern\xc3\xbd k\xc5\xaf\xc5\x88'
}}}
}}}
{{{#!td 
{{{
>>> b'Příšerný kůň'
SyntaxError: bytes can only contain ASCII literal characters.
}}}
}}}

For Python 3, bytes objects can not contain character literals other than ASCII, therefore, we use bytes() to convert from unicode/string to byte object.
{{{
>>> bytes('Příšerný kůň', encoding='utf-8')
b'P\xc5\x99\xc3\xad\xc5\xa1ern\xc3\xbd k\xc5\xaf\xc5\x88'
}}}
To decode, use decode():
{{{
>>>b'P\xc5\x99\xc3\xad\xc5\xa1ern\xc3\xbd k\xc5\xaf\xc5\x88'.decode()
'Příšerný kůň'
}}}
We already have encode and decode functions available in (from grass.script.utils import encode, decode) lib/python/script/utils.py that makes it easy for us to convert back and forth. To make it work with Python3, made changes in those functions to avoid syntax errors and exceptions.

**Dictionary Differences**
{{{#!th style="background: #ddd"
'''Python 2'''
}}}
{{{#!th style="background: #ddd"
'''Python 3'''
}}}
|-----------------------
{{{#!td
{{{
>>> d = {'a': 1, b'b': 2, u'c':3}

>>>d['a']
1
>>>d[b'a']
1
>>>d[u'a']
1

>>>d['b']
2
>>>d[b'b']
2
>>>d[u'b']
2

>>>d['c']
3
>>>d[b'c']
3
>>>d[u'c']
3
}}}
}}}
{{{#!td 
{{{
>>> d = {'a': 1, b'b': 2, u'c':3}

>>>d['a']
1
>>>d[b'a']
KeyError: b'a'
>>>d[u'a']
1

>>>d['b']
KeyError: 'b'
>>>d[b'b']
2
>>>d[u'b']
KeyError: 'b'

>>>d['c']
3
>>>d[b'c']
KeyError: b'c'
>>>d[u'c']
3
}}}
}}}
|-----------------------
{{{#!th colspan=2
When special characters are involved:
}}}
|-----------------------
{{{#!td
{{{
>>> d = {'ř': 1, b'š': 2, u'ý':3}

>>>d['ř']
1
>>>d[b'ř']
1
>>>d[u'ř']
KeyError: u '\u0159'

>>>d['š']
1
>>>d[b'š']
1
>>>d[u'š']
KeyError: u '\u0161'

>>>d['ý']
KeyError: '\xc3\xbd'
>>>d[b'ý']
KeyError: '\xc3\xbd'
>>>d[u'ý']
3
}}}
}}}
{{{#!td 
{{{
>>> d = {'ř': 1, b's': 2, u'ý':3}

>>>d['ř']
1
>>>d[b'ř']
SyntaxError
>>>d[u'ř']
1

>>>d['s']
KeyError: 's'
>>>d[b's']
2
>>>d[u's']
KeyError: 's'

>>>d['ý']
3
>>>d[b'ý']
SyntaxError
>>>d[u'ý']
3
}}}
}}}

== How to write Python 3 compatible code
To check which Python version is being used, use sys.verson_info like:
{{{
import sys
if sys.version_info.major >= 3:
    //…
else:
    //...
}}}

{{{#!th style="background: #ddd"
'''Label'''
}}}
{{{#!th style="background: #ddd"
'''Python 2'''
}}}
{{{#!th style="background: #ddd"
'''Python 3'''
}}}
{{{#!th style="background: #ddd"
'''Python 2 and 3 compatible solution'''
}}}
|-----------------------
{{{#!td 
Strings -
bytes/str/unicode
}}}
{{{#!td 
}}}
{{{#!td 
}}}
{{{#!td 
{{{
Use decode and encode functions from grass.script.utils
}}}
}}}
|-----------------------
{{{#!td 
String functions
}}}
{{{#!td 
{{{
String.split
String.join
}}}
}}}
{{{#!td
{{{
"".split(), "".join(" ")
}}}
}}}
{{{#!td 
Use functions as in Python 3
}}}
|-----------------------
{{{#!td 
String types
}}}
{{{#!td 
{{{
Types.StringTypes
Types.Stringtype
}}}
}}}
{{{#!td
{{{
str or unicode
str
}}}
}}}
{{{#!td 
Use direct types like str, unicode, bytes as:
{{{
if sys.version_info.major == 3:
    unicode = str
else:
    bytes = str
}}}
}}}
|-----------------------
{{{#!td 
ps.communicate():
stdout, stderror
}}}
{{{#!td 
}}}
{{{#!td
}}}
{{{#!td 
Use decode: 
{{{
from grass.script.utils import decode

stdout = decode(stdout)
stderr = decode(stdout)
}}}
}}}
|-----------------------
{{{#!td 
Opening files
}}}
{{{#!td
{{{
fileName = file(“example.txt”, w)
}}}
}}}
{{{#!td
{{{
fileName = open(“example.txt”, w)
}}}
}}}
{{{#!td 
Replace file() with open() to open files
}}}
|-----------------------
{{{#!td 
Filter function
}}}
{{{#!td
{{{
filter(a_function, a_sequence)
}}}
Filter returns a list
}}}
{{{#!td
{{{
filter(a_function, a_sequence)
}}}
Filter returns an iterator, not a list
}}}
{{{#!td 
Explicit conversion to list using:
{{{
list( filter(a_function, a_sequence) )
}}}
}}}
|-----------------------
{{{#!td 
Urlopen proxies
}}}
{{{#!td
}}}
{{{#!td
}}}
{{{#!td 
Create proxy handler:
https://docs.python.org/3.0/library/urllib.request.html#urllib.request.ProxyHandler

Read more:
http://www.diveintopython3.net/porting-code-to-python-3-with-2to3.html#urllib
}}}
|-----------------------
{{{#!td 
Import errors
}}}
{{{#!td
}}}
{{{#!td
}}}
{{{#!td 
Use try/except to catch Import errors and deal accordingly
}}}
|-----------------------
{{{#!td 
Relative Imports
}}}
{{{#!td
{{{
import example
from x import y
}}}
}}}
{{{#!td
{{{
from . import example
from .x import y
}}}
}}}
{{{#!td 
Use period (.) in import calls
}}}
|-----------------------
{{{#!td 
Dictionary methods
}}}
{{{#!td
{{{
a_dictionary.keys()
a_dictionary.items()
a_dictionary.iterkeys()
}}}
}}}
{{{#!td
{{{
list(a_dictionary.keys())
list(a_dictionary.items())
iter(a_dictionary.keys())
}}}
}}}
{{{#!td 
Read more:
http://www.diveintopython3.net/porting-code-to-python-3-with-2to3.html#dict
}}}
|-----------------------
{{{#!td 
Other dictionary iterators
}}}
{{{#!td
{{{
a_dictionary.iterkeys()
a_dictionary.iteritems()
a_dictionary.itervalues()
}}}
}}}
{{{#!td
{{{
six.iterkeys(a_dictionary)
six.iteritems(a_dictionary)
six.itervalues(a_dictionary)
}}}
}}}
{{{#!td 
Use function from six library
}}}
|-----------------------
{{{#!td 
Map list
}}}
{{{#!td
{{{
map(x, y)
}}}
Returns list
}}}
{{{#!td
{{{
map(x,y)
}}}
Returns iterator
}}}
{{{#!td
{{{
list(map(x,y))
}}}
Use list to wrap around map
}}}

=== Other recommendations:
Use .format specifier for the strings and parameters. 
For example instead of using:
{{{
'%s %s' % ('one', 'two')
}}}
Use:
{{{
'{} {}'.format('one', 'two')
}}}
.format is compatible with both Python 2 and Python 3.

Read more at:
https://pyformat.info/


== wxPython GUI
There are a lot of changes found in wxPython Phoenix version. It is recommended to follow the MIgration guide (https://docs.wxpython.org/MigrationGuide.html) to properly migrate from the Classic version of wxPython. To support both the versions. The wrap.py includes a lot of new classes that work as a wrapper to accommodate both the versions of wxPython and Python itself.

All the changes for Classic vs Phoenix can be found here:
https://wxpython.org/Phoenix/docs/html/classic_vs_phoenix.html

We have created a wrap.py class that contains overloaded classes for wx classes to support both versions.
Example:
{{{
from gui_core.wrap TextCtrl, StaticText
}}}
Deprecated warnings can be removed by appropriately using the wx classes. Refer to the changes in both versions and see if the wrapper class is already created for the wx class; if not, create a new class in a similar manner as other wrapper classes in the wrap.py file.

cmp function is not available in Python3, it has been custom created and included in gui/wxpython/core/utils.py file. Be sure to include it where cmp() is used.

== How to test ==

This section explains how to compile GRASS GIS ([source:grass/trunk/ trunk]) with Python3 support and how to run it.
=== Linux ===

We can rather easily create an isolated environment with Python3 using virtualenv which lets you test GRASS GIS with Python3 support while not cluttering your standard system.

Installation of Python-3 virtualenv:
{{{
# on Fedora
sudo dnf install python3-virtualenv
# on Debian/Ubuntu
sudo apt install python3-virtualenv
}}}

Preparation of the virtual Python3 environment. There are two options, build wx package on your own (it takes time like 30+min and some GB to build!) or simply use system-wide packages:

{{{
# USE DISTRO PROVIDED PACKAGES
# 3a) on Fedora
sudo dnf install python3-gdal python3-six python3-numpy python3-wxpython4
# 3a) on Debian/Ubuntu system
sudo apt install python3-six python3-numpy python3-wxgtk4.0

# OR BUILD WX PACKAGE DEPENDENCIES
# 3b) on Fedora (not recommended)
# sudo dnf install python3-six gtk3-devel gstreamer-devel gstreamer-plugins-base-devel python3-sip python3-sip-devel webkit2gtk3-devel

# 3b) on Debian/Ubuntu
sudo apt install libgtk-3-dev libgstreamer-plugins-base1.0-dev python3-six python3-numpy python3-wxgtk4.0 libglu1-mesa-dev libxmu-dev build-essential libgtk-3-dev python3-opengl libwxgtk-webview3.0-gtk3-0v5 libwxgtk-webview3.0-gtk3-dev python-wxgtk-webview3.0 python3-wxgtk-webview4.0

}}}

Create the virtual environment:

{{{
# FOR CURRENT USER ONLY
cd $HOME
# 4a) on Fedora
virtualenv -p python3 virtenv_grasspy3
# 4a) on Debian/Ubuntu
virtualenv -p python3 virtenv_grasspy3

# OR AS SYSTEM-SIDE VIRTUALENV
cd $HOME
# 4b) on Fedora
virtualenv -p python3 virtenv_grasspy3 --system-site-packages
# 4b) on Debian/Ubuntu
virtualenv -p python3 virtenv_grasspy3 --system-site-packages
}}}

Activating the virtual Python3 environment for testing:

{{{
# activate it (this will change the terminal prompt so that you know where you are...)
source $HOME/virtenv_grasspy3/bin/activate
}}}

and build within this environment 'wx' package (NOTE: only needed if 3b) above was used!)
{{{
# run within virtualenv
# install required Python3 packages for GRASS GIS (it takes time like 30min and some GB to build! not recommended)
pip install six
pip install wxpython
pip install numpy
pip install requests

# if desired, also GDAL (not recommended)
pip install --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL==`gdal-config --version`
}}}

We are now settled with the dependencies.

Download [[wiki:DownloadSource|GRASS GIS 7.7+ source code]] and compile it in the virtual environment with Python3:
{{{
# cd where/the/source/is/
configure...
make
make install
}}}

Test GRASS GIS with Python3:
{{{
# just run it in the virtualenv session
grass77 --gui
}}}

In order to deactivate the virtualenv environment, run
{{{
deactivate
}}}

=== Windows ===

To test a **winGRASS compilation** with **python3 support**, install the dependencies according the [https://trac.osgeo.org/grass/wiki/CompileOnWindows#InstalltheMSYS2directorystructure windows native compilation guidelines].

in MSYS2/mingw environment

{{{
cd /usr/src
virtualenv --system-site-packages --no-pip --no-wheel --no-setuptools grasspy3

Using base prefix 'C:/msys64/mingw64'
New python executable in C:/msys64/usr/src/grasspy3/Scripts/python3.exe
Also creating executable in C:/msys64/usr/src/grasspy3/Scripts/python.exe
}}}

change into the grasspy3 directory and do a svn checkout

{{{
cd grasspy3
svn checkout https://svn.osgeo.org/grass/grass/trunk grass_trunk
}}}

activate the virtual environment

{{{
$ source Scripts/activate
(grasspy3)
}}}

change into the grass_trunk directory and start the compilation

{{{
cd grass_trunk
# for daily builds on 64bit
PACKAGE_POSTFIX=-daily OSGEO4W_POSTFIX=64 ./mswindows/osgeo4w/package.sh
}}}

=== MacOS X ===

To test a **mac OSX GRASS compilation** with **python3 support**, do...

TODO (perhaps see http://grassmac.wikidot.com/downloads)
== Troubleshooting ==

Problem:

{{{
     File "/<grassbin_path>/etc/python/grass/lib/vector.py",
    line 5860
        PORT_LONG_MAX = 2147483647L
                                  ^
    SyntaxError: invalid syntax
}}}

Solution: make sure to compile GRASS GIS with Python 3 (the error occurs when GRASS GIS was compiled with Python2 and then opened with Python 3).

== References
http://sebastianraschka.com/Articles/2014_python_2_3_key_diff.html\\
http://osgeo-org.1560.x6.nabble.com/Python-3-porting-and-unicode-td5344215.html\\
http://python3porting.com/toc.html\\
https://docs.python.org/3.7/howto/pyporting.html\\
http://python-future.org/index.html\\
http://python-future.org/compatible_idioms.html\\
http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/\\
https://pypi.python.org/pypi/six\\
https://pypi.python.org/pypi/autopep8\\
https://docs.python.org/3.0/library/urllib.request.html#urllib.request.ProxyHandler\\
http://www.diveintopython3.net/porting-code-to-python-3-with-2to3.html#urllib\\
http://www.diveintopython3.net/porting-code-to-python-3-with-2to3.html#dict\\
https://pyformat.info/\\
https://docs.wxpython.org/MigrationGuide.html\\
https://wxpython.org/Phoenix/docs/html/classic_vs_phoenix.html