= Full support of Python 3 in GRASS GIS =
|| Title: || '''Full support of Python 3 in GRASS GIS''' ||
||Student Name: || Sanjeet Bhatti ||
||Organization: || [http://www.osgeo.org OSGeo - Open Source Geospatial Foundation] ||
|| Mentor Name: || Anna Petrášová and Luca Delucchi ||
|| GSoC proposal: ||[https://docs.google.com/document/d/18kD6I7adTKmtXpGOKr4A8hf1pBcf0LNU7jiiRoi9wJ8 View proposal] ||
|| Link to code: || [https://github.com/sanjeetbhatti/FullSupportPython3 Code Repository] ||
== Abstract ==
The current GRASS GIS version has partial support for Python 3. This project aims to update core GRASS components for Python 3 and update all the python modules, libraries and components like script package, python/ctypes, pygrass, temporal, wxGUI, etc. At the end of this project, GRASS GIS will be compatible with both the versions of python, i.e Python 2.7 and Python 3 with its GUI having support for wxPython 4.
== Timeline ==
{{{#!th style="background: #ddd" rowspan=2
'''Time Period'''
}}}
{{{#!th style="background: #ddd" colspan=2
'''Milestones'''
}}}
|-----------------------
{{{#!th
Tasks
}}}
{{{#!th
Status
}}}
|-----------------------
{{{#!td
April 23 - May 14 \\
''Community Bonding''
}}}
{{{#!td
- Introduction to Community
- Create Wiki page
- Set up code repository
- Add links to community wiki pages
- Set up development environment
- Discussion with mentors about the libraries tests for the project
- Get familiar with the testing framework
- Update current tests of the python script library, adjust them to work with Python3
- Get familiar with the overall working of GRASS GIS and understand Python dependencies in the system
- Read documentation
- Practice fixing bugs
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td
May 14 - May 20 \\
''Week 1''
}}}
{{{#!td
- Dealing with Unicode & binary data (encoding and decoding).
- Setting up the foundation for porting the main libraries and components
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td
May 21 - May 27 \\
''Week 2''
}}}
{{{#!td
Porting script package module and its dependencies
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td
May 28 - June 3 \\
''Week 3''
}}}
{{{#!td
- Porting other scripts and small libraries
- Write tests to check the consistency of libs subsequent to the changes
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td
June 4 - June 10 \\
''Week 4''
}}}
{{{#!td
- Finish pending work for this phase, if any.
- Doing some research on ctypesgen and pygrass
- Preparing and testing the changes for the evaluation
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td style="background: #ddd"
July 11 - July 15 \\
''Evaluation 1''
}}}
{{{#!td style="background: #ddd"
- Evaluation of work done in the previous phase
- Next phase discussions
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td
June 11 - June 17 \\
''Week 5''
}}}
{{{#!td
- Incorporate any changes, as necessary, from the discussions and feedback
- Work on Porting ctypes module
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td
June 18 - June 24 \\
''Week 6''
}}}
{{{#!td
Porting pygrass module and its dependencies
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td
June 25 - July 1 \\
''Week 7''
}}}
{{{#!td
- Porting temporal module and its dependencies
- Porting remaining libs and their dependencies
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td
July 2 - July 8 \\
''Week 8''
}}}
{{{#!td
- Finish pending work for this phase, if any.
- Preparing and testing the changes for the evaluation
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td style="background: #ddd"
July 9 - July 13 \\
''Evaluation 2''
}}}
{{{#!td style="background: #ddd"
- Evaluation of work done in the previous phase
- Next phase discussions
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td
July 9 - July 29 \\
''Week 9-11''
}}}
{{{#!td
- Based on the feedback, make necessary changes
- Writing additional classes in wrap.py for updating wxGUI to be compatible with both wxPython 3 and 4.
- Finish porting to Python 3
- Fixing all deprecated messages
- Testing the changes.
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td
July 30 - August 5 \\
''Week 12''
}}}
{{{#!td
- Preparing and testing all the changes
- Writing documentation for the final evaluation
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
|-----------------------
{{{#!td style="background: #ddd"
August 6 - August 14 \\
''Evaluation 3 (Final)''
}}}
{{{#!td style="background: #ddd"
- Final code submission
- Evaluation of final submission
}}}
{{{#!td
{{{#!html
✔
}}}
}}}
== Community Bonding Period Report ==
During the Community Bonding Period I performed the following activities:
'''Introduction to Community'''\\
I wrote an introductory email to the SoC mailing list and grass-dev mailing list and gave a description about my project and shared my wiki page, public repository, and project proposal links.
'''Create Wiki page'''\\
I requested the write access to setup a project wiki page. Then, I created my wiki page detailing the project description, containing link to my GSoC application, a detailed timeline, link to my public repository where all my commits and progress can be followed.
'''Set up code repository'''\\
I have setup my code repository on github for tracking the applied changesets.
'''Set up development environment'''\\
I setup my development environment (Ubuntu 18.04) for two Python versions (Python 2.7.x and Python 3.6.x) for which I am using virtual environments. This was done because I need to make sure that when I update any code it should be compatible with both the versions.
'''Add links to community wiki pages'''\\
I added my project links to the OSGeo's Google Summer of Code 2018 Accepted students wiki page and trac wiki page: https://wiki.osgeo.org/wiki/Google_Summer_of_Code_2018_Accepted and https://trac.osgeo.org/grass/wiki/GSoC.
'''Discussion with mentors about the libraries tests for the project'''\\
I got in touch with my mentors to discuss the right approach to test the code and submit files. We discussed about the python libraries and their testsuites, starting with Python script library.
'''Get familiar with the testing framework'''\\
I followed the wiki page to get familiar with the testing framework(https://grasswiki.osgeo.org/wiki/GRASS_and_Python). I also followed the grass Python Scripting tutorials from this link:
https://github.com/wenzeslaus/python-grass-addon
'''Update current tests of the python script library, adjust them to work with Python3'''\\
I updated the code in the python script library to make it work on Python3 and submitted the patch files for those. I used the standard North Carolina dataset that uses the basic version of the dataset.
(https://grass.osgeo.org/download/sample-data/)
I have looked through the summary reports for the current tests of GRASS which is going to be useful when I will update tests for the libraries in the system. (http://fatra.cnr.ncsu.edu/grassgistests/summary_report/)
'''Get familiar with the overall working of GRASS GIS and understand Python dependencies in the system'''\\
I went through the GRASS wiki pages to get familiar with the working of the system (https://grass.osgeo.org/grass75/manuals/libpython/) and looked through the source code. Also, I found the code dependencies that I will be working with throughout the coding period.
'''Read documentation'''\\
I read some documentation related to GSoC guidelines about working throughout the GSoC period.
I also read GRASS GIS Python library documentation: (https://grass.osgeo.org/grass75/manuals/libpython/)
'''Practice fixing bugs'''\\
I wrote a bug fix related to wxPython(https://trac.osgeo.org/grass/ticket/3510) during Pre-bonding period and submitted the patch file.
== Weekly reports ==
=== Week 1 ===
'''What did I complete this week?'''\\
- Updated the script for redundant lines in the patch submitted for utils.py
- Tested the patches on Python versions 2.7.x, 3.5.x and 3.6.x
- Read and understood the main differences about Strings, Unicode strings and Byte strings in Python 2 and 3 versions. Wrote some brief differences for quick reference(https://docs.google.com/document/d/18Tvl0vpTSwOfO4xGKXpAo-h6o6WAAZmHX8neYIsHTtA)
- Read python's package futurize's documentation for Py2 to Py3 conversion
- Updated source code in python lib files (uncommitted, still testing)
'''What am I going to achieve for the next week?'''\\
- Finish porting 'script' library to support Python3
- Test the changes for both the versions
- Implement any tests required for testing
'''Is there any blocking issue?'''\\
Not at the moment.
=== Week 2 ===
'''What did I complete this week?'''\\
- Updated the encode and decode function in utils.py to accept the encoding type
- Ported functions from 'Script' library to deal with unicode/bytes (https://gist.github.com/sanjeetbhatti/6714dff40b8a0a6d30df25ce9e4faa20)
- Tested the changes by running tests (testsuites)
- Tested against the python-grass-addon scripts available from here (https://github.com/wenzeslaus/python-grass-addon)
'''What am I going to achieve for the next week?'''\\
- Write more tests for script library to test the changes
- Look into the issue of running scripts for Python 3.5 and Python 3.6 (some tests fail on either of the versions)
- Port other small libraries and scripts
'''Is there any blocking issue?'''\\
Not at the moment.
=== Week 3 ===
'''What did I complete this week?'''\\
- Wrote new tests for 'Script' library (https://github.com/sanjeetbhatti/FullSupportPython3/tree/master/grass_trunk/lib-python-script)
- Made changes to the existing source files according to the failing tests (https://docs.google.com/spreadsheets/d/1eAIJtegNWSbBscFChhQzMTl0vmbGPjmrW_013hSw9bI/#gid=0)
- Updated scripts in 'trunk/Scripts' (https://github.com/sanjeetbhatti/FullSupportPython3/tree/master/grass_trunk/scripts)
- Created a spreadsheet to keep track of the status for all trunk/scripts (https://docs.google.com/spreadsheets/d/1eAIJtegNWSbBscFChhQzMTl0vmbGPjmrW_013hSw9bI/#gid=278198865)
- Tested the changes with Python versions 2.7, 3.5 and 3.6
- Cleaned the patches according to the pep8 guidelines and pushed those patches to the repository
'''What am I going to achieve for the next week?'''\\
- After discussion and feedback from the mentors, finish pending work
- Look into ctypes and pygrass libraries
- Discuss with mentors about possible solutions for ctypes
- Test all the changes done for the evaluation period
'''Is there any blocking issue?'''\\
Not at the moment.
=== Week 4 ===
'''What did I complete this week?'''\\
- Wrote new tests for 'trunk/scripts' according to the priority (https://github.com/sanjeetbhatti/FullSupportPython3/tree/master/grass_trunk/scripts)
- Updated some files in 'gunittest' library pertaining to the 'trunk/scripts' (https://github.com/sanjeetbhatti/FullSupportPython3/tree/master/grass_trunk/lib-python-gunittest)
- Looked into different forks of ctypes library
- Got the fork with latest changes, tried compiling with Py3
- Tested the changes done, updated the repository (https://github.com/sanjeetbhatti/FullSupportPython3/)
'''What am I going to achieve for the next week?'''\\
- Submit the first evaluation
- Work on any changes necessary after discussion and feedback from the mentors
- Start working on porting the ctypes library
'''Is there any blocking issue?'''\\
Not at the moment.
=== Week 5 ===
'''What did I complete this week?'''\\
- Updated tests for 'trunk/scripts' as per the feedback (https://github.com/sanjeetbhatti/FullSupportPython3/tree/master/grass_trunk/scripts)
- Made changes in the 'ctypes' library to make it compile with Py3 (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/lib-python-ctypes/patch_ctypes.diff)
- Added another patch for 'ctypes' to include the changes from the latest fork of ctypes (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/lib-python-ctypes/patch_ctypes_changes_from_fork.diff)
- Tested the changes with Python versions 2.7, 3.5 and 3.6
'''What am I going to achieve for the next week?'''\\
- Work on 'pygrass' module
- Compare changes of ctypes compiled version among all 3 versions with and without changes
- Write new tests for some of the remaining scripts in 'trunk/scripts'
'''Is there any blocking issue?'''\\
Not at the moment.
=== Week 6 ===
'''What did I complete this week?'''\\
- Fixed few issues and updated ctypes (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/lib-python-ctypes/patch_ctypes_changes_from_fork.diff)
- Worked on 'Pygrass' library to make it work with the updated ctypes (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/lib-python-pygrass/patch_pygrass.diff)
- Updated tests in 'pygrass' library
'''What am I going to achieve for the next week?'''\\
- Continue working on 'pygrass' module
- Work on 'temporal' library
- Fix pending issues in pygrass
'''Is there any blocking issue?'''\\
Not at the moment.
=== Week 7 ===
'''What did I complete this week?'''\\
- Updated ctypes to deal with bytes(https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/lib-python-ctypes/patch_ctypes_changes_from_fork.diff)
- Worked on 'temporal' library(https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/lib-python-temporal/patch_temporal.diff), a lot of errors were encountered
- Looked into issues for 'pygrass' testsuites which were failing too
- Created a spreadsheet(https://docs.google.com/spreadsheets/d/1eAIJtegNWSbBscFChhQzMTl0vmbGPjmrW_013hSw9bI/edit#gid=408919374) detailing errors and information regarding the temporal testsuites
- Also, updated the pygrass testsuite details on spreadsheet(https://docs.google.com/spreadsheets/d/1eAIJtegNWSbBscFChhQzMTl0vmbGPjmrW_013hSw9bI/edit#gid=587128881)
'''What am I going to achieve for the next week?'''\\
- Start exploring GUI library, starting with mapdisp and imgr
- Continue working on fixing ctypes module
- Continue working on 'temporal' library
'''Is there any blocking issue?'''\\
Ctypes are not fully compatible with the Python3 even if they are compiling without any issues. Discussed with mentors about the next steps and how to deal with ctypes.
Will discuss more about the upcoming milestones and how ctypes handling is to be done so as not to interfere too much with the pending work.
=== Week 8 ===
'''What did I complete this week?'''\\
- Updated 'pygrass' library (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/lib-python-pygrass/patch_pygrass.diff) fixed failing errors
- Updated 'pydispatch' to deal with dict iterator (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/lib-python-pydispatch/patch_pydispatch.diff)
- Fixed 'ctypes' patch (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/lib-python-ctypes/patch_ctypes_changes_from_fork.diff), was failing to apply
- Fixed import issues in GUI library (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/gui/patch_gui.diff)
- Fixed gconsole, goutput and gcmd errors to run the wxpython4 on Py3
- Updated all dict iterators (keys, values, items) in GUI library
'''What am I going to achieve for the next week?'''\\
- Fix deprecated code in GUI library
- Keep working on ctypes-pygrass-temporal libraries
- Write tests for trunk/scripts
'''Is there any blocking issue?'''\\
Not at the moment.
=== Week 9 ===
'''What did I complete this week?'''\\
- Fixed 'temporal' library's connection issue (t.connect) (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/lib-python-temporal/patch_temporal.diff)
- Tested the code with temporal t.connect tests
- Fixed code for relative paths
- Fixed g.gui.animation part of gui (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/gui/patch_gui.diff)
- Wrote two new tests for trunk/scripts (testing pending)
'''What am I going to achieve for the next week?'''\\
- Fix deprecated code in GUI library
- Fix other gui command files
- Keep working on temporal library
- Test all the reamining files in temporal testsuite
'''Is there any blocking issue?'''\\
Not at the moment.
=== Week 10 ===
'''What did I complete this week?'''\\
- Replaced deprecated code warnings for set-tooltip (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/gui/patch_gui.diff)
- Fixed psmap issue for map
- Created new wrapper classes for widgets
- Fixed animation and mapswipe events binding deprecation warnings
- Fixed 'cmp' function calls, created custom function to deal with all instances
'''What am I going to achieve for the next week?'''\\
- Replace remaining code deprecated warnings in GUI library
- Fix remaining gui command files
- Work on forms.py to fix string errors
- Replace all instances of old wx.* code with wrapper classes where suitable
'''Is there any blocking issue?'''\\
Not at the moment.
=== Week 11 ===
'''What did I complete this week?'''\\
- Replaced instances of old wx.* code with wrapper classes where suitable (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/gui/patch_gui.diff)
- Fixed forms.py string errors
- Added cmp function in utils
- Fixed remaining gui command files
- Decoded strings where necessary in base.py
- Added encoding in gmcd before hashing
'''What am I going to achieve for the next week?'''\\
- Fix assertion errors
- Fix remaining warnings in GUI library
- Do testing for GUI
- Prepare for evaluation, prepare deliverables
'''Is there any blocking issue?'''\\
Not at the moment.
=== Week 12 ===
'''What did I complete this week?'''\\
- Fixed assertion errors (https://github.com/sanjeetbhatti/FullSupportPython3/blob/master/grass_trunk/gui/patch_gui.diff)
- Tested GUI and fixed deprecation warnings and errors
- Fixed toolboxs.py for bytes
- Fixed gmodeler, subprocess errors
- Fixed profile tool and map
- Fixed list errors
'''What am I going to achieve for the next week?'''\\
- Do more testing, and fix bugs and errors
- Prepare deliverables, update all patches with the latest revision
- Prepare guidelines for Py2/3 compatibility code
- Write Final Report
'''Is there any blocking issue?'''\\
Not at the moment.
=== Week 13 - Final Report ===
**Title:** Full support of Python 3 in GRASS GIS\\
**Community:** GRASS GIS - OSGeo
**Abstract:**\\
This project updates the GRASS GIS software to support Python 3, where earlier it only worked with the Python 2.7 version. All the libraries, modules and components of the system have been updated to include the latest changes that have been introduced in Python 3. One of the important updates was a way to deal with unicode and bytes strings that helped in making GRASS compatible with both the versions of Python by properly encoding and decoding the strings. Additionally, the ctypes library, which was only working with Python 2.7, has been upgraded to deal with Python 3 which further helped in modifying the temporal and pygrass libraries. The pydispatch and guinittest modules have also been altered to properly deal with unicodes while the trunk/scripts have been modified and tested by writing new testsuites for them. The GUI module of GRASS is built with wxPython Classic version, which now with the help of wrapper classes works well with the latest wxPython 4 - Phoenix version.
**The state of the art BEFORE the start of GSoC:**\\
Before the start of the project, GRASS GIS had partial support for Python 3. The ctypes library failed to compile with Python 3 while a lot of code in other modules was also incompatible with Python 3 syntax. The GUI module worked with wxPython 3 (Classic version) but not with the latest wxPython 4 (Phoenix). There were a lot of assertion errors and deprecated code warnings that were seen when GRASS was coupled with the wxPython 4.
**The state of the art AFTER GSoC:**\\
After the GSoC, GRASS can now be compiled with Python 3 version. The different libraries and components of GRASS have been updated and are thus compilable with Python 3.\\
The GUI module now has support for both the versions of wxPython, i.e. Classic (3 or older) and Phoenix (4).
**Challenges:**\\
The biggest challenge in updating GRASS was dealing with the ‘ctypes’ library. The official library has not been updated since some time now and the Python 3 support is not complete. It was a major hurdle to get the 'ctypes' working in GRASS to compile other modules/libraries that are dependent on this library. After making the desired changes in the library, the GRASS system is able to compile but there are some lingering issues that make it hard to fix other errors that persist in the temporal and pygrass libraries.
**Next Steps:**\\
The next step would be to discuss ‘ctypes’ issue in further detail with mentors and the development community. Even though some issues related to ctypes have been solved in this project, it still requires some significant efforts and time to make it fully compatible with all Python versions. Moreover, fixing ctypes will help in solving all the remaining issues in other GRASS modules that are closely dependent on this library.
**Permanent Links:**\\
Code Repository for GSoC:\\
https://github.com/sanjeetbhatti/FullSupportPython3
Weekly Reports:\\
https://trac.osgeo.org/grass/wiki/GSoC/2018/FullSupportPython3
Python 2 - 3 and wxPython compatibility guidelines for developers:\\
https://trac.osgeo.org/grass/wiki/Python3Support
Image for project representation:\\
https://raw.githubusercontent.com/sanjeetbhatti/FullSupportPython3/master/GSoC-final-report-project-pic.png