GSoC 2021: Improved Integration of GRASS GIS and Jupyter Notebooks

Title:	Improved Integration of GRASS GIS and Jupyter Notebooks
Student Name:	Caitlin Haedrich, ​North Carolina State University
Organization:	​OSGeo - Open Source Geospatial Foundation
Mentor Name:	Vaclav Petras, Helena Mitasova, Stephan Blumentrath
GitHub Fork:	​View Repo
GSoC proposal:	​View Proposal

​Try grass.jupyter in Binder here

Abstract

The previous integration of GRASS GIS and Jupyter Notebooks required a cumbersome environment variable setup after launching GRASS from within the notebook and only allowed for simple, non-interactive map displays. The grass.jupyter subpackage for addresses both these issues by introducing a new startup function, init(), and two display classes, GrassRenderer and InteractiveMap. GrassRenderer renders GRASS displays as PNG images using a more intuitive syntax. InteractiveMap displays rasters and vectors interactively with ​folium, a leaflet library for Python.

Goal

This project had three main objectives:

1. creating new initiation functions for the launch of GRASS GIS in Jupyter Notebooks (init())

2. creating functions for more intuitive map display (GrassRenderer())

3. introducing an interactive map display function (InteractiveMap())

Timeline

Time Period	Milestones
	Tasks	Status
May 17th - June 7th Community Bonding	Introduce myself in dev and SOC mailing lists Get in contact with mentors and discuss project Prepare the wiki page Set up the GitHub repository for project Set up developer environment	Ok Ok Ok Ok Ok
June 7th - June 11th Week 1	Finish Binder Set Up Write grass.jupyter initiation functions	Ok Ok
June 14th - June 18th Week 2	Write non-interactive display functions	Ok
June 21st - June 25 Week 3	Begin writing interactive/folium map functions Edit non-interactive display functions	Ok Ok
June 28 - July 2 Week 4	Continue writing interactive/folium map functions for vectors	Ok
July 5th - July 9th Week 5	Continue writing interactive vector functions Begin writing interactive raster functions	Ok Ok
July 12th - July 16th Week 6: Evaluations	Receive feedback and write evaluation Switch to temporary files and directories Continue working on folium functionality	Ok Ok Ok
July 19th - July 23rd Week 7	Improve documentation and error handling Draft test module for non-interactive rendering Continue raster functions for folium	Ok Ok Ok
July 26th - July 30th Week 8	Finish draft of raster methods with folium Improve test module	Ok Ok
August 2nd - August 6th Week 9	Improve raster handling in folium Add save as HTML option Improve docstrings, readability	Ok Ok Ok
August 9th - August 13th 'Week 10'	Finish InteractiveMap Example Notebooks	Ok Ok
August 16th- August 23rd Week 11: Final Evaluation and Code Submission	Submit code and final evaluation. Example Notebooks Final docstring improvements	Ok Ok Ok

Bonding period report

With the bonding period wrapped up, here is a brief update on what I've been working on so far.

1. What did I accomplish during the bonding period?

• I introduced myself in dev and SOC mailing lists (​1).
• I had a productive meeting with my mentors, Vashek Petras, Helena Mitasova and Stefan Blumentrath last Friday. We discussed the best development environment (see second to last bullet point) and setting up the main GRASS repository to run in Binder (see last bullet point).
• Prepared the wiki page.
• Set up the GitHub repository for project (​2).
• Set up developer environment. Since development will be easier on Linux, I set up a VirtualBox with Linux (Ubuntu) on my Windows machine. I compiled and installed GRASS and installed Jupyter Notebooks in Ubuntu.
• Opened my first PR (​3) which contains binder dependencies.

2. What do you plan on doing next week?

I've started working on setting up the GRASS repository to run in Binder (​3). This week I plan to continue working on that, set up branches for each of the PRs, and begin writing functions for the grass.jupyter library.

3. Are you blocked on anything?

No, I'm not currently blocked on anything.

Weekly reports

Week 1

With Week 1 wrapping up, here is an update on what I worked on this week and where I'm headed next week.

1) What did I get done this week?

• Finished binder setup for GRASS main repo. PR was approved. There's a new folder in the main GRASS repo that contains the necessary files to launch the repository in binder.
  
• Added Binder Button. PR under review. This button will go in the README.md and allow users to launch a Jupyter Lab of the repository in binder. Users can run GRASS (i.e. a compiled version of the main GRASS repo) in Jupyter Notebooks there.
  
• Created an ​example notebook that is linked from the Binder Button. The notebook was copied from ​here and demonstrates the existing Jupyter/GRASS integration.
  
• Wrote draft of GRASS initiation functions for Jupyter (​PR).
  
• Created Makefile and init.py file for grass.jupyter (​PR).
  
• Created a Jupyter Notebook where others can view and test the grass.jupyter functions (​PR).
  

2) What do I plan on doing next week? I plan to start working on display functions and familiarizing myself with Folium.

3) Am I blocked on anything? No, I'm not currently blocked on anything.

Week 2

Here is a brief update from Week 2. You can also check out how the new grass.jupyter module works so far in Binder at ​this link

1) What did I accomplish this week?

• Had a productive discussion with my mentors: Vaclav Petras, Helena Mitasova and Stefan Blumentrath.
• Edited and merged Binder button into README ​PR and GRASS session initiation functions for Jupyter ​PR.
• Wrote functions for displaying non-interactive maps in Jupyter and opened PR for discussion ​PR.
• Began researching and drafting interactive display options with folium. In particular, figuring out how to best pass GRASS data/layers to folium.

2) What do I plan on doing next week?

• Edit non-interactive display functions.
• Continue working on passing GRASS data/layers to folium.

3) Am I blocked on anything? No, I'm not blocked but I didn't get much done this week because I had a family event.

Any suggestions and feedback are welcome!

Week 3

Here is a brief update on what I accomplished during Week 3 of Google Summer of Code for the new GRASS library grass.jupyter. I have also started posting links to Binder Jupyter Notebooks in each PR if you would like to try any of the features and functions I've been working on.

1) What did I accomplish this week?

• Edited and discussed functions for displaying non-interactive maps in Jupyter ​PR.
• Drafted an interactive display function for rasters with folium ​PR.
• Made a few minor edits to the example_notebook.ipynb that is linked in the README ​PR.

2) What do I plan on doing next week?

• I have a meeting planned with my mentors.
• I will continue working on interactive functions for Jupyter. In particular, I'm working on passing GRASS rasters to folium.
• I'll begin writing functions for displaying GRASS vectors with folium.

3) Am I blocked on anything? I'm not blocked at the moment. But, I have been working on finding a good way to pass raster data between GRASS and Jupyter and I'm not sure my current method is particularly fast or robust [3]. Luckily, I have a meeting tomorrow with my mentors where we will discuss this.

Any suggestions and feedback are welcome!

Week 4

With week 4 wrapping up, here is a brief update of what I've been working on. Per usual, you can try proposed changes out in the Binder link in the description of each PR.

1) What did I accomplish during Week 1?

• I had a productive meeting with my mentors: Vaclav Petras, Helena Mitasova and Stefan Blumentrath. We discussed how environments work and the purpose of copying the environment each time an instance of GrassRenderer is called.
• I finished and merged a class called GrassRenderer for non-interactive map displays ​PR.
• I continued to work on interactive mapping with folium. In particular, I changed how data is passed to folium so that it is first reprojected to WGS84 (required by folium/leaflet) in a new location/mapset ​PR.

2) What do I plan on doing next week?

• Continue to work on interactive functions with folium.

3) Am I blocked on anything?

• No, I'm not blocked on anything at the moment.

Any suggestions and feedback welcome!

Week 5

Here is a weekly update on my project, Improved Integration of GRASS GIS and Jupyter Notebooks.

1) What did I accomplish this week?

• This week, I spent most of my time improving and editing a new class called InteractiveMap to display vectors interactively with folium. You can test the current functionality in ​Binder.
• I had a productive meeting with my mentor, Vaclav Petras on Thursday. We discussed several things that I was a little stuck on and had questions about.
• Updated grass.jupyter demonstration ​Notebook.
• Resolved merge conflicts with master - now, the interactive and non-interactive classes are functional together ​PR.
• Found unexpected behavior in the legend display. Calling folium.map.LayerControl() adds a legend to the map but also seems to prevent further modification of the map.

2) What do I plan to do next week?

• Meet with my mentors tomorrow.
• GSoC Evaluations.
• Finish and merge interactive vector methods.
• Begin interactive maps for rasters.

3) Am I blocked on anything?

• No, I'm not currently blocked on anything.

Feedback, comments, questions and ideas welcome!

Week 6

Here's a weekly update on my GSoC project, Improved Integration of GRASS GIS and Jupyter Notebooks.

1) What did I accomplish this week?

• This week, I had two meetings with my mentor, Vaclav Petras. We've been discussing how to improve my workflow, best practices and conventions for GRASS and Python and questions that come up as I'm working.
• Instituted the use of temporary PNG files for displaying static/non-interactive maps ​PR and temporary directories for passing vectors between GRASS and folium ​link (PR coming soon).
• Continued to edit the InteractiveMap class and mapping GRASS vectors in folium ​PR
• Created a shortcut for calling GRASS display modules using getattr ​PR
• Completed evaluation and received feedback.

2) What do I plan to do next week?

• I have another meeting planning with Vaclav tomorrow.
• Finish and merge existing PRs
• Continue writing test module for Non-Interactive Vectors
• Support Raster display in folium map
• Clean-up function

3) Am I blocked on anything?

• No, I'm not currently blocked on anything.

Feedback, comments, questions and ideas welcome!

Week 7

Here's a quick update on what I worked on this past week.

1) What did I accomplish this week?

• This week, I had another two meetings with my mentor, Vaclav Petras. We've been discussing best practices and conventions for GRASS and Python and questions that come up as I'm working.
• Merged 'add_vector' method for InteractiveMaps in folium [2].
• Improved error handling and documentation on the shortcut for calling GRASS display modules for "GrassRenderer" [3].
• Wrote a test module for the "GrassRenderer" class - a class which displays GRASS maps in Jupyter Notebooks as PNG images. I haven't opened a PR since there are a couple things I need to add after some other PRs are merged.
• Continued to work on overlaying rasters in folium - this is taking me a while because the raster and raster bounds have to be reprojected since folium only takes WGS84 and Pseudo-Mercator projections.

2) What do I plan to do next week?

• I have another meeting planning with Vaclav on Monday.
• Finish and merge existing PRs
• Continue writing test module for GrassRenderer

3) Am I blocked on anything?

• No, I'm not currently blocked on anything.

Feedback, comments, questions and ideas welcome!

Week 8

Here's a quick update on what I worked on this past week. I came down with a stomach bug this week and didn't get as much as I hoped done but I did wrap up some things I'd been working on for a while.

1) What did I accomplish this week?

• I had a brief meeting with my mentor, Vaclav Petras. We discussed some of the existing PR's I had open and goals for the rest of the summer.
• Got rasters to display with folium (finally)! You can check out the functionality in Binder ​here and see the PR ​here. It takes a while to reproject the raster (folium only excepts rasters in Pseudo-mercator). As Stefan recommended, I'm hoping to write a "simple CRS" option that won't require reproject and will be faster. But that's for next week...
• Merged shortcuts and temporary files for GrassRenderer (​PR and ​PR).
• Continued working on test script for GrassRenderer, the non-interactive display class ​PR.

2) What do I plan to do next week?

• I have a meeting with my mentors planned for tomorrow
• Add simple CRS option for raster display in folium
• Edit and finish exiting PRs
• Add option to save/export folium map as HTML

3) Am I blocked on anything?

• No, I'm not currently blocked on anything.

Feedback, comments, questions and ideas welcome!

Week 9

Happy last week of coding! Here's an update of my GSoC project, "Improved Integration of GRASS GIS and Jupyter Notebooks."

1) What did I accomplish this week?

• I had a meeting with my mentors, Vaclav Petras and Helena Mitasova. We discussed some of the existing PR's and goals for the rest of the summer.
• Rasters reproject and display much faster now! You can test it out in ​Binder. As Vaclav pointed out, it was oversampling the raster when it reprojected - adding a method to estimate the resolution in target location fixed this.
• Re-arranged code for InteractiveMap and improved documentation ​PR
• Added save as HTML method for InteractiveMap ​PR

2) What do I plan to do next week?

• I have a meeting with my mentors planned for today
• Add simple CRS option for raster display in folium
• Finish test script ​PR
• Improve/finish GRASS-Jupyter Demonstration notebook

3) Am I blocked on anything? No, I'm not currently blocked on anything.

Feedback, comments, questions and ideas welcome!

Week 10

Last week of coding! Here's an update on how the project is progressing

1) What did I accomplish this week?

• I had a meeting with my mentors, Vaclav Petras and Stephan Blumentrath. We discussed goals for the rest of the summer and decided to leave the simple CRS for future work.
• Merged raster support in folium to main! It was a pretty substantial PR in the end ​PR
• Began improving example notebooks and adding additional tutorials.

2) What do I plan to do next week?

• Finalize docstrings
• Finish example notebooks
• Evaluations and final submission

3) Am I blocked on anything? No, I'm not currently blocked on anything.

Feedback, comments, questions and ideas welcome - especially for Future Work!

Final Report

Title: Improved Integration of Grass GIS and Jupyter Notebooks
Community: GRASS GIS - OSGeo

Abstract:
This project introduces a new subpackage for GRASS GIS, grass.jupyter that improves the integration of GRASS GIS with Jupyter Notebooks. Previously, using GRASS in Jupyter Notebooks required a cumbersome environment variable setup after launching GRASS from within the notebook and only allowed for simple, non-interactive map displays. The grass.jupyter subpackage addresses both these issues by introducing a new startup function, init(), and two display classes, GrassRenderer and InteractiveMap. GrassRenderer renders GRASS displays as PNG images using an intuitive syntax. InteractiveMap displays rasters and vectors interactively with ​folium, a leaflet library for Python.

The state of integration BEFORE the start of GSoC:
The previous integration of GRASS GIS and Jupyter Notebooks required a cumbersome environment variable setup after launching GRASS from within the notebook. There is an external python library grass_session that can be installed to shorten this launch substantially but, as an external library, it is not included in a typical GRASS install. Additionally, the previous integration allowed for maps to rendered as PNG images uses a unintuitive sequence of calling d.erase, then modules from the display family and finally rendering the image with IPython.display Image(). For an example this workflow, see ​basic_example.ipynb.

The state of integration AFTER GSoC:
With the help of my mentors, I introduced a new package grass.jupyter that contains a new init() function to shorten the launch and two display-related classes, GrassRenderer and InteractiveMap. GrassRenderer wraps the previous approach, rendering PNG images but with a more intuitive syntax. Users create an instance of GrassRenderer then add elements to the rendering with GRASS display modules. The GRASS modules are called by using the name of module as a class method and replacing "." with "_" in the name. For example, to add a raster, one would call GrassRenderer.d_rast(map="raster_name").

InteractiveMap, the other display-related class in grass.jupyter, allows users to view GRASS vectors and rasters in ​folium, a leaflet library for Python. After creating an instance of InteractiveMap, users can add vectors and rasters with add_vector() and add_raster(). Users can also add a layer control element with add_layer_control(). Folium only supports EPSG 4326 (for raster overlay) and EPSG 3857 (for vectors and coordinates) so InteractiveMap creates a temporary location, reprojects data then saves it to a temporary directory where it is imported by folium.

Conclusion:
In this project, I was successful in accomplishing the three goals stated at the beginning (thanks to my mentors!):

1. creating new initiation functions for the launch of GRASS GIS in Jupyter Notebooks (init())

2. creating functions for more intuitive map display (GrassRenderer())

3. introducing an interactive map display function (InteractiveMap())

The work I accomplished this summer paves the way for many future improvements, listed below in the Future Work section. In addition to some smaller modifications to init() and 'GrassRenderer', there remains a lot of work to be done on InteractiveMap or with other folium-GRASS class to fully access folium. I am grateful for the support I've received this summer and for the opportunity to contribute to GRASS GIS. I'm looking forward to continuing to improve grass.jupyter.

Future Work:

• Height and width defaults in GrassRenderer should be derived from computational region
• init() should fail and report an appropriate error if a mapset that doesn't exist is provided
• Add folium Tooltip method to InteractiveMap, allowing users to access vector attribute data by clicking on feature
• Add simpleCRS option to add_raster method in InteractiveMap
• Add option to display rasters as vectors (pixels -> polygon) in InteractiveMap
• Clip vectors to computational region in InteractiveMap (currently the whole vector dataset is displayed)
• Add more interactive functions: timeline slider for temporal datasets, etc.
• InteractiveMap doesn't allow users to fully access folium. In the future, a new interface that allows users direct access to folium should be added. For example, it could look like: gj.Raster("elevation").add_to(folium_map))

Permanent Links:

Pull requests related to grass.jupyter:

Title	Pull Request
Added Binder setup files	​https://github.com/OSGeo/grass/pull/1603
Binder Button	​https://github.com/OSGeo/grass/pull/1628
Session Initiation functions for Jupyter Notebooks	​https://github.com/OSGeo/grass/pull/1629
Added additional GRASS info to example_notebook	​https://github.com/OSGeo/grass/pull/1686
Jupyter: Non-interactive display	​https://github.com/OSGeo/grass/pull/1668
Interactive vector maps for Jupyter Notebooks	​https://github.com/OSGeo/grass/pull/1710
__getattr__ shortcut for calling GRASS display modules	​https://github.com/OSGeo/grass/pull/1723
Add temporary files for Non-Interactive Display	​https://github.com/OSGeo/grass/pull/1727
Raster Support for Interactive Jupyter maps with folium	​https://github.com/OSGeo/grass/pull/1769
Test Module for Non-interactive display in Jupyter Notebooks	​https://github.com/OSGeo/grass/pull/1739
Docstring improvements	​https://github.com/OSGeo/grass/pull/1800
Example notebooks	​https://github.com/OSGeo/grass/pull/1787

OSGeo Wiki Page:
​https://trac.osgeo.org/grass/wiki/GSoC/2021/JupyterAndGRASS

OSGeo-GRASS !github project page:
​https://github.com/OSGeo/grass/projects/7

Github Fork and grass.jupyter directory:
​https://github.com/chaedri/grass/tree/master/python/grass/jupyter

Binder Examples:
​grass_jupyter.ipynb ​basic_example_grass_jupyter.ipynb ​hydrology.ipynb ​viewshed_analysis.ipynb ​solar_potential.ipynb