{{{
#!html
GSoC 2017 | GRASS GIS Locations Created from Public Data
Abstract
This project will focus on creating scripts that convert widely-used open data sets into standardized
formats for a given location or projection. The scripts, modules, project documentation, and data sets
generated from the project would be made available as a service and development framework to
GRASS GIS users.
Goal
Novice GRASS users often have difficulty with the complexities of re-formatting data, solving map
projection issues, and working with centralized data organization. Many existing solutions require
users to create their own custom scripts geared towards a specific use-case, which can be difficult
for some users to adapt to other data sources or coordinate projection systems.
With the scripts created for this project, users will be able to download multiple data sets and appropriately
format the data according to a user-defined location. Automated data-formatting will allow users to
more efficiently utilize information and create automatically formatted data sets. Furthermore, the underlying
framework generated by my project will allow users to develop their own scripts for a given service and/or location.
Timeline
| Community Bonding Period | Status
|
| MAY 4 - MAY 26 | Complete preliminary GSoC, OSGeo, mentor, and proposal requirements |
|
| Phase 01
|
| MAY 30 - JUNE 2 | Explore existing efforts, primary community needs, and potential project difficulties |
|
| JUNE 5 - JUNE 9 | Identify data sources and formatting conventions |
|
| JUNE 12 - 16 | Psuedocode scripts |
|
| JUNE 19 - 23 | Draft python scripts |
|
| JUNE 26 - 30 | Phase 1 formatting scripts completed |
|
| JUNE 30 | PHASE 01 EVALUATION DEADLINE |
|
| Phase 02
|
| JULY 3 - 7 | Integrate existing APIs into scripts |
|
| JULY 10 - 14 | Unavailable |
|
| JULY 17 - 21 | Develop data storage, web services, and cross-platform compatibility |
|
| JULY 24 - 28 | Create use-case documentation and GUI mock-up (wxPython) |
|
| JULY 28 | PHASE 02 EVALUATION DEADLINE |
|
| Phase 03
|
| JULY 31 - AUGUST 4 | Develop formatting scripts that format multiple data-types and coordinate systems |
|
| AUGUST 7 - 11 | Further develop data storage, web services, and cross-platform compatibility |
|
| AUGUST 14 - 18 | Finalize each of the data formatting pipelines, GUI, services, and processes |
|
| AUGUST 21 - 25 | Finalize each of the data formatting pipelines, GUI, services, and processes |
|
| AUGUST 28 - 29 | FINAL WEEK
|
| Create documentation and submit completed work |
|
| SEPTEMBER 5 | FINAL EVALUATION SUBMITTED |
|
Requirements
GRASS 7.3
Development
Weekly reports
| COMMUNITY BONDING: MAY 04 - MAY 29
|
| ACCOMPLISHED |
- Meetings with Anna and Vashek
- OSGeo Wiki and Trac pages
- GSoC Weekly reports
- GRASS Development pipeline
- GRASS, Linux, Python resources
- Compiled and install GRASS SVN on Linux virtual machine.
- Reviewed tutorials and resources
- Created Trac and Wiki pages.
- Reviewed coordinate systems and projection issues
- Researched existing data sources and download methods
|
| WEEK 01: MAY 30 - JUNE 02
|
| ACCOMPLISHED |
- 05/30 Meeting with Anna, Vashek, and Paul
- Refined project scope
- Selected target data sources and scale
- Discussed coordinate projection issues and solutions
- Researched USGS NED data download process and web services
- Reviewed GRASS Location Wizard functionality
- Created user scenarios for various software possibilities
- Set up local development tools and coding environment
- Reviewed existing methods and Python scripts for data download into GIS formats
|
| NEXT WEEK |
- Create initial data download script for NED data
- Establish user defined boundaries for import data
- Reconcile coordinate projection difficulties
- Review wxPython
- Reach out to GRASS community for thoughts and resources
|
| BLOCKS |
None
|
| WEEK 02: JUNE 05 - JUNE 09
|
| ACCOMPLISHED |
- Reviewed Python scripting for GRASS documentation and existing GRASS data import modules
- Wrote SRTM 30 download script using Python Elevation library and GRASS output
- Researched USGS data download methods, REST API’s and HTTP protocol
- Reviewed The National Map (TNM) API documentation and tested coordinate output methods from g.region as input to TNM API for SRTM ⅓ arc sec NED tiles
|
| NEXT WEEK |
- How to import and process subsets of IMG tiles into single layer with TNM API through GRASS
- Review wxPython
- Figure out basemap service for user region selection
|
| BLOCKS |
None
|
| WEEK 03: JUNE 11 - JUNE 17
|
| ACCOMPLISHED |
- Worked through different methods of importing and processing NED data with GDAL's vsizip/vsicurl, TNM API, python requests lib, urllib2
- Wrote DRAFT version of r.in.usgsned python script that downloads 1/3 arc-sec NED tiles from the USGS TNM API based on GRASS computational region
- Updated git repo
|
| NEXT WEEK |
- Continue working on r.in.usgsned module script
- Develop GUI and GRASS integration
- Finish basic script functionality to import, patch downloaded tiles
- Get feedback on script functionality, formatting, and implementation
|
| CHALLENGES |
- Thinking through broader implementations of this module
- What are the most useful and stable data repositories to access?
- How to address initial problem of coordinate projections and accuracy across multiple SRSs'
|
| WEEK 04: JUNE 19 - JUNE 23
|
| ACCOMPLISHED |
- Developed GRASS GUI through script flags and options
- Revised script to integrate dynamic paths and user input
- Improved SRS translation
- Began creating format for further USGS data selection
|
| NEXT WEEK |
- Complete r.in.usgsned
- Begin work on r.in.usgs for other datasets
- Test accuracy of SRS conversion for elevation
- Get feedback on script functionality, formatting, and implementation
|
| CHALLENGES |
|
| WEEK 05: JUNE 26 - JUNE 30
|
| ACCOMPLISHED |
- Refined GUI options and flags
- Added GRASS messages, exits, and error handling
- Created "i" flag information report for module
- Converted 'requests' module and JSON handling to built-in urlllib2 and JSON libraries
- Developed data dictionary format for NED data type
- Improved git workflow, issues, and added r.in.usgsned project tasks
|
| NEXT WEEK |
- Integrate clean-up function for temp files from hardware or network interruptions
- Develop automated reprojection handling
- Refine zip archive indexing and download methodl
- Test for conditions not caught by current error handling
|
| CHALLENGES |
|
| WEEK 06: JULY 03 - JULY 07
|
| ACCOMPLISHED |
- Added functionality to handle unique formatting across multiple NED source dates. [1]
- Created dynamic data dictionary for all USGS data sources from TNM API GET request [not pushed to repo]
- Created universal method of indexing and downloading .img file from USGS zip archive[1]
- Refined error handling and GRASS messages for single vs multiple tiles [1]
- Discussed clean-up functionality and project development with mentors
- [1] Most recent git commit
|
| NEXT WEEK |
- Finalize stable version of r.in.usgsned
- Create hard-coded data dictionaries for r.in.usgs data products
- Alter clean-up function to remove .img tiles, keep zip
- Integrate best interpolation methods for data types during r.import
|
| CHALLENGES |
- NOTE: Most recent version of script is not functioning due to incomplete changes.
Will make sure from now on to always push a universally functional version of the script.
- NOTE 2: I will be unavailable 07/12 - 07/21. I will submit next week's report EOD 07/11 and will not submit a report for the following week.
|
| WEEK 07: JULY 10 - JULY 14
|
| ACCOMPLISHED |
- Made significant changes to error handling to improve clean-up functionality, script crashes, and several other structural changes[1]
- Implemented new method of pulling .img files from zip archive[1]
- Code now checks if complete zip files already exist and does not download files that are already available[1]
- Refined error handling and GRASS messages for handling incomplete partially downloaded files[1]
- Created clean-up function
- Changed 'r' flag to 'k' flag that keeps zip archives if checked, removes all temp files by default
[1] Most recent git commit
|
| NEXT WEEK |
|
| CHALLENGES |
|
| WEEK 08: JULY 17 - JULY 21
|
| ACCOMPLISHED |
|
| NEXT WEEK |
- Review and complete ned tasks
- Complete r.in.usgsned
- Begin converting r.in.usgsned into r.in.usgs
- Create hardcoded USGS data dictionaries, continue working on dynamic dictionaries
- Implement other USGS datasets into module
|
| CHALLENGES |
|