GSoC 2017 SOS tools in GRASS GIS

Title:	SOS tools in GRASS GIS
Student Name:	Ondrej Pesek, ​Czech Technical University in Prague
Organization:	​OSGeo - Open Source Geospatial Foundation
Mentor Name:	​Luca Delucchi, Matteo De Stefano
GSoC proposal:	​view proposal
Repositories:	Github for development:	​https://github.com/pesekon2/GRASS-GIS-SOS-tools
	GRASS SVN AddOns for final modules:	​https://svn.osgeo.org/grass/grass-addons/grass7/
	OWSlib fork with SOS patches:	​https://github.com/pesekon2/OWSLib
	istSOS fork with some patches:	​https://github.com/pesekon2/istsos2

Abstract

GRASS GIS doesn’t have any module to work with Sensor Observation Service (SOS). SOS is an Open Geospatial Consortium (OGC) standard and it is useful to implement modules to work with this standard into GRASS.

Goal

Intended modules would enable the user to create a vector with or without the observation values, create a raster for each queried day and create a space time vector or raster dataset. One module would also allow the user to convert a space time vector dataset into a raster dataset. The user should be also allowed to get the capabilities to get info about sensors from these modules and filter the results.

Intended modules would also use OWSLib to work with the server and data. There are a few things to improve in OWSLib, so this should be the first part of my GSoC project. I would also like to use pyGrass (a python library allowing users to access the low-level GRASS API) to create new modules.

Timeline

		status
Community bounding period	Studying OWSLib and SOS	*
Let the summer begin
MAY 30 - JUNE 2	Testing OWSLib functionalities on different SOS servers	*
JUNE 5 - JUNE 9	Improving OWSLib functionalities	*
JUNE 12 - 16	Improving OWSLib, developing background for forthcoming modules	*
JUNE 19 - 23	v.in.sos (creating a vector with or without the observation values)	*
JUNE 26 - 30	v.in.sos upgrades, t.vect.in.sos shuck	*
First evaluations
JULY 3 - 14	I am sorry but I will have no access to internet and neither to electricity due to my work at summer camp	*
JULY 17 - 21	t.vect.in.sos (creating a space time vector dataset)	*
JULY 24 - 28	r.in.sos (creating a raster for each queried day)	*
Second evaluations
JULY 31 - AUGUST 4	t.rast.in.sos (creating a space time raster dataset)	*
AUGUST 7 - 11	t.vect.to.rast (convert space time vector dataset into space time raster dataset)	*
AUGUST 14 - 18	Writing documentation, buffer in case of delay
AUGUST 21 - 29	Submitting the final work product
Final evaluations

Requirements

​GRASS 7.3

Development

Weekly reports

May 30 - June 2

Studied OWSLib and its work with SOS (especially with istSOS) and in which formats are available in response.

Solved ​this issue in OWSLib.

Created converter between normal JSON output from OWSLib and geoJSON for points

June 5 - June 9

I had a problem with xml parser in OWSLib for SOS observations, because it works completely different way than we expected. After few conversations I have decided to work with raw output and write parser by myself.

Created pull request solving ​encoding issue for OWSLib.

Created converter between text/xml;subtype="om/1.0.0" output from OWSLib and geoJSON for points

Created a shuck of v.in.sos module

June 12 - June 16

Created v.in.sos

• Makefile and first html
• UI
• Prints informations about sensor offerings
• Imports the observation into GRASS GIS
• This version is returning just the last observation

June 19 - June 23

v.in.sos

• flags for printing timestamps of begin and end of observation
• working with event_time (interval of observation is used in query)
• v.in.sos returns one map with many layers (layers count = observed properties count)

xml2geojson

• working with separated blocks of values, not just with single values

June 26 - June 30

v.in.sos

• using vectors in pyGRASS instead of v.in.sos
• support multiple offerings
• layers are for offerings and observed properties, rows are for one feature
• handle data input with no observations for
• handle too big input with warnings and errors

t.vect.in.sos

• first version (just draft with some bugs, they are saved as issues and will be solved after my return)

July 3 - July 7

I was out of internet and electricity during the last week. So I wasn't able to do anything connected with GSoC.

July 10 - July 14

I was out of internet and electricity during the last week. So I wasn't able to do anything connected with GSoC.

July 17 - July 21

Visited FOSS4G-E

Implemented shell script style output into modules

t.vect.in.sos

• new vector for each offering and observed property

r.in.sos

• first version

July 24 - July 28

All modules

• Shell script style update
• Handle import errors
• Flags working with multiple offerings
• Not given options are generated just temporarily

xml2geojson

• Observed properties can be given as their ids

t.vect.in.sos

• Creating stvds

r.in.sos

• Creating raster maps

July 31 - August 4

r.in.sos

• flag for deleting vector maps created as intermediates
• importing vectors and values in one step

v.in.sos

• importing vectors and values in one step

All modules

• not stop when using just -g flag
• code cleaning
• manuals updates

A little pull request for istSOS

• allow user to define offering in csv2istsos

August 7 - August 11

v.in.sos

• finally working the desired way even with more layers
  • uses yet existing columns for timestamps instead of creating new ones

t.vect.in.sos

• finally working with layers-based vectors

t.rast.in.sos

August 14 - August 18

all modules

• made quiet, e.g. when creating temporal dataset with hundreds of maps, it doesn't show hundreds of builds
• worked on documentation

t.vect.in.sos

• uses yet existing timestamped layers instead of creating new ones

t.vect.to.rast

• created

r.in.sos

• updating yet existing maps instead of creating new ones
• option for granularity and aggregations

Final Report

Project Title:

SOS tools in GRASS GIS

Organization:

Google Summer of Code 2017

Open Source Geospatial Foundation (OSGeo)

GRASS GIS

Abstract:

Intended modules would enable the user to create a vector, a raster based on some aggregations and create a space time vector or raster dataset, everything directly from user's access to the SOS server. t.vect.to.rast would also allow the user to convert a space time vector dataset into a raster dataset. The user should be also allowed to get the capabilities to get info about sensors from these modules and filter the results.

Pre-GSoC: GRASS GIS didn't have any module to work with Sensor Observation Service (SOS). When the user wanted to use data from his SOS server, he had to access them through OWSLib, convert them into GRASS GIS or OGR supported format ad then import it somehow to GRASS GIS as one huge file.

Added value: v.in.sos imports data from SOS server as a vector map to GRASS GIS. It creates one layer for each offering and observed property.

r.in.sos imports data from SOS server as a raster maps to GRASS GIS. It creates new raster map for each timestamp.

t.vect.in.sos imports data from SOS server to GRASS GIS as a spatio-temporal vector dataset. It creates new stvds for each offering and observed property (created from one vector map as an intermediate).

t.rast.in.sos imports data from SOS server to GRASS GIS as a spatio-temporal raster dataset. It creates new strds for each property and each procedure (registered from raster maps created as intermediates).

t.vect.to.rast doesn't import data from SOS server to GRASS GIS. It converts a space time vector dataset into a space time raster dataset.

User is allowed to use some aggregations and granularity to filter data in all modules.

Continued Work: There are some issues or possible enhancements in the modules.

v.in.sos uses timestamps as column names for each layer. The problem is that it is not possible to have more than 3000 columns in SQLite table (GRASS GIS attribute table) without SQLite recompilation. It is little bit solved by granularity and this module isn't so necessary when there is t.vect.in.sos, but it is still useful for some purposes and this is really lack.

t.vect.to.rast is working, but if you take a look at t.rast.to.vect, you can see much more options. It would be great to involve them also into this module.

It would be also great to have a flag for ignoring empty procedures in all the SOS modules.

Link:

The modules github repository: [​https://github.com/pesekon2/GRASS-GIS-SOS-tools

(they should be moved into the official GRASS GIS add-ons repository in the future)

One image to show how can visualized sensors look on the OSM map: ​https://raw.githubusercontent.com/pesekon2/GRASS-GIS-SOS-tools/d566c868a50f12ba6fc4e1d6b953e9708e4b0061/image_vector_import.png

What do you need to test these AddOns?

To use this module, you need your OWSLib up-to-date. If you don't have it, please install it from its ​github repository.

Because of granularity options, you need to be using GRASS > 7.2 to have access to some functions.