Testing framework for GRASS GIS

Title:

Testing framework for GRASS GIS

Student:

Vaclav Petras, ​North Carolina State University, Open Source Geospatial Research and Education Laboratory

Organization:

​OSGeo - Open Source Geospatial Foundation

Mentors:

​Sören Gebbert, ​Helena Mitasova

GSoC link:

​abstract

Abstract

GRASS GIS is one of the core projects in the OSGeo Foundation. GRASS provides wide range of geospatial analyses including raster and vector analyses and image processing. However, there is no system for regular testing of it's algorithms. To ensure software quality and reliability, a standardized way of testing needs to be introduced. This project will implement a testing framework which can be used for writing and running tests of GRASS GIS modules, C/C++ libraries and Python libraries.

Introduction

GRASS GIS is one of the core projects in the OSGeo Foundation and is used by several other free and open source projects to perform geoprocessing tasks. The software quality and reliability is crucial. Thus, proper testing is needed. So far, the testing was done manually by both developers and users. This is questionable in terms of test coverage and frequency of the tests and moreover, it is inconvenient. This project will implement a testing framework which can be used for writing and running tests for GRASS GIS. This will be beneficial not only for the quality of GRASS GIS but also for everyday development of GRASS GIS because it will help to identify problems with the new code at the time when the change is done.

Background

There was already several attempts to establish testing infrastructure for GRASS GIS, namely ​quality assessment and monitoring mailing list which is inactive for several years, then older test suite which was never integrated into GRASS GIS itself, and most recently a ​test suite proposal which was trying to interpret shell scripts as test cases. Also, an experience​ with usage of Python ​doctest at different circumstances shows that this solution is not applicable everywhere.

These previous experiences give us a clear idea what is not working (e.g. tests outside main source code), what is overcomplicated (e.g. reimplementing shell) and what is oversimplified (e.g. shell scripts without clear set up and tear down steps), and point us to the direction of an implementation which will be most efficient (general but simple enough), integrated in GRASS source code, and which will be accepted by the GRASS development team. The long preceding discussions also showed what is necessary to have in the testing framework and what should be left out.

The idea

The purpose of this project is to develop a general mechanism which would be applicable for testing GRASS modules, libraries or workflows with different data sets. Tests will be part of GRASS main source code, cross-platform, and as easy to write and run as possible. The testing framework will enable the use of different testing data sets because different test cases might need special data. The testing framework will be implemented in Python and based on testing tools included in standard Python distribution (most notably ​unittest) which will not bring a new dependency but also it will avoid writing everything from scratch. The usage of Makefile system will be limited to triggering the test or tests with the right parameters for particular location in the source tree, everything else will be implemented in Python to ensure maximum re-usability, maintainability, and availability across platforms.

This project will focus on building infrastructure to test modules, C/C++ libraries (using ctypes interface), and Python libraries. It is expected that testing of Python GUI code will be limited to pure Python parts. The focus will be on the majority of GRASS modules and functionality while special cases such as rendering, creation of locations, external data sources and databases, and downloading of extensions from GRASS Addons will be left for future work. Moreover, this project will not cover tests of graphical user interface, server side automatic testing (e.g. commit hooks), using testing shell scripts or C/C++ programs, and testing of internal functions in C/C++ code (e.g. static functions in libraries and functions in modules). Creation of HTML, XML, or other rich outputs will not be completely solved but the implementation will consider the need for a presentation of test results. Finally, writing the tests for particular parts will not be part of this project, however several sample tests for different parts of code, especially modules, will be written to test the testing framework.

Project plan

date	proposed task
2014-05-19 - 2014-05-23 (week 01)	Designing a basic template for the test case and interface of test suite class(es)
2014-05-26 - 2014-05-30 (week 02)	Basic implementation
2014-06-02 - 2014-06-06 (week 03)	Dealing with evaluation and comparison of textual and numerical outputs
2014-06-09 - 2014-06-13 (week 04)	Dealing with evaluation and comparison of map outputs and other outputs
2014-06-16 - 2014-06-20 (week 05)	Re-writing some existing tests using testing framework
2014-06-23 - 2014-06-27 (week 06)	Testing of what was written so far and evaluating current design and implementation
June 23	Mentors and students can begin submitting mid-term evaluations
June 27	Mid-term evaluations deadline
2014-06-30 - 2014-07-04 (week 07)	Integration with GRASS source code, documentation and build system
2014-07-07 - 2014-07-11 (week 08)	Implementation of location switching
2014-07-14 - 2014-07-18 (week 09)	Dealing with evaluation and comparison of so far unresolved outputs
2014-07-21 - 2014-07-25 (week 10)	Implementing the basic test results reports
2014-07-28 - 2014-08-01 (week 11)	Re-writing some other existing tests using testing framework
2014-08-04 - 2014-08-08 (week 12)	Writing documentation of framework internals and guidelines how to write tests
2014-08-11 - 2014-08-15 (week 13)	Polish the code and documentation
August 11	Suggested 'pencils down' date. Take a week to scrub code, write tests, improve documentation, etc.
2014-08-18 - 2014-08-22 (week 14)	Submit evaluation and code to Google
August 18	Firm 'pencils down' date. Mentors, students and organization administrators can begin submitting final evaluations to Google.
August 22	Final evaluation deadline
August 22	Students can begin submitting required code samples to Google

Design of testing API

import unittest
import grass.pygrass.modules as gmodules

# alternatively, these can be private to module with setter and getter
# or it can be in a class
USE_VALGRIND = False


class GrassTestCase(unittest.TestCase):
    """Base class for GRASS test cases."""

    def run_module(self, module):
        """Method to run the module. It will probably use some class or instance variables"""
        # get command from pygrass module
        command = module.make_cmd()
        # run command using valgrind if desired and module is not python script
        # see also valgrind notes at be end of this section
        if is_not_python_script(command[0]) and USE_VALGRIND:
            command = ['valgrind', '--tool=...', '--xml=...', '--xml-file=...'] + command
        # run command
        # store valgrind output (memcheck has XML output to a file)
        # store module return code, stdout and stderr, how to distinguish from valgrind?
        # return code, stdout and stderr could be returned in tuple
    
    def assertRasterMap(self, actual, reference, msg=None):
        # e.g. g.compare.md5 from addons
        # uses msg if provided, generates its own if not,
        # or both if self.longMessage is True (unittest.TestCase.longMessage)
        # precision should be considered too (for FCELL and DCELL but perhaps also for CELL)
        # the actual implementation will be in separate module, so it can be reused by doctests or elsewhere
        # this is necessary considering the number and potential complexity of functions
        # and it is better design anyway

        if check sums not equal:
            self.fail(...)  # unittest.TestCase.fail

class SomeModuleTestCase(GrassTestCase):
    """Example of test case for a module."""
    
    def test_flag_g(self):
        """Test to validate the output of r.info using flag "g"
        """
        # Configure a r.info test 
        module = gmodules.Module("r.info", map="test", flags="g", run_=False)

        self.run_module(module=module)
        # it is not clear where to store stdout and stderr
        self.assertStdout(actual=module.stdout, reference="r_info_g.ref")
        
    def test_something_complicated(self):
        """Test something which has several outputs
        """
        # Configure a r.info test 
        module = gmodules.Module("r.complex", rast="test", vect="test", flags="p", run_=False)
        
        (ret, stdout, stderr) = self.run_module(module=module)
        self.assertEqual(ret, 0, "Module should have suceed but return code is not 0")
        self.assertStdout(actual=stdout, reference="r_complex_stdout.ref")
        self.assertRasterMap(actual=module.rast, reference="r_complex_rast.ref")
        self.assertVectorMap(actual=module.vect, reference="r_complex_vect.ref")

Compared to suggestion in ticket:2105#comment:4 it does not solve everything in test_module (run_module) function but it uses self.assert* similarly to unittest.TestCase which (syntactically) allows to check more then one thing.

Test scripts must have module/package character (as unittests requires for test discovery). This applies true for unittests and doctests, no exceptions. Doctests (inside normal module code or in separate doc) will be wrapped as unittest test cases (in testsuite directory). There is a ​standard way to do it. To have the possibility of import, all the GRASS Python libraries shouldn't do anything fancy at import time. For example, doctests currently don't work with grass.script unless you call a set of functions​ to deal with function _ because of installing translate function as buildin _ function while _ is used also in doctest. (This is fixed for GUI but still applies to Python libraries).

Analyzing module run using valgrind or others

Modules (or any tests?) can run with valgrind (probably --tool=memcheck). This could be done on the level of testing classes but the better option is to integrate this functionality (optional running with valgrind). Environmental variable (GRASS_PYGRASS_VALGRIND) or additional option valgrind_=True (similarly to overwrite) would invoke module with valgrind (works for both binaries and scripts). Additional options can be passed to valgrind using valgrind's environmental variable $VALGRIND_OPTS. Output would be saved in file to not interfere with module output.

We may want to use also some (runtime checking) tools other than valgrind, for example clang/LLVM sanitizers (as for example ​Python does). However, it is unclear how to handle more than one tool as well as it is unclear how to store the results for any of these (including valgrind) because one test can have multiple module calls (or none), module calls can be indirect (function in Python lib which calls a module or module calling module) and there is no standard way in unittest to pass additional test details.

PyGRASS or specialized PyGRASS module runner (function) in testing framework can have function, global variable, or environmental variable which would specify which tool should run a module (if any) and what are the parameters (besides the possibility to set parameters by environmental variable defined by the tool). The should ideally be separated from the module output and go to a file in the test output directory (and it could be later linked from/included into the main report).

Data types to be checked

We must deal especially with GRASS specific files such as raster maps. We consider that comparison of simple things such as strings and individual numbers is already implemented by ​unittest.

• raster map
  • composite? reclassified map?
  • color table included
• vector map
• 3D raster map
• color table
• SQL table
• file

Most of the outputs can be checked with different numerical precision.

Resources:

• ​http://grasswiki.osgeo.org/wiki/Test_Suite#Modules

Naming conventions

The ​unittest.TestLoader.discover function requires that module names are importable (i.e. are valid Python identifiers). Consequently, names of files with tests should contain dots (except for the .py suffix).

Methods with tests must start with test_ to be recognized by the ​unittest framework (with default setting but there is no reason to not keep this convention).

Name for directory with test is "testsuite". It also fits to how unittest is using this term (set of test cases and other test suites). "test" and "tests" is simpler and you can see it, for example ​in Python, but might be too general. "unittest" would confuse with the module unittest.

Layout of directories and files

Test scripts are located in a dedicated directory within module or library directories. All reference files or additionally needed data should be located there.

The same directory as tested would work well for one or two Python files but not for number of reference files. In case of C/C++ this would mean mixing Python and C/C++ files in one directory which is strange. One directory in root with separate tree is something which would not work either because tests are not close enough to actual code to be often run and extended when appropriate.

Invoking tests

Test scripts will work when directly executed from command line and when executed from the make system.

Tests should be executable by itself (i.e. should they have main() function) to encourage running them often. It is not clear if this should be be used by the testing framework when running multiple tests or if it is better to import.

Example run

cd lib/python/temporal

Now there are two options to run the tests. First execution by hand in my current location:

cd testsuite
for test in `ls *.py` ; do
    python ${test} >> /tmp/tgis_lib.txt 2>&1
done

The test output will be written to stdout and stderr (piped to a file in this case).

Second option is an execution by the make system (still in lib/python/temporal):

make tests

Locations, mapsets and data

Weekly reports