| | 1 | [[TOC]] |
| | 2 | |
| | 3 | = Testing framework for GRASS GIS = |
| | 4 | |
| | 5 | |
| | 6 | || Title: || '''Testing framework for GRASS GIS'''|| |
| | 7 | |
| | 8 | || Student: || Vaclav Petras, [http://gis.ncsu.edu/osgeorel/ North Carolina State University, Open Source Geospatial Research and Education Laboratory]|| |
| | 9 | |
| | 10 | || Organization: || [http://www.osgeo.org OSGeo - Open Source Geospatial Foundation]|| |
| | 11 | |
| | 12 | || Mentors: || [http://grasswiki.osgeo.org/wiki/User:Huhabla Sören Gebbert], [http://www4.ncsu.edu/~hmitaso/ Helena Mitasova] || |
| | 13 | |
| | 14 | || GSoC link: || [https://www.google-melange.com/gsoc/project/details/google/gsoc2014/wenzeslaus/5741031244955648 abstract] || |
| | 15 | |
| | 16 | == Abstract == |
| | 17 | |
| | 18 | GRASS GIS is one of the core projects in the OSGeo Foundation. GRASS provides wide range of geospatial analyses including raster and vector analyses and image processing. However, there is no system for regular testing of it's algorithms. To ensure software quality and reliability, a standardized way of testing needs to be introduced. This project will implement a testing framework which can be used for writing and running tests of GRASS GIS modules, C/C++ libraries and Python libraries. |
| | 19 | |
| | 20 | |
| | 21 | == Introduction == |
| | 22 | |
| | 23 | GRASS GIS is one of the core projects in the OSGeo Foundation and is used by several other free and open source projects to perform geoprocessing tasks. The software quality and reliability is crucial. Thus, proper testing is needed. So far, the testing was done manually by both developers and users. This is questionable in terms of test coverage and frequency of the tests and moreover, it is inconvenient. This project will implement a testing framework which can be used for writing and running tests for GRASS GIS. This will be beneficial not only for the quality of GRASS GIS but also for everyday development of GRASS GIS because it will help to identify problems with the new code at the time when the change is done. |
| | 24 | |
| | 25 | |
| | 26 | == Background == |
| | 27 | |
| | 28 | There was already several attempts to establish testing infrastructure for GRASS GIS, namely quality assessment and monitoring mailing list which is inactive for several years, then older test suite which was never integrated into GRASS GIS itself, and most recently a test suite proposal which was trying to interpret shell scripts as test cases. Also, an experience with usage of Python doctest at different circumstances shows that this solution is not applicable everywhere. |
| | 29 | |
| | 30 | These previous experiences give us a clear idea what is not working (e.g. tests outside main source code), what is overcomplicated (e.g. reimplementing shell) and what is oversimplified (e.g. shell scripts without clear set up and tear down steps), and point us to the direction of an implementation which will be most efficient (general but simple enough), integrated in GRASS source code, and which will be accepted by the GRASS development team. The long preceding discussions also showed what is necessary to have in the testing framework and what should be left out. |
| | 31 | |
| | 32 | |
| | 33 | == The idea == |
| | 34 | |
| | 35 | The purpose of this project is to develop a general mechanism which would be applicable for testing GRASS modules, libraries or workflows with different data sets. Tests will be part of GRASS main source code, cross-platform, and as easy to write and run as possible. The testing framework will enable the use of different testing data sets because different test cases might need special data. The testing framework will be implemented in Python and based on testing tools included in standard Python distribution (most notably unittest) which will not bring a new dependency but also it will avoid writing everything from scratch. The usage of Makefile system will be limited to triggering the test or tests with the right parameters for particular location in the source tree, everything else will be implemented in Python to ensure maximum re-usability, maintainability, and availability across platforms. |
| | 36 | |
| | 37 | This project will focus on building infrastructure to test modules, C/C++ libraries (using ctypes interface), and Python libraries. It is expected that testing of Python GUI code will be limited to pure Python parts. The focus will be on the majority of GRASS modules and functionality while special cases such as rendering, creation of locations, external data sources and databases, and downloading of extensions from GRASS Addons will be left for future work. Moreover, this project will not cover tests of graphical user interface, server side automatic testing (e.g. commit hooks), testing using shell scripts or C++/C programs, and testing of internal functions in C/C++ code (e.g. static functions in libraries and functions in modules). Creation of HTML, XML, or other rich outputs will not be completely solved but the implementation will consider the need for a presentation of test results. Finally, writing the tests for particular parts will not be part of this project, however several sample tests for different parts of code, especially modules, will be written to test the testing framework. |
| | 38 | |
| | 39 | == Project plan == |
| | 40 | |
| | 41 | || date || proposed task || |
| | 42 | || 2014-05-19 - 2014-05-23 (week 01) || Designing a basic template for the test case and interface of test suite class(es) || |
| | 43 | || 2014-05-26 - 2014-05-30 (week 02) || Basic implementation |
| | 44 | || 2014-06-02 - 2014-06-06 (week 03) || Dealing with evaluation and comparison of textual and numerical outputs |
| | 45 | || 2014-06-09 - 2014-06-13 (week 04) || Dealing with evaluation and comparison of map outputs and other outputs |
| | 46 | || 2014-06-16 - 2014-06-20 (week 05) || Re-writing some existing tests using testing framework |
| | 47 | || 2014-06-23 - 2014-06-27 (week 06) || Testing of what was written so far and evaluating current design and implementation |
| | 48 | || June 23 || Mentors and students can begin submitting mid-term evaluations |
| | 49 | || June 27 || Mid-term evaluations deadline |
| | 50 | || 2014-06-30 - 2014-07-04 (week 07) || Integration with GRASS source code, documentation and build system |
| | 51 | || 2014-07-07 - 2014-07-11 (week 08) || Implementation of location switching |
| | 52 | || 2014-07-14 - 2014-07-18 (week 09) || Dealing with evaluation and comparison of so far unresolved outputs |
| | 53 | || 2014-07-21 - 2014-07-25 (week 10) || Implementing the basic test results reports |
| | 54 | || 2014-07-28 - 2014-08-01 (week 11) || Re-writing some other existing tests using testing framework |
| | 55 | || 2014-08-04 - 2014-08-08 (week 12) || Writing documentation of framework internals and guidelines how to write tests |
| | 56 | || 2014-08-11 - 2014-08-15 (week 13) || Polish the code and documentation |
| | 57 | || August 11 || Suggested 'pencils down' date. Take a week to scrub code, write tests, improve documentation, etc. |
| | 58 | || 2014-08-18 - 2014-08-22 (week 14) || Submit evaluation and code to Google |
| | 59 | || August 18 || Firm 'pencils down' date. Mentors, students and organization administrators can begin submitting final evaluations to Google. |
| | 60 | || August 22 || Final evaluation deadline |
| | 61 | || August 22 || Students can begin submitting required code samples to Google |
