Changes between Version 2 and Version 3 of GSoC/2014/TestingFrameworkForGRASS

Timestamp:

05/19/14 12:28:02 (11 years ago)

Author:

wenzeslaus

Comment:

first draft of basic part of API

GSoC/2014/TestingFrameworkForGRASS

@@ -60 +60 @@
 || August 22 || Final evaluation deadline
 || August 22 || Students can begin submitting required code samples to Google
+
+== Design of testing API ==
+
+{{{
+#!python
+import unittest
+import grass.pygrass.modules as gmodules
+
+# alternatively, these can be private to module with setter and getter
+# or it can be in a class
+USE_VALGRIND = False
+
+
+class GrassTestCase(unittest.TestCase):
+    """Base class for GRASS test cases."""
+
+    def run_module(self, module):
+        """Method to run the module. It will probably use some class or instance variables"""
+        # get command from pygrass module
+        command = module.make_cmd()
+        # run command using valgrind if desired and module is not python script
+        if is_not_python_script(command[0]) and USE_VALGRIND:
+            command = ['valgrind', '--tool=...', '--xml=...', '--xml-file=...'] + command
+        # run command
+        # store valgrind output (memcheck has XML output to a file)
+        # store module return code, stdout and stderr, how to distinguish from valgrind?
+        # return code, stdout and stderr could be returned in tuple
+    
+    def assertRasterMap(self, actual, reference, msg=None):
+        # e.g. g.compare.md5 from addons
+        # uses msg if provided, generates its own if not,
+        # or both if self.longMessage is True (unittest.TestCase.longMessage)
+        # precision should be considered too (for FCELL and DCELL but perhaps also for CELL)
+        if check sums not equal:
+            self.fail(...)  # unittest.TestCase.fail
+}}}
+
+{{{
+#!python
+class SomeModuleTestCase(GrassTestCase):
+    """Example of test case for a module."""
+    
+    def test_flag_g(self):
+        """Test to validate the output of r.info using flag "g"
+        """
+        # Configure a r.info test 
+        module = gmodules.Module("r.info", map="test", flags="g", run_=False)
+
+        self.run_module(module=module)
+        # it is not clear where to store stdout and stderr
+        self.assertStdout(actual=module.stdout, reference="r_info_g.ref")
+        
+    def test_something_complicated(self):
+        """Test something which has several outputs
+        """
+        # Configure a r.info test 
+        module = gmodules.Module("r.complex", rast="test", vect="test", flags="p", run_=False)
+        
+        (ret, stdout, stderr) = self.run_module(module=module)
+        self.assertEqual(ret, 0, "Module should have suceed but return code is not 0")
+        self.assertStdout(actual=stdout, reference="r_complex_stdout.ref")
+        self.assertRasterMap(actual=module.rast, reference="r_complex_rast.ref")
+        self.assertVectorMap(actual=module.vect, reference="r_complex_vect.ref")
+}}}
+
+Compared to suggestion in ticket:2105#comment:4 it does not solve everything in `test_module` (`run_module`) function but it uses `self.assert*` similarly to `unittest.TestCase` which (syntactically) allows to check more then one thing.
+
+== Data types to be checked ==
+
+We must deal especially with GRASS specific files such as raster maps. We consider that comparison of simple things such as strings and individual numbers is already implemented by [https://docs.python.org/2/library/unittest.html#test-cases unittest].
+
+ * raster map
+  * composite? reclassified map?
+  * color table included
+ * vector map
+ * 3D raster map
+ * color table
+ * SQL table
+ * file
+
+Most of the outputs can be checked with different numerical precision.
+
+Resources:
+ * http://grasswiki.osgeo.org/wiki/Test_Suite#Modules