Changes between Version 49 and Version 50 of GSoC/2014/TestingFrameworkForGRASS

Timestamp:

07/25/14 21:25:47 (10 years ago)

Author:

wenzeslaus

Comment:

Weekly report 10

GSoC/2014/TestingFrameworkForGRASS

@@ -763 +763 @@
 }}}
 
+=== Week 10 ===
+
+1. What did you get done this week?
+
+I significantly improved the test file page which is the central page of the report in the sense that there you can get all the details. I also added details about successful and failed (individual) tests to main index page (the one with links to all test files). Additionally, I created different set of pages which sorts the test files from the point of view of testsuite directories which might be the primary way of browsing tests since it is aligned with how tests are included in the source code. The page for a given testsuite also contains approximate information about authors of tests and authors of tested code.
+
+Each test file has now a key-value file which contains information about success, number of tests, and other information. These files can and are collected to obtain summaries. The key-value format was selected mainly because of its usage in GRASS GIS (otherwise, for example [https://docs.python.org/2/tutorial/inputoutput.html#saving-structured-data-with-json Python documentation] for inputs and outputs recommends JSON).
+
+I believe that I designed report classes in the way that they can be extended but already I can see some room for refactoring or at least for better organization. This might be a topic for week 13.
+
+I did not created any XML output. The situation seems unclear and a lot of transformations are done to get XML working with different tools. On the other hand, with some tools even trivial XML with numbers (test statistics) should work and this should be easy to create from key-value files or directly. Also there is a [https://github.com/xmlrunner/unittest-xml-reporting unittest-xml-reporting package] for Python `unittest` which should create XML in the style of Ant/JUnit. In theory this should work directly with `gunittest`. I was not considering database output much but I think that it is not needed now and it can be created from key-value files if necessary.
+
+A also did not focus on CSS of the HTML reports. They are readable but that's all. Some CSS might be added in the future.
+
+There is currently 20 successfully running test files and 5 failing test files, 9 successfully running testsuites and 4 failing testsuites, and 686 successfully running tests and 7 failing tests (533 of the tests are from PyGRASS module test).
+
+I also added some tests for `r.mapcalc` `rand()` function and new behavior with seed in r61355.
+
+2. What do you plan on doing next week?
+
+I plan to rewrite some of the existing tests using `gunittest` or write new ones. I already started to write something for `r.slope.aspect` and `r.mapcalc`.
+
+If time allows, I would like to add some key-value summary files to reports and also introduce more options to report generation (currently, it cannot be configured).
+
+3. Are you blocked on anything?
+
+I think that some CSS and !JavaScript can improve the reports significantly but I did not included any nice design to HTML pages because it would cost me too much time (and it was not part of the GSoC plan anyway).
+
+
+==== Index page for all test files ====
+
+Contains a table with one row for each test file. The testsuite directory is in the first column (precisely, it is a directory of testsuite directory of a test file). Table contains information about the failure/success of the file itself (i.e. return code) and also number of tests in each file. This page is generated during invocation of test files. The detail about tests are collected from key-value files which are also updated or created if they were not written by test file itself (e.g. because of segfault). Date and time when the test started and SVN revision number are included too. Some tests may not be included if the test file ends without writing this information (or if test file is not using `gunittest`).
+
+[[Image(test_files_index.png)]]
+
+==== Testsuite oriented pages ====
+
+Testsuite pages are compiled after all tests was completed based on the information stored in the key-value files. One main index page with links and statistics for a testsuite directories is created also a page for each testsuite directory is created. The main index page contains status of the whole directory, number of test files in it and number of tests in these files. The testsuite page contains a table with the same columns as the index page for all test files but only test files from testsuite directory are present. Additionally, this page contains information about authors of code and authors of tests which is obtained using SVN. The information is not precise because of limitations of what SVN can give and how the needed information can be extracted (more precise analyses could be done only using `svn blame` on each file and counting lines which would be very costly and still it would contain hight amount of uncertainty).
+
+[[Image(testsuites_index.png)]]
+[[Image(testsuite_gunittest.png)]]
+[[Image(testsuite_gmlist.png)]]
+
+==== Successful test file ====
+
+Successful test file report page contains short summary and links to stdout and stdin. If the test case (TestCase class, or anything in test file) creates and registers some files with details about test (currently only HTML diff of two vector ASCII files), these are linked, too. If test file collected information about which modules were tested, summary table should contain this information (currently broken).
+
+[[Image(test_file_trastextract.png)]]
+
+==== Failed test file ====
+
+Failed test file report page contains, in addition to what the successful page contains, also stderr (shortened if necessary). Stderr contains besides other things output from a clone of Python `unittest` test result runner and reporter classes, so here is the main information about what failed.
+
+[[Image(test_file_r3assert.png)]]
+
+=== Week 11 ===
+
+=== Week 12 ===
+
+=== Week 13 ===
+
+=== Week 14 ===