Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes

[AnikaBettge]

Tzhe multirunner in trunk currently fails with Python3:

(grasspy3) ✔ ~/src/grass-7.7.svn/testsuite/examples 
11:41 $ bash test_framework_GRASS_GIS_with_NC.sh test_framework_GRASS_GIS_with_NC_AB.conf
--2019-01-23 13:42:48--  https://grass.osgeo.org/sampledata/north_carolina/nc_spm_08_grass7.tar.gz
...

Starting GRASS GIS...
Cleaning up temporary files...
Executing <sh /home/abettge/grassdata/tests-grassdata/tmp_rename.sh> ...
WARNUNG: <basin> already exists. File not copied.
WARNUNG: <boundary> already exists. File not copied.
...
Execution of <sh /home/abettge/grassdata/tests-grassdata/tmp_rename.sh> finished.
Cleaning up default sqlite database ...
Cleaning up temporary files...
Traceback (most recent call last):
  File "/home/abettge/src/grass-7.7.svn/lib/python/gunittest/multirunner.py", line 125, in <module>
    sys.exit(main())
  File "/home/abettge/src/grass-7.7.svn/lib/python/gunittest/multirunner.py", line 76, in main
    os.environ['GISBASE'] = gisbase
  File "/home/abettge/src/grass-7.7.svn/grasspy3/lib/python3.5/os.py", line 730, in __setitem__
    value = self.encodevalue(value)
  File "/home/abettge/src/grass-7.7.svn/grasspy3/lib/python3.5/os.py", line 798, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not bytes

Version:

grass77 --config svn_revision
74007

[pmav99]

In order to make multirunner.py run on Python 3 you need to apply something like this.

@@ -64,13 +64,13 @@
     # we assume that GRASS GIS' start script is available and in the PATH
     # the shell=True is here because of MS Windows? (code taken from wiki)
     startcmd = grass7bin + ' --config path'
     p = subprocess.Popen(startcmd, shell=True, 
                          stdout=subprocess.PIPE, stderr=subprocess.PIPE)
     out, err = p.communicate()
     if p.returncode != 0:
         print("ERROR: Cannot find GRASS GIS 7 start script (%s):\n%s" % (startcmd, err), file=sys.stderr)
         return 1
-    gisbase = out.strip('\n')
+    gisbase = out.strip().decode("utf-8")
 
     # set GISBASE environment variable
     os.environ['GISBASE'] = gisbase
@@ -95,7 +95,7 @@
     gsetup.init(gisbase, gisdb, location, mapset)
 
     reports = []
-    for location, location_type in itertools.izip(locations, locations_types):
+    for location, location_type in zip(locations, locations_types):
         # here it is quite a good place to parallelize
         # including also type to make it unique and preserve it for sure
         report = 'report_for_' + location + '_' + location_type

Specifying \n is probably redundant. str.strip() already strips that. More specifically it strips all the characters contained in string.whitespace.

Other than that, in python3 subprocess output is bytes. The .decode("utf-8") call is converting the bytes to python3 strings which are the equivalent of unicode strings in python2.

Note: the patch should work with python2 too, but I have not tested it.

[pmav99]

That being said, even after applying the patch the testrunner is still not running as expected. More specifically all the tests are failing with this message on stderr.txt:

.../stderr.txt
Unknown option: -3
usage: python3 [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.

Python 2 does indeed have a -3 switch but Python 3 does not. I am looking but I can't find where this switch is added. If anyone has any idea, please post.

[pmav99]

This is also needed

Index: lib/python/gunittest/invoker.py
===================================================================
--- lib/python/gunittest/invoker.py	(revision 74025)
+++ lib/python/gunittest/invoker.py	(working copy)
@@ -161,7 +161,7 @@
             # ignoring shebang line to use current Python
             # and also pass parameters to it
             # add also '-Qwarn'?
-            p = subprocess.Popen([sys.executable, '-tt', '-3',
+            p = subprocess.Popen([sys.executable, '-tt',
                                   module.abs_file_path],
                                  cwd=cwd, env=env,
                                  stdout=stdout, stderr=stderr)

[pmav99]

After applying the previous patches, the vast majority of the tests were again failing due to trying to mix strings and bytes in shutil_which(). The following patch fixes that and the tests can finally run. There are still failing ones, but that's a different issue. I opted to add an additional decode() call since it seems less likely to break anything.

Index: script/core.py
===================================================================
--- script/core.py	(revision 74025)
+++ script/core.py	(working copy)
@@ -234,7 +234,9 @@
         if not normdir in seen:
             seen.add(normdir)
             for thefile in files:
-                name = os.path.join(encode(dir), thefile)
+                name = os.path.join(encode(dir), encode(thefile))
                 if _access_check(name, mode):
                     return name
     return None

[annakrat]

Thanks for looking into it, I fixed the things in gunittest in r74032, but I have still some local changes related to shutil_which in core.py I need to test more first.

[pmav99]

Hello Anna, is it necessary to redefine encode() and decode()? They are already defined in a bunch of places already...

$ ag -l 'def encode' lib/python

lib/python/script/utils.py
lib/python/ctypes/preamble.py
lib/python/ctypes/ctypesgencore/printer/preamble.py

The one in script/utils.py seems to be OK.

[annakrat]

Replying to pmav99:

Hello Anna, is it necessary to redefine encode() and decode()? They are already defined in a bunch of places already...

$ ag -l 'def encode' lib/python

lib/python/script/utils.py
lib/python/ctypes/preamble.py
lib/python/ctypes/ctypesgencore/printer/preamble.py

The one in script/utils.py seems to be OK.

True, but the library is not loaded at that point when I need to decode.

[AnikaBettge]

Cleaning up default sqlite database ...
Cleaning up temporary files...
raster3d_lib_test from ./lib/raster3d failed (3 tests failed)
test_assertions from ./lib/python/gunittest failed (3 tests failed)
test_checkers from ./lib/python/gunittest failed (1 test failed)
test_gmodules from ./lib/python/gunittest failed (4 tests failed)
test_assertions_vect from ./lib/python/gunittest failed (7 tests failed)
test_assertions_rast3d from ./lib/python/gunittest failed (2 tests failed)
test_module_assertions from ./lib/python/gunittest failed (2 tests failed)
test_doctests from ./lib/python/gunittest failed (2 tests failed)
unittests_temporal_raster_conditionals from ./lib/python/temporal failed (50 tests failed)
unittests_temporal_raster_conditionals_complement_else from ./lib/python/temporal failed (4 tests failed)
unittests_temporal_vector_algebra from ./lib/python/temporal failed (14 tests failed)
unittests_temporal_conditionals from ./lib/python/temporal failed (34 tests failed)
unittests_temporal_raster_algebra_grs from ./lib/python/temporal failed (13 tests failed)
unittests_temporal_algebra_grs from ./lib/python/temporal failed (6 tests failed)
unittests_temporal_raster_algebra from ./lib/python/temporal failed (80 tests failed)
unittests_temporal_raster3d_algebra from ./lib/python/temporal failed (4 tests failed)
unittests_temporal_raster_algebra_equal_ts from ./lib/python/temporal failed (10 tests failed)
unittests_temporal_algebra from ./lib/python/temporal failed (39 tests failed)
unittests_temporal_raster_algebra_spatial_topology from ./lib/python/temporal failed (12 tests failed)
test_doctests from ./lib/python/temporal failed (15 tests failed)
unittests_temporal_algebra_mixed_stds from ./lib/python/temporal failed (9 tests failed)
test_doctests from ./lib/python/script failed (1 test failed)
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/main.py", line 178, in <module>
    sys.exit(main())
  File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/main.py", line 174, in main
    results_dir=results_dir)
  File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/invoker.py", line 237, in run_in_location
    gisdbase=gisdbase, location=location)
  File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/invoker.py", line 193, in _run_test_module
    self._file_anonymizer.anonymize([stdout_path, stderr_path])
  File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/reporters.py", line 103, in anonymize
    replace_in_file(filename, path + path_end, '')
  File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/reporters.py", line 61, in replace_in_file
    for line in old_file:
  File "/home/abettge/src/grass-7.7.svn/grasspy3/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 439: invalid continuation byte
GRASS Location <other_location> does not exist in GRASS Database </home/abettge/grassdata/tests-grassdata>
Traceback (most recent call last):
  File "/home/abettge/src/grass-7.7.svn/lib/python/gunittest/multireport.py", line 28, in <module>
    from grass.gunittest.checkers import text_to_keyvalue
  File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/checkers.py", line 18, in <module>
    from grass.script.core import KeyValue
  File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/script/__init__.py", line 5, in <module>
    from .core   import *
  File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/script/core.py", line 38, in <module>
    gettext.install('grasslibs', os.path.join(os.getenv("GISBASE"), 'locale'))
  File "/home/abettge/src/grass-7.7.svn/grasspy3/lib/python3.6/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

grass77 --config svn_revision 74046M

[neteler]

State as of today:

g.version -rge
version=7.8.dev
date=2019
revision=726b55ea2
build_date=2019-09-06
build_platform=x86_64-pc-linux-gnu
build_off_t_size=8
libgis_revision=00000
libgis_date="?"
proj4=5.2.0
gdal=2.3.2
geos=3.7.1
sqlite=3.26.0


bash test_framework_GRASS_GIS_with_NC.sh test_framework_GRASS_GIS_with_NC.conf 
--2019-09-06 22:11:34--  http://fatra.cnr.ncsu.edu/data/nc_spm_full_v2alpha.tar.gz
Resolving fatra.cnr.ncsu.edu (fatra.cnr.ncsu.edu)... 152.1.72.43
Connecting to fatra.cnr.ncsu.edu (fatra.cnr.ncsu.edu)|152.1.72.43|:80... connected.
HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable

    The file is already fully retrieved; nothing to do.

+ echo 'Testing of GRASS GIS started: 2019-09-06-20-11'
+ '[' no = yes ']'
+ cd testreports/reports_for_date-2019-09-06-20-11
+ python /home/mneteler/software/grass78_git/lib/python/gunittest/multirunner.py --grassbin /home/mneteler/software/grass78_git/bin.x86_64-pc-linux-gnu/grass78 --grasssrc /home/mneteler/software/grass78_git --grassdata /home/mneteler/grassdata --location nc_spm_full_v2alpha --location-type nc
test_v_out_lidar from ./vector/v.out.lidar failed (2 tests failed)
decimation_test from ./vector/v.in.lidar failed (1 test failed)
test_v_in_lidar_filter from ./vector/v.in.lidar failed (1 test failed)
test_v_in_lidar_basic from ./vector/v.in.lidar failed (1 test failed)
mask_test from ./vector/v.in.lidar failed (1 test failed)
test_v_in_pdal_basic from ./vector/v.in.pdal failed (1 test failed)
test_v_in_pdal_filter from ./vector/v.in.pdal failed (1 test failed)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/main.py", line 178, in <module>
    sys.exit(main())
  File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/main.py", line 174, in main
    results_dir=results_dir)
  File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/invoker.py", line 271, in run_in_location
    gisdbase=gisdbase, location=location)
  File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/invoker.py", line 218, in _run_test_module
    stdout_file.write(stdout)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 596-597: ordinal not in range(128)
+ export PYTHONPATH=/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python:
+ PYTHONPATH=/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python:
+ python /home/mneteler/software/grass78_git/lib/python/gunittest/multireport.py --output summary_report 'reports_for_date-*/*'
Traceback (most recent call last):
  File "/home/mneteler/software/grass78_git/lib/python/gunittest/multireport.py", line 28, in <module>
    from grass.gunittest.checkers import text_to_keyvalue
  File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python/grass/__init__.py", line 21, in <module>
    _LOCALE_DIR = os.path.join(os.getenv("GISBASE"), 'locale')
  File "/usr/lib64/python2.7/posixpath.py", line 70, in join
    elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'

[neteler]

Ticket retargeted after milestone closed

[neteler]

Ticket retargeted after milestone closed

[neteler]

Ticket retargeted after milestone closed