FTBFS: r.mapcalc in trunk

[hamish]

Hi,

latest trunk:

"make -j8" on debian/stable, 64bit quad-core

<stdout>:1555: warning: 'yyunput' defined but not used
<stdout>:1598: warning: 'input' defined but not used
gcc -L/usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-gnu/lib -L/usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-gnu/lib -Wl,--export-dynamic -Wl,-rpath-link,/usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-gnu/lib   -o /usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-gnu/bin/r.mapcalc OBJ.x86_64-unknown-linux-gnu/check.o OBJ.x86_64-unknown-linux-gnu/column_shift.o OBJ.x86_64-unknown-linux-gnu/evaluate.o OBJ.x86_64-unknown-linux-gnu/expression.o OBJ.x86_64-unknown-linux-gnu/function.o OBJ.x86_64-unknown-linux-gnu/main.o OBJ.x86_64-unknown-linux-gnu/map.o OBJ.x86_64-unknown-linux-gnu/mapcalc.tab.o OBJ.x86_64-unknown-linux-gnu/mapcalc.yy.o OBJ.x86_64-unknown-linux-gnu/xabs.o OBJ.x86_64-unknown-linux-gnu/xacos.o OBJ.x86_64-unknown-linux-gnu/xadd.o OBJ.x86_64-unknown-linux-gnu/xand.o OBJ.x86_64-unknown-linux-gnu/xand2.o OBJ.x86_64-unknown-linux-gnu/xasin.o OBJ.x86_64-unknown-linux-gnu/xatan.o OBJ.x86_64-unknown-linux-gnu/xbitand.o OBJ.x86_64-unknown-linux-gnu/xbitnot.o OBJ.x86_64-unknown-linux-gnu/xbitor.o OBJ.x86_64-unknown-linux-gnu/xbitxor.o OBJ.x86_64-unknown-linux-gnu/xcoor.o OBJ.x86_64-unknown-linux-gnu/xcos.o OBJ.x86_64-unknown-linux-gnu/xdiv.o OBJ.x86_64-unknown-linux-gnu/xdouble.o OBJ.x86_64-unknown-linux-gnu/xeq.o OBJ.x86_64-unknown-linux-gnu/xeval.o OBJ.x86_64-unknown-linux-gnu/xexp.o OBJ.x86_64-unknown-linux-gnu/xfloat.o OBJ.x86_64-unknown-linux-gnu/xge.o OBJ.x86_64-unknown-linux-gnu/xgraph.o OBJ.x86_64-unknown-linux-gnu/xgt.o OBJ.x86_64-unknown-linux-gnu/xif.o OBJ.x86_64-unknown-linux-gnu/xint.o OBJ.x86_64-unknown-linux-gnu/xisnull.o OBJ.x86_64-unknown-linux-gnu/xle.o OBJ.x86_64-unknown-linux-gnu/xlog.o OBJ.x86_64-unknown-linux-gnu/xlt.o OBJ.x86_64-unknown-linux-gnu/xmax.o OBJ.x86_64-unknown-linux-gnu/xmedian.o OBJ.x86_64-unknown-linux-gnu/xmin.o OBJ.x86_64-unknown-linux-gnu/xmod.o OBJ.x86_64-unknown-linux-gnu/xmode.o OBJ.x86_64-unknown-linux-gnu/xmul.o OBJ.x86_64-unknown-linux-gnu/xne.o OBJ.x86_64-unknown-linux-gnu/xneg.o OBJ.x86_64-unknown-linux-gnu/xnot.o OBJ.x86_64-unknown-linux-gnu/xnull.o OBJ.x86_64-unknown-linux-gnu/xor.o OBJ.x86_64-unknown-linux-gnu/xor2.o OBJ.x86_64-unknown-linux-gnu/xpow.o OBJ.x86_64-unknown-linux-gnu/xrand.o OBJ.x86_64-unknown-linux-gnu/xres.o OBJ.x86_64-unknown-linux-gnu/xround.o OBJ.x86_64-unknown-linux-gnu/xrowcol.o OBJ.x86_64-unknown-linux-gnu/xshiftl.o OBJ.x86_64-unknown-linux-gnu/xshiftr.o OBJ.x86_64-unknown-linux-gnu/xshiftru.o OBJ.x86_64-unknown-linux-gnu/xsin.o OBJ.x86_64-unknown-linux-gnu/xsqrt.o OBJ.x86_64-unknown-linux-gnu/xsub.o OBJ.x86_64-unknown-linux-gnu/xtan.o  -lgrass_gis -lgrass_raster -lgrass_btree  -lreadline  -lhistory   -lpthread   -lm

/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.tab.c:1442: undefined reference to `yylex'
OBJ.x86_64-unknown-linux-gnu/mapcalc.tab.o: In function `parse_string':
/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.y:270: undefined reference to `initialize_scanner_string'
OBJ.x86_64-unknown-linux-gnu/mapcalc.tab.o: In function `parse_stream':
/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.y:278: undefined reference to `initialize_scanner_stream'
collect2: ld returned 1 exit status
make[3]: *** [/usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-gnu/bin/r.mapcalc] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory `/usr/local/src/grass/svn/trunk/raster/r.mapcalc'

Hamish

[hamish]

after cd r.mapcalc and make clean I can build it successfully. so a makefile dep problem I guess.

also, I noticed this whiz by:

mapcalc.l:216: warning, -s option given but default rule can be matched

Hamish

[glynn]

Replying to hamish:

latest trunk:

"make -j8" on debian/stable, 64bit quad-core

gcc ... OBJ.x86_64-unknown-linux-gnu/mapcalc.yy.o ...
...
/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.tab.c:1442: undefined reference to `yylex'
...
/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.y:270: undefined reference to `initialize_scanner_string'
...

These should be in mapcalc.yy.o, compiled from mapcalc.yy.c, generated by lex from mapcalc.l.

mapcalc.yy.o is in the list of object files being linked, so make knows that it needs to build it and believes that it has done so, and gcc isn't complaining about its absence. Can you provide the make output corresponding to the generation and compilation of mapcalc.yy.c?

[glynn]

Replying to hamish:

also, I noticed this whiz by:

mapcalc.l:216: warning, -s option given but default rule can be matched

Harmless. It indicates that not all possible inputs are valid, i.e. it's possible for the expression to have syntax errors. Using the -s flag causes syntax errors to generate an error rather than silently writing unmatched text to stdout.

[hamish]

Replying to glynn:

These should be in mapcalc.yy.o, compiled from mapcalc.yy.c, generated by lex from mapcalc.l.

mapcalc.yy.o is in the list of object files being linked, so make knows that it needs to build it and believes that it has done so, and gcc isn't complaining about its absence. Can you provide the make output corresponding to the generation and compilation of mapcalc.yy.c?

sure. due to the -j8 the output is a bit jumbled up so I'll send you the entire build log off-list. I just tried again today and it built ok, but I guess who wins the race is just a whim of timing.

Hamish

[glynn]

Replying to hamish:

build log with faulty r.mapcalc build attached.

Looking at that...

3175:	make[4]: Entering directory `/home/hbowman/dev/grass/svn/trunk/raster/r.colors'
3176:	make -C ../r.mapcalc
...
3178:	make[5]: Entering directory `/home/hbowman/dev/grass/svn/trunk/raster/r.mapcalc'
...
...
3569:	make -C r.mapcalc || echo /home/hbowman/dev/grass/svn/trunk/raster/r.mapcalc >> /home/hbowman/dev/grass/svn/trunk/error.log
...
3577:	make[3]: Entering directory `/home/hbowman/dev/grass/svn/trunk/raster/r.mapcalc'
...
3589:	gcc ... -o OBJ.x86_64-unknown-linux-gnu/mapcalc.yy.o -c mapcalc.yy.c
...
3796:	gcc ... -o OBJ.x86_64-unknown-linux-gnu/mapcalc.yy.o -c mapcalc.yy.c

IOW, it's compiling it twice.

Note: r.colors requires r.mapcalc for the thumbnails.

It appears that the raster -> r.mapcalc build starts while the raster -> r.colors -> r.mapcalc build is still ongoing. The first build compiles mapcalc.yy.c to mapcalc.yy.o, then later tries to link it while the second build is compiling it, and ends up using an incomplete file.

Unrelated parallel make tasks don't communicate with each other (if they did, the second task would note that the first task was already building mapcalc.yy.o, and just wait for it to finish).

When raster/Makefile "make"s r.colors and r.mapcalc concurrently, the tasks are considered unrelated, and that doesn't change when they bump into each other in r.mapcalc.

One option is to simply remove r.mapcalc from the SUBDIRS list in raster/Makefile, as r.colors is going to build it. At least, that was my first thought, but if r.colors.html is up-to-date, it won't happen; and, if r.mapcalc already exists, it won't be re-made.

Another option is to move the thumbnail generation into e.g. man/Makefile. But that won't re-build the thumbnails if r.colors is re-made (OTOH, the thumbnail images are really a derivative of lib/gis/colors/ rather than r.colors per se).

Another option is to change raster/Makefile thus:

-htmldir: parsubdirs 
+htmldir:
	$(MAKE) -C r.mapcalc
	$(MAKE) parsubdirs

BTW, given that there are 200 compilation and 40 linking commands occurring in the interval in question, I'm quite surprised that you managed to actually trigger this error.

[hamish]

BTW, given that there are 200 compilation and 40 linking commands occurring in the interval in question, I'm quite surprised that you managed to actually trigger this error.

I'm not, I've always had this natural ability to cause computers to crash in strange and interesting ways.

... maybe the colorbar thumbnails python script is relatively slow to run versus compiling a dozen C modules? (for one thing it has to start python which can take a little while..) (the entire grass compile is done in less than 2 minutes on that machine if I give it -j8)

Hamish

[neteler]

Untouched for 4 years, still an issue?

[hamish]

In the past I was able to trigger this with regularity. For now it seems to be working on a similar machine but with an upgraded version of the linux distro. AFAIK it was never explicitly fixed, although it may have been incidentally fixed, so closing the ticket but will keep an eye out for it in case it comes back.

Hamish

[glynn]

Replying to neteler:

Untouched for 4 years, still an issue?

I believe so.

It's caused by a race condition, so it's a matter of chance whether it actually happens. The probability increases with the number of threads.

FWIW, the use of the "-s" flag was eliminated by r55504.

[neteler]

What is the state of this ticket?

[neteler]

Ticket retargeted after milestone closed

[neteler]

Ticket retargeted after 7.0.3 milestone closed

[martinl]

No activity, closing, feel free to reopen if needed.