3.4.0-beta1 fails to build (postgis--TEMPLATED--TO--ANY.sql is missing)

[Bas Couwenberg]

The Debian package build for 3.4.0-beta1 failed to build:

---- Making install-extension-upgrades-from-known-versions in postgis
make[5]: Entering directory '/build/postgis-3.4.0~beta1+dfsg/extensions/postgis'
/usr/bin/perl ../../loader/postgis.pl \
        install-extension-upgrades \
        --extension postgis \
        --pg_sharedir /usr/share/postgresql/15 \
        2.0.0 2.0.1 2.0.2 2.0.3 2.0.4 2.0.5 2.0.6 2.0.7 2.1.0 2.1.1 2.1.2 2.1.3 2.1.4 2.1.5 2.1.6 2.1.7 2.1.8 2.1.9 2.2.0 2.2.1 2.2.2 2.2.3 2.2.4 2.2.5 2.2.6 2.2.7 2.2.8 2.3.0 2.3.1 2.3.2 2.3.3 2.3.4 2.3.5 2.3.6 2.3.7 2.3.8 2.3.9 2.3.10 2.3.11 2.4.0 2.4.1 2.4.2 2.4.3 2.4.4 2.4.5 2.4.6 2.4.7 2.4.8 2.4.9 2.4.10 2.5.0 2.5.1 2.5.2 2.5.3 2.5.4 2.5.5 2.5.6 2.5.7 2.5.8 2.5.9 3.0.0 3.0.1 3.0.2 3.0.3 3.0.4 3.0.5 3.0.6 3.0.7 3.0.8 3.0.9 3.1.0 3.1.1 3.1.2 3.1.3 3.1.4 3.1.5 3.1.9 3.1.6 3.1.7 3.1.8 3.2.0 3.2.1 3.2.2 3.2.3 3.2.4 3.2.5 3.3.0 3.3.1 3.3.2 3.3.3 3.3.4 3.4.0dev
File /usr/share/postgresql/15/extension/postgis--TEMPLATED--TO--ANY.sql is missing at ../../loader/postgis.pl line 88.
Installation target: /usr/share/postgresql/15 postgis
make[5]: *** [../upgrade-paths-rules.mk:28: install-extension-upgrades-from-known-versions] Error 2

The extension hasn't been installed yet, hence you should use the file in $DESTDIR:

/usr/bin/install -c -m 644 "sql/postgis--TEMPLATED--TO--ANY.sql" "/build/postgis-3.4.0~beta1+dfsg/debian/tmp/usr/share/postgresql/15/extension/postgis--TEMPLATED--TO--ANY.sql"; \

[Bas Couwenberg]

Use DESTDIR for pg_sharedir

[Bas Couwenberg]

The attached patch resolves the issue for the Debian package build.

[strk]

Thanks, pushed as commit [01d3914e97cd264cb3e93c8b79df32e9ca857569/git]

[Bas Couwenberg]

There is still a dependency problem which parallel builds sometimes trigger:

---- Making install in postgis
make[4]: Entering directory '/build/postgis/extensions/postgis'
/bin/mkdir -p '/build/postgis/debian/tmp/usr/share/postgresql/15/extension'
tpl='postgis--ANY--3.4.0beta1.sql'; \
/usr/bin/install -c -m 644 sql/${tpl} "/build/postgis/debian/tmp/usr/share/postgresql/15/extension/${tpl}"; \
/usr/bin/install -c -m 644 "sql/postgis--TEMPLATED--TO--ANY.sql" "/build/postgis/debian/tmp/usr/share/postgresql/15/extension/postgis--TEMPLATED--TO--ANY.sql"; \
ln -fs "postgis--TEMPLATED--TO--ANY.sql" /build/postgis/debian/tmp/usr/share/postgresql/15/extension/postgis--3.4.0beta1--ANY.sql;
/usr/bin/install: cannot create regular file '/build/postgis/debian/tmp/usr/share/postgresql/15/extension/postgis--ANY--3.4.0beta1.sql': No such file or directory
/usr/bin/install: cannot create regular file '/build/postgis/debian/tmp/usr/share/postgresql/15/extension/postgis--TEMPLATED--TO--ANY.sql': No such file or directory
ln: failed to create symbolic link '/build/postgis/debian/tmp/usr/share/postgresql/15/extension/postgis--3.4.0beta1--ANY.sql': No such file or directory
make[4]: *** [../upgrade-paths-rules.mk:22: install-upgrade-paths] Error 1

The install-upgrade-paths target does not depend on install-extension-upgrades-from-known-versions to ensure that postgis--TEMPLATED--TO--ANY.sql exists before it is symlinked.

[Bas Couwenberg]

The Debian package build on arm64 failed despite the patch for this issue:

---- Making install-extension-upgrades-from-known-versions in postgis
make[5]: Entering directory '/<<PKGBUILDDIR>>/extensions/postgis'
/usr/bin/perl ../../loader/postgis.pl \
	install-extension-upgrades \
	--extension postgis \
	--pg_sharedir /<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15 \
	2.0.0 2.0.1 2.0.2 2.0.3 2.0.4 2.0.5 2.0.6 2.0.7 2.1.0 2.1.1 2.1.2 2.1.3 2.1.4 2.1.5 2.1.6 2.1.7 2.1.8 2.1.9 2.2.0 2.2.1 2.2.2 2.2.3 2.2.4 2.2.5 2.2.6 2.2.7 2.2.8 2.3.0 2.3.1 2.3.2 2.3.3 2.3.4 2.3.5 2.3.6 2.3.7 2.3.8 2.3.9 2.3.10 2.3.11 2.4.0 2.4.1 2.4.2 2.4.3 2.4.4 2.4.5 2.4.6 2.4.7 2.4.8 2.4.9 2.4.10 2.5.0 2.5.1 2.5.2 2.5.3 2.5.4 2.5.5 2.5.6 2.5.7 2.5.8 2.5.9 3.0.0 3.0.1 3.0.2 3.0.3 3.0.4 3.0.5 3.0.6 3.0.7 3.0.8 3.0.9 3.1.0 3.1.1 3.1.2 3.1.3 3.1.4 3.1.5 3.1.9 3.1.6 3.1.7 3.1.8 3.2.0 3.2.1 3.2.2 3.2.3 3.2.4 3.2.5 3.3.0 3.3.1 3.3.2 3.3.3 3.3.4 3.4.0dev
File /<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/postgis--TEMPLATED--TO--ANY.sql is missing at ../../loader/postgis.pl line 88.
Installation target: /<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15 postgis
make[5]: *** [../upgrade-paths-rules.mk:28: install-extension-upgrades-from-known-versions] Error 2

Full buildlog: ​https://buildd.debian.org/status/fetch.php?pkg=postgis&arch=arm64&ver=3.4.0%7Ebeta1%2Bdfsg-1%7Eexp4&stamp=1689487120&raw=0

[strk]

I don't think install targets should depend on other install targets, as those targets are phony so will run again if invoked again.

From the logs it is clear that the install fo the TEMPLATED--TO--ANY file is done before the attempt to symlink it.

Is <<PKGBUILDDIR>> the actual directory name or some string replacement before printing logs ?

The "is missing" message is emitted by perl's -f operator here: And then not found by the -f perl operator here: ​https://git.osgeo.org/gitea/postgis/postgis/src/tag/3.4.0beta1/loader/postgis.pl#L89

[Bas Couwenberg]

Is <<PKGBUILDDIR>> the actual directory name or some string replacement before printing logs ?

It's a replacement.

From the buildlog:

I: NOTICE: Log filtering will replace 'build/postgis-o8Pezi/postgis-3.4.0~beta1+dfsg' with '<<PKGBUILDDIR>>'
I: NOTICE: Log filtering will replace 'build/postgis-o8Pezi' with '<<BUILDDIR>>'

So the absolute path is:

/build/postgis-o8Pezi/postgis-3.4.0~beta1+dfsg/debian/tmp/usr/share/postgresql/15

[strk]

Closer inspection of those logs reveal a problem occurred at make install time, but didn't interrupt make:

---- Making install in postgis
make[4]: Entering directory '/<<PKGBUILDDIR>>/extensions/postgis'
/bin/mkdir -p '/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension'
tpl='postgis--ANY--3.4.0beta1.sql'; \
/usr/bin/install -c -m 644 sql/${tpl} "/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/${tpl}"; \
/usr/bin/install -c -m 644 "sql/postgis--TEMPLATED--TO--ANY.sql" "/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/postgis--TEMPLATED--TO--ANY.sql"; \
ln -fs "postgis--TEMPLATED--TO--ANY.sql" /<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/postgis--3.4.0beta1--ANY.sql;
/usr/bin/install: cannot create regular file '/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/postgis--ANY--3.4.0beta1.sql': No such file or directory
/usr/bin/install: cannot create regular file '/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/postgis--TEMPLATED--TO--ANY.sql': No such file or directory
/bin/mkdir -p '/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension'

[strk]

I don't understand why the target directory is found to be unexistent as it was craeted as the first thing:

---- Making install in postgis
make[4]: Entering directory '/<<PKGBUILDDIR>>/extensions/postgis'
/bin/mkdir -p '/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension'
tpl='postgis--ANY--3.4.0beta1.sql'; \
/usr/bin/install -c -m 644 sql/${tpl} "/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/${tpl}"; \
/usr/bin/install -c -m 644 "sql/postgis--TEMPLATED--TO--ANY.sql" "/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/postgis--TEMPLATED--TO--ANY.sql"; \
ln -fs "postgis--TEMPLATED--TO--ANY.sql" /<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/postgis--3.4.0beta1--ANY.sql;
/usr/bin/install: cannot create regular file '/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/postgis--ANY--3.4.0beta1.sql': No such file or directory
/usr/bin/install: cannot create regular file '/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/postgis--TEMPLATED--TO--ANY.sql': No such file or directory
/bin/mkdir -p '/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension'
/usr/bin/install -c -m 644 .//postgis.control '/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/'
/usr/bin/install -c -m 644  postgis.control sql/postgis--3.4.0beta1.sql sql/postgis--unpackaged.sql sql/postgis--unpackaged--3.4.0beta1.sql sql/postgis--ANY--3.4.0beta1.sql  '/<<PKGBUILDDIR>>/debian/tmp/usr/share/postgresql/15/extension/'
make[4]: Leaving directory '/<<PKGBUILDDIR>>/extensions/postgis'

It's also created twice for some reason

[strk]

In any case I think this dependency is encoded wrong:

install: install-upgrade-paths

As it's really the other way around, but as we're using PGXS that construct was the easiest to attach something to the "install" pre-defined target. We'd need a way to run something *after* the predefined install target, I guess.

[strk]

Bas can you check if this fixes the issue for you ?

diff --git a/extensions/upgrade-paths-rules.mk b/extensions/upgrade-paths-rules.mk
index e543db61c..93e12cdfa 100644
--- a/extensions/upgrade-paths-rules.mk
+++ b/extensions/upgrade-paths-rules.mk
@@ -18,7 +18,7 @@ install: install-upgrade-paths
 #   ALTER EXTENSION postgis UPDATE TO 'ANY';
 #   ALTER EXTENSION postgis UPDATE;
 #
-install-upgrade-paths: tag-as-any
+install-upgrade-paths: installdirs tag-as-any
        tpl='$(EXTENSION)--ANY--$(EXTVERSION).sql'; \
        $(INSTALL_DATA) sql/$${tpl} "$(EXTDIR)/$${tpl}"; \
        $(INSTALL_DATA) "sql/$(TAG_UPGRADE)" "$(EXTDIR)/$(TAG_UPGRADE)"; \

[Sandro Santilli <strk@…>]

In dc3b6d1/git:

Improve robustness of install-upgrade-paths Makefile target

• Do not hide errors on install
• Improve comment
• Ensure target dir exists before installing to it

References #5444

[strk]

I've added an explicit mkdir call in that target, as of [dc3b6d15396404abdfc61891c68f5eec0105ec0b/git] it should be good, let me know!

[Bas Couwenberg]

Added the patch from comment:10 in postgis (3.4.0~beta1+dfsg-1~exp6) and uploaded it to experimental.

Since this is a hard to reproduce flaky issue, the successful builds are no guarantee. The most recent postgis (3.4.0~beta1+dfsg-1~exp5) builds without the patch also succeeded.

[strk]

exp7 seems all green: ​https://buildd.debian.org/status/logs.php?pkg=postgis&ver=3.4.0~beta1%2Bdfsg-1~exp7 can we close this ticket now ?

[Bas Couwenberg]

Replying to strk:

exp7 seems all green: ​https://buildd.debian.org/status/logs.php?pkg=postgis&ver=3.4.0~beta1%2Bdfsg-1~exp7 can we close this ticket now ?

This URL is better:

​https://buildd.debian.org/status/package.php?p=postgis&suite=experimental

It shows that the amd64, i386, mips64el, and mipsel builds are still pending.

Since this is a hard to reproduce flaky issue, any successful builds are no guarantee. You can close this issue if you think it's definitely fixed now, I can reopen it again when failing builds prove otherwise.

[robe]

I'm going to close out. The link above shows all green. We can reopen if still an issue.