Mapbox vector tile output support via ST_AsMVT

[Björn Harrtell]

Implementation and details here: ​https://git.osgeo.org/gogs/postgis/postgis/pulls/5

[Björn Harrtell]

Agh forgot to fix link and cannot change ticket description, should be ​https://git.osgeo.org/gogs/postgis/postgis/pulls/5.

[robe]

Fixed link

[strk]

In 15304:

Mapbox vector tile output support via ST_AsMVT

Implementation by Björn Harrtell / CARTO

References #3661

[strk]

Merged, thank you ! r15304 = fee914f1174c1ab6869fbc9e23634fcd68a8455c (refs/remotes/trunk)

[robe]

I'm not sure if this is an issue with configure logic or debbie. After the protobuf commit r15304 getting this

checking for shared library run path origin... /bin/bash: ./config.rpath: No such file or directory
done
checking for CFPreferencesCopyAppValue... no
checking for CFLocaleCopyCurrent... no
checking for GNU gettext in libc... yes
checking whether to use NLS... yes
checking where the gettext function comes from... libc
checking proj_api.h usability... yes
checking proj_api.h presence... yes
checking for proj_api.h... yes
checking for pj_get_release in -lproj... yes
checking json/json.h usability... no
checking json/json.h presence... no
checking for json/json.h... no
checking json-c/json.h usability... yes
checking json-c/json.h presence... yes
checking for json-c/json.h... yes
checking for json_object_get in -ljson-c... yes
./configure: line 16565: syntax error near unexpected token `PROTOBUFC,'
./configure: line 16565: `PKG_CHECK_MODULES(PROTOBUFC, libprotobuf-c >= 1.1.0, HAVE_PROTOBUF=yes, HAVE_PROTOBUF=no)'
Build step 'Execute shell' marked build as failure

But winnie seems fine though, and I don't have any protobuf installed on her.

[Björn Harrtell]

From what I can tell this problem occurs because of the use of PKG_CHECK_MODULES. I could not find another suitable method to inspect and require the specific version of libprotobuf-c which is the minimum required version that can handle the vector tile protobuf specification.

The specific (unhelpful) error message is given when pkg-config isn't installed.

I'd like to know debbie better, and perhaps also find an alternative to PKG_CHECK_MODULES, but for now would it be possible to add pkg-config to her? (​https://packages.debian.org/search?keywords=pkg-config)

[robe]

Know debbie better :) That made me laugh. She was a Debian 8, now she reads sid/stretchy. I must have accidentally upgraded her in one of my reckless drunken update frenzies.

Okay I did a

apt-get install pkg-config

on her and now rerunning the job. First set of runs have passed so I think we are good.

I'm okay with this dependency as long as we add it to the docs and ideally also check in configure if possible.

[robe]

I think this section needs to be updated as well to reflect where to get proto-buf and version needed and configure to use non-standard path.

​http://postgis.net/docs/manual-dev/postgis_installation.html#PGInstall

[robe]

Okay I installed following on debbie

 apt-get install libprotobuf-c-dev
 apt-get install libprotobuf-c1
 apt-get install libprotoc-dev

seems I'm still missing something cause I get this

checking protobuf-c/protobuf-c.h usability... yes
checking protobuf-c/protobuf-c.h presence... yes
checking for protobuf-c/protobuf-c.h... yes
checking for protobuf_c_message_check in -lprotobuf-c... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for PROTOBUFC... yes
checking for protoc-c... no
configure: WARNING: Protobuf compiler missing, disabling protobuf support.

What am I missing?

[Björn Harrtell]

I think you are missing ​https://packages.debian.org/jessie/protobuf-compiler.

I'll try to get to a follow up PR to clean up this dependency mess, or at least provide the documentation as you point out.

[strk]

I think a failed pkg-config should anyway just disable AsMVT functionality, not drop the whole build configuration

[robe]

Thanks

I've installed:

apt-get install protobuf-compiler

I'll rerun the job in a bit.

For windows (to install on winnie), I'm building that from scratch.

So far I have the below compiled and installed and Protobuf - 3.2.0 - ​https://github.com/google/protobuf

Protobuf-c v. 1.2.1 ​https://github.com/protobuf-c/protobuf-c

anything else I need, I haven't started to build postgis yet.

Does the PROTOBUFDIR configure variable assume that both protobuf and protobuf-c are installed in the same folder. Out of habit I installed them in separate folders.

[Björn Harrtell]

As far as I know protobuf (C++) is only needed for the C version of the protobuf compiler which is only required for building. In error, I pointed you to the C++ protobuf compiler before - the needed package is protobuf-c-compiler which depends on protobuf.

The configure variable only needs to point out protobuf-c. The configure scripts checks the PATH for existence of the protobuf compiler command (here ​https://github.com/postgis/postgis/blob/fee914f1174c1ab6869fbc9e23634fcd68a8455c/configure.ac#L947).

[strk]

In 15307:

Add check for pkg-config

Patch by Björn Harrtell

References #3661

[robe]

Thanks that explains why it still didn't work.

Okay I have done:

apt-get install protobuf-c-compiler

Which gave this, so I assume that's good enough

Need to get 78.4 kB of archives.
After this operation, 252 kB of additional disk space will be used.
Get:1 http://ftp.us.debian.org/debian/ sid/main protobuf-c-compiler amd64 1.2.1-2 [78.4 kB]
Fetched 78.4 kB in 0s (584 kB/s)
Selecting previously unselected package protobuf-c-compiler.
(Reading database ... 103526 files and directories currently installed.)
Preparing to unpack .../protobuf-c-compiler_1.2.1-2_amd64.deb ...
Unpacking protobuf-c-compiler (1.2.1-2) ...
Setting up protobuf-c-compiler (1.2.1-2) ...

[robe]

Success, debbie is now building with protobuf-c

 -------------- Dependencies -------------- 
  GEOS config:          /var/lib/jenkins/workspace/geos/rel-3.7.0devw64/bin/geos-config
  GEOS version:         3.7.0dev
  GDAL config:          /var/lib/jenkins/workspace/gdal/rel-2.1w64/bin/gdal-config
  GDAL version:         2.1.3
  SFCGAL config:        /usr/bin/sfcgal-config
  SFCGAL version:       1.3.0
  PostgreSQL config:    /var/lib/jenkins/workspace/pg/rel/pg9.6w64/bin/pg_config
  PostgreSQL version:   PostgreSQL 9.6.2
  PROJ4 version:        49
  Libxml2 config:       /usr/bin/xml2-config
  Libxml2 version:      2.9.4
  JSON-C support:       yes
  protobuf-c support:   yes
  PCRE support:         yes
  PostGIS debug level:  0
  Perl:                 /usr/bin/perl

 --------------- Extensions --------------- 

Now to work on winnie.

[aschardong]

Awesome! I know you mention that you gonna work on winnie. Just a a heads up, when I run the function it's showing the message that protobuflib is missing. Is there a pre-build library for windows that I could just copy to the bin folder?

[robe]

It's not as simple as copying. The whole logic is disabled at compile time. I got side tracked this week, but hope to have this in place this weekend for winnie. I've already got the protobuf libs compiled under windows. I'll close update and close this ticket after I have the windows build going.

[robe]

I think we may not to work a little more on configure. I couldn't get protobuf to be picked up without, first having my protobuf-c in my include, lib folder of protobuf.

I also needed to include the path in my path settings. In path settings, configure picks up but it didn't compile, so I had to also add

--with-protobufdir=path/to/protobufandprotobufc 

I tried fussing with PKG_CONFIG_PATH doing

export PKG_CONFIG_PATH=path/to/protobuf/lib/pkgconfig

but that didn't seem to help things

[robe]

I take that back. Seems I also need the PKG_CONFIG_PATH thing to get it to pickup protobuf. Anyway I have winnie almost ready to start building with protobuf. I'll close this ticket once I have her setup and open another for investigating the configure headaches.

[robe]

In 15311:

Logic for winnie to build and package with protobuf-c support
references #3661

[robe]

In 15312:

forgot pkg_config_path and reverted CGAL version change
references #3661

[robe]

aschardong I have winnie building 2.4 with protobuf support now, Please give her experimental builds a try.

​http://postgis.net/windows_downloads/

I'm going to close this ticket now.

[flippmoke]

I noticed some issues with the code per the MVT specification and commented on them on the pull request.

​https://git.osgeo.org/gogs/postgis/postgis/pulls/5#issuecomment-2966

[strk]

Issues reported will be tracked by #3712