FDO RFC 21 - New Linux Buildsystem Cmake Based

This page contains a request for comments document (RFC) for the FDO Open Source project. More FDO RFCs can be found on the RFCs page.

Status

RFC Template Version	(1.0)
Submission Date	May 20, 2008
Last Modified	Helio Castro Timestamp
Author	Helio Castro
RFC Status	Draft
Implementation Status	Under Development
Proposed Milestone	3.3.x
Assigned PSC guide(s)
Voting History	(vote date)

Overview

On the current status of FDO over Linux and *nix platforms there's a huge need to rely on third party included sources, as well all providers can't be independently built, depending of whole infrastructure of core FDO sources to be available. This proposal is to move whole builds for a modern text based, human readble buildsystem, suitable to even become windows buildsystem in future.

Motivation

FDO is suppose to become *the* standard for map manipulation on opensource, but current automake,autoconf linux and *nix buildsystem together with the difficulty of rely in private headers and very old versions of some libraries, like cppunit, make almost impossible to provide it as a consistent package for distribution. The situation is workst when we deal that in many cases static and shared libraries are been linked together, and compilers like gcc sometimes not play so nice with this ( really ).

This creates a complication over the present distributions situations, as some libraries present conflicts with the thirdparty ones provided, and make FDO opensource not widespread available on major distributions.

• Question from Greg Boone: The purpose of including the Thirdparty directory is to simplify the user’s development experience so that he/she do not have to determine which versions/builds of the relevant thirdparty software are supported. By including a version of a thirdparty component in our SVN, we guarantee that our software works with that version. How do you propose to handle the cases where a user a) does not want to have to figure out which version (or if his version) is supported or b) that the version he uses runs but is not 100% compatible, resulting in runtime errors, loss of functionality, etc?
  • Is still be possible to have a ThirdParty directory as a convenience and/or make recomendation about which is the prefered/recommended versions by FDO team. Providing a thirdparty dir indeed relieve the user of need to discover what is happening, but you are narrowing the possibility of deploying this widespread. Is quite common as an example the distro have one only db devel version, and forcing user to choose a different one and add on your system without any update/security updates garanteed is complicated. For an shor answer to a), provides a link to the version recomended on documentation and a build script for everything and for b) quite the same, except that now we would be able to create a database of results, not stucked on old somtimes not so safe code, even been working.

The whole motivation behind the change is make FDO not touch any kind of internal thirdparty sources, using only provided distribution libraries, and make it easy to compile and work with core or providers separated. CMake is one of the frameworks that allow us to do this.

With this, the possibilities of improvement are huge, as will become easy to deal with debugging/analysing independent parts of code, without rely on restart whole bs scripting.

• Question from Greg Boone: Can you elaborate on the meaning of this statement in greater detail and further document the expected improvements that will result? What does “bs scripting” refer to?
  • auto*tools are not easy to write more else to change. From the point of view of final user, they runs the configure and done. But from the poiunt of view of maintenance, is a very difficult and complicated issue. Any process that need test and change something in a makefile needs regenerate whole auto structure. More dangerous is that we need to rely in multiple different versions of auto*toos provided on distros. In this specific case, dictate which libtool, automake, autoconf version is the fdo prefered won't be possible, because this is too sensitive for distros. The nature of cmake is way differently, since the simple text based buildfile, in no complex language ( similar a standard shell script ), and the easy ability to do out of source builds help in the matters that we can compile without polute source dir and change specific points in realtime not need to depend on many tools, like m4, automake, autoconf, libtoolize, etc..

Proposed Solution

Most of this proposals presented here are in the READY TO DEPLOY state, unless some exceptions stated below

Get rid of whole auto*tools

• None of the current .am .in tools will be necessary anymore.

Get rid of ThirdParty tools

• With the new buildsystem, it will be suppose to use external shared only system libraries. The only present issue is about sqllite internal header dependency to solve.
• Common tools like mkcatdefs are splitted in an independent source standalone with their own cmake build.

Been able to compile any provider without need to be inside FDO main code"

• Providers compile itself, just requiring that fdo core lib and headers is installed and cmake knows where to look.
• Providers not need anymore been developed inside the fdo tree.
  • Question from Greg Boone: Can you describe this section in greater detail? How has this improvement been tested? What happens if no FDO is installed and the FDO binaries currently reside in their current build system locations?
    • In current setup, core fdo need be installed, but the old behavior, reside in their current build system locations, can be added with minimal work.

Rearrange fdo headers install

• To enable providers become an entity, some headers ignored before are now needed to be installed in their own include dir. New buildsystem do this automatically
  • Question from Greg Boone: Can you describe this in greater detail ? We have external applications which rely on the current install directory pattern.
    • This can be easily fixed, if needed ( i really believe that none was affected ), as i have list of applications and how are their requirements. My only main application as a test base is Mapguide

Create proper library naming/module

• Today fdo provides .so only files, mixing devel and not devel symbols. FDO core is moved to have prober soname and devel .so. Same to Providers, but further study can make providers to become module only, not needing rely anymore to become .so.*
  • Question from Greg Boone: Can you describe this in greater detail? We have applications in production which rely on the current library names.
    • The providers xml file is already pointing for new libraries. In case of build this application against the new build mode, the application wil automatically depends on new library naming. The .so naming was not changed, we just added the proper split/not split devel mode, which mean that we still have the old library naming as a convenience. But talking claerly, is not expected that old papplications built against old fdo build mode will just upgrade as is if non proper compilation methods are used.

Been able to install in any directory withou harm applications using it

• New buildsystem do automatically the necessary changes in files to provide access doesn't matter the dir user decides to install. The original place was /usr/local/fdo-<version>
  • Question from Greg Boone: Installation into a user specified directory is already supported as long as the $FDOHOME environment variable is set. Even if the cmake system is adopted, setting $FDOHOME will still be required. What level of testing has been done to validate enhanced install location support?
    • The current test is during config see if INSTALL_DIR is set. Adding FDOHOME is doable easily too, so no changes in this matter. But setting FDOHOME is no more required at all, becomes optional.

Rewrite SQLiteInterface utility

• Current sqlite interface utility is dealing with internal sqlite headers. Sdf provider needs that. Mapguide not compiles without Sdf providers ( suggests to create a test to compile or not on mapguide ). To make it feasible, a rewrite is needed to use only standard sqlite headers.
  • Question from Greg Boone: If MapGuide needs SDF, then how can a solution be implemented that involves not compiling SDF ? Also, if a rewrite is required, that must be explicitly spelled out in this document.
    • The intention is have SDF enabled. And yes, mapguide can be build without sdf or shp part, just adding proper testing if SDF or SHP provider is available ( easy task ) and set the define on build. This is really not an issue.

Test Plan

• Patches are ready to be provided against fdo-3.3.1 code and tested.
• The new buildsystem can be introduced an commited in current trunk tree without affect any of current development and touching old files for now. 100% safe transition.

Issues

• Not all providers are converted. Is just manual work. Currently sdf, shp, postgis and gdal are done.
  • Question from Greg Boone: Are you proposing that all providers be converted before adopting such a system or only a subset?
    • All providers, is a matter of a day to do the work.

• Still need to rely in private sources ( sqlite )
• Not tested in many linux distros ( did on Mandriva, Suse and RH/Fedora )
• Not tested in *nix ( did on OpenSolaris build only )

Funding/Resources

Questions from FDO Team

• From Greg Boone: What source code changes are being proposed, if any ?
  • Minimal changes are did. A undef in LLONG_MAX/MIN in TestCommonConstraints.cpp to solve conflicts in linux/limits.h, A ifdef in RegistryUtility.cpp to test FDO_LIBRARY_DIR and been able to load library in any user desired install, instead of using the ugly hardcoded "/usr/local/fdo-3.3.0/lib/" ( even in 3.3.1 ) and a int/long changee to x86_64 in ClientServices/prefix.cpp

    #if defined(__x86_64__)
                   if (((unsigned long) symbol) >= start && ((unsigned long) symbol) < end)
    #else
                    if (((unsigned int) symbol) >= start && ((unsigned int) symbol) < end)
    #endif
    

• From Greg Boone: What installed content are being proposed as a result of the new build system? Can we get a comparison of the before/after result of building a full version of FDO and all it’s providers? What changes are applications expected to make in order to use the installed output?
  • The installed content is just more headers and libraries that are only static in past and now are installed and compiled as shared. I worked to not change anything as possible, just adding the necessary extra headers to enable providers to compile outside. I will provide an diff as soon as possible.

• From Greg Boone: Can you give a brief outline of the source/build files that will be added/removed/modified in the SVN ?
  • Added
    • Subdirectory cmake with test files
    • CMakeLists.txt files for every subdirectory ( all txt files )
  • Modified:
    • ClientServices/prefix.cpp
    • TestCommonConstraints.cpp
    • RegistryUtility.cpp
  • Removed ( but not necessary during the transition ):
    • auto*tools related

• From Greg Boone: Will the build_linux.sh files be maintained ?
  • Yes, if necessary. Now is way easier to build without the need of build_linux.sh. This give a wonderfull idea to add the build of thirdparty already on main cmakew scripts ( easy step too ) and just add simple instructions on documentation to build

• From Greg Boone: Who will document the new build system for our users ?
  • The poor soul that write this own RFC ? :-)

• From Greg Boone: On which Linux systems will the new build system be officially tested ?
  • From my side, i work for Mandriva as lead KDE team developer and core distro developer. I'm able to test ( and often have personal contact with some developers ) with Suse team and Red Hat team, so this can be what i can propose, Mandriva, Suse and RedHat/Fedora. I know that Ubuntu is a desired target too, a know personally some people working there and can try persuade some people to help, but my deep knowledge relied on Mandriva, Suse and Red Hat.

• From Greg Boone: How are Windows .vcproj and .sln files generated from the cmake files ? How can custom .vcproj actions currently defined in the .vcproj files be captured by the cmake files ?
  • This can be answered by own cmake 2.6 documentation link ​Cmake Generators. You will be surprised how many project types are supported. A simple cmake -G <generator> and done. The bidirectional issue of the second part of your question, i don't have immediate answer for this.

• From Jason Birch: For instance, for user-installers of !FDO, does this allow for an easier build process across multiple Linux distributions, or even automated creation of installer packages (apt, yum, whatever)? For build managers, does it allow for the maintenance of Linux, VS2005, and VS2008 solutions in a central interface?
  • Yes, would be easy to even create packages, from the point of view of packagers.

• From Jason Birch: What would be the effect on individual developers; would they have to install additional dependencies (for cmake)? Is cmake available across multiple platforms?
  • cmake itself doesn't depend on many things. As been available for all platforms, even more than current supported for !FDO, linux distros have at least the 2.4.x version available, and windows and mac have their own installer packages ready.

• From Jason Birch: What kind of impact will this have on existing build procedures and end-user documentation for building FDO ?
  • For now, introiducing the cmake build as additional, will not impact in anything the current behavior. For end user documentation, see Greg's question above

• From Jason Birch: I'm personally interested in cpack... do you have any experience with using this to create packages/installers for Linux and Windows ?
  • I did some small profiling in cpack yesterday, and yes, cpack can create easily withou almost no effort a suitable package to intall in linux, a nsis based install on windows. I did not tested rpm or deb, but i know is possible. The only issue about usage of this ones, is that usually distributions rely on their own policies for spec files.

• From Greg Boone: How would a user build all of FDO, Utilities and Providers in the SVN ?
  • Simple steps for FDO core:
    • Create a build dir in any place ( really )
    • Enter on dir
    • type cmake <yourbuilddir>
    • cmake test if all dependencies exists
    • type make
    • type make install. make DESTDIR=/my/test/dir install if you want a different install dir with same tree ( this is what packagers do )

• From Greg Boone: How would a user build a user-specified subset of FDO, Utilities and Providers ?
  • For fdo and utilities, enter on build dir ( see above ) the specified directory and type make. For providers, unpack the provider in any place and follow the same procedure as above