| | 1 | |
| | 2 | = !MapGuide RFC 149 - Lightweight MapGuide Source Tree = |
| | 3 | |
| | 4 | This page contains a change request (RFC) for the !MapGuide Open Source project. |
| | 5 | More !MapGuide RFCs can be found on the [wiki:MapGuideRfcs RFCs] page. |
| | 6 | |
| | 7 | == Status == |
| | 8 | |
| | 9 | ||RFC Template Version||(1.0)|| |
| | 10 | ||Submission Date||28 May 2015|| |
| | 11 | ||Last Modified|||| |
| | 12 | ||Author||Jackie Ng|| |
| | 13 | ||RFC Status||draft|| |
| | 14 | ||Implementation Status||pending|| |
| | 15 | ||Proposed Milestone||3.1|| |
| | 16 | ||Assigned PSC guide(s)||(when determined)|| |
| | 17 | ||'''Voting History'''||(vote date)|| |
| | 18 | ||+1|||| |
| | 19 | ||+0|||| |
| | 20 | ||-0|||| |
| | 21 | ||-1|||| |
| | 22 | ||no vote|| || |
| | 23 | |
| | 24 | == Overview == |
| | 25 | |
| | 26 | This RFC proposes to separate OEM from the MgDev source tree, with some major structural changes to the build system on Windows and Linux to reduce svn checkout footprint and to simplify/speed up the MapGuide build process |
| | 27 | |
| | 28 | == Motivation == |
| | 29 | |
| | 30 | The MapGuide OEM tree is something that rarely changes between MapGuide releases, except when a certain component needs to be upgraded to a newer version. |
| | 31 | |
| | 32 | Having this included with the main MapGuide source adds unnecessary overheads to our build process and workflows: |
| | 33 | |
| | 34 | * Initial checkout is *expensive*. The OEM source tree currently weighs in at 1.9GB of source code. |
| | 35 | * Working copy operations (switch, commit, revert) also are slow due to having to process the OEM directory that never changes. |
| | 36 | * A major fraction of the overall build time in MapGuide is spent building OEM components that do not change. |
| | 37 | |
| | 38 | By separating OEM out of the MgDev tree, we have |
| | 39 | |
| | 40 | * An OEM source tree that '''only needs to be built once''' per release branch of MapGuide |
| | 41 | * OEM component updates can happen independently of the MapGuide source |
| | 42 | * Initial checkout time of the MapGuide source is significantly reduced as it will no longer contain OEM component source. Combined with a pre-built OEM components for that particular release of MapGuide. The main build of MapGuide itself will be significantly faster, lending itself to the feasibility of having MapGuide under *continuous integration and delivery* whether through an in-house CI/CD system or through 3rd party services like Travis/AppVeyor/etc. |
| | 43 | * Lends itself to allowing our OEM components to be replaced with system-installed versions when building for Linux |
| | 44 | |
| | 45 | == Solution == |
| | 46 | |
| | 47 | The Oem tree will be moved up one level to become a sibling of MgDev |
| | 48 | |
| | 49 | A new build script will be added to Oem that will build all Oem compoents and package the resulting headers/libs/dlls into a "buildpack" archive. To keep this archive as small as possible, anything that is not a header, import library or dll (so's in Linux) is omitted. |
| | 50 | |
| | 51 | A new script will be added to Oem that will set the appropriate Oem component include/linker paths as environment variables. This will be bundled with the "buildpack" archive. |
| | 52 | |
| | 53 | The existing Server/Web/Desktop projects will be modified to use our Oem component environment variables. |
| | 54 | |
| | 55 | On Linux, the main configure script will be modified to accept individual `--with` options for specific oem component paths, along with a general `--with-oem` option which when pointed to the root path of an oem buildpack will automatically set the paths of all required oem components. All existing Makefile.am files will be modified to use parameterized include/linker paths based on the new OEM component variables defined by the configure script. |
| | 56 | |
| | 57 | Where possible and convenience, we no longer keep binaries of key dependencies and opt for a download-and-cache approach. Such dependencies include: |
| | 58 | |
| | 59 | * PHP: If required we can download the official distribution from the PHP windows download site. |
| | 60 | * Apache HTTPD: This is an apachelounge distribution, but apachelounge do not maintain download archives. We will use our OSGeo download location as a local mirror for this purpose. |
| | 61 | |
| | 62 | This will reduce the checkout footprint of the installer components. |
| | 63 | |
| | 64 | The new build workflows that are possible under this scheme are outlined as follows |
| | 65 | |
| | 66 | === Workflow 1: Full build from fresh checkout (headless) === |
| | 67 | |
| | 68 | This is the longest as it requires a full checkout and build of both MapGuide and Oem (ie. What we currently do) |
| | 69 | |
| | 70 | 1. Checkout $SVN_ROOT/MgDev to $MG_SRC |
| | 71 | 2. Checkout $SVN_ROOT/Oem to $MG_OEM_SRC |
| | 72 | 3. Checkout $SVN_ROOT/Installer to $MG_INST_SRC |
| | 73 | 4. Run $MG_OEM_SRC/build.bat to compile the Oem tree for $ARCH and $CONFIG, which produces a mapguide-buildpack-$VERSION-$ARCH-$CONFIG archive |
| | 74 | 5. Extract mapguide-buildpack-$VERSION-$ARCH-$PLATFORM archive to $MG_OEM_LIBS |
| | 75 | 6. Run $MG_OEM_LIBS/setenvironment $ARCH $CONFIG to set OEM component environment variables |
| | 76 | 7. Run $MG_SRC/setenviroment[64].bat to set additional environment variables |
| | 77 | 8. Run $MG_SRC/build.bat to build to the MapGuide source |
| | 78 | 9. Run $MG_SRC/build.bat -a=install to build the MapGuide source and copy binaries to $MG_STAGING_AREA |
| | 79 | 10. Run $MG_INST_SRC/build.bat -a=prepare to add additional binaries to $MG_STAGING_AREA |
| | 80 | 11. Run $MG_INST_SRC/build.bat -a=generate to generate the installer file listing $MG_STAGING_AREA |
| | 81 | 12. Run $MG_INST_SRC/build.bat to build the windows installer using the files from $MG_STAGING_AREA |
| | 82 | |
| | 83 | === Workflow 2: IDE build/debug from fresh checkout === |
| | 84 | |
| | 85 | 1. Checkout $SVN_ROOT/MgDev to $MG_SRC |
| | 86 | 2. Checkout $SVN_ROOT/Oem to $MG_OEM_SRC |
| | 87 | 3. Checkout $SVN_ROOT/Installer to $MG_INST_SRC |
| | 88 | 4. Run $MG_OEM_SRC/build.bat to compile the Oem tree for $ARCH and $CONFIG, which produces a mapguide-buildpack-$VERSION-$ARCH-$CONFIG archive |
| | 89 | 5. Extract mapguide-buildpack-$VERSION-$ARCH-$PLATFORM archive to $MG_OEM_LIBS |
| | 90 | 6. Run $MG_OEM_LIBS/setenvironment $ARCH $CONFIG to set OEM component environment variables |
| | 91 | 7. Run $MG_SRC/setenviroment[64].bat to set additional environment variables |
| | 92 | 8. Run Visual Studio (devenv) from that same command line session |
| | 93 | 9. Open server solution and build/debug in $CONFIG|$ARCH |
| | 94 | 10. Open web solution and build/debug in $CONFIG|$ARCH |
| | 95 | |
| | 96 | === Workflow 3: IDE build/debug with pre-built OEM components === |
| | 97 | |
| | 98 | Given a pre-existing mapguide-buildpack-$VERSION-$ARCH-$CONFIG archive and a FDO binary SDK of the correct bitness |
| | 99 | |
| | 100 | 1. Checkout $SVN_ROOT/MgDev to $MG_SRC |
| | 101 | 2. Extract mapguide-buildpack-$VERSION-$ARCH-$CONFIG archive to $MG_OEM_LIBS |
| | 102 | 3. Extract a pre-existing FDO binary SDK into the FDO directory in $MG_OEM_LIBS |
| | 103 | 4. Run $MG_OEM_LIBS/setenvironment $ARCH $CONFIG to set OEM component environment variables |
| | 104 | 5. Run $MG_SRC/setenviroment[64].bat to set additional environment variables |
| | 105 | 6. Run Visual Studio (devenv) from that same command line session |
| | 106 | 7. Open server solution and build/debug in $CONFIG|$ARCH |
| | 107 | 8. Open web solution and build/debug in $CONFIG|$ARCH |
| | 108 | |
| | 109 | Benefits gained: |
| | 110 | |
| | 111 | * No OEM checkout overhead |
| | 112 | * No OEM build overhead as the buildpack archive already contains the headers/libs/dlls |
| | 113 | * $MG_OEM_LIBS can stay for the duration of needing to build that particular release branch. It only needs to be replaced if building for a different release branch of MapGuide that has different required OEM component versions |
| | 114 | |
| | 115 | === Workflow 4: headless build with pre-built OEM components === |
| | 116 | |
| | 117 | Given a pre-existing mapguide-buildpack-$VERSION-$ARCH-$CONFIG archive and a FDO binary SDK of the correct bitness |
| | 118 | |
| | 119 | 1. Checkout $SVN_ROOT/MgDev to $MG_SRC |
| | 120 | 2. Extract mapguide-buildpack-$VERSION-$ARCH-$CONFIG archive to $MG_OEM_LIBS |
| | 121 | 3. Extract a pre-existing FDO binary SDK into the FDO directory in $MG_OEM_LIBS |
| | 122 | 4. Run $MG_OEM_LIBS/setenvironment $ARCH $CONFIG to set OEM component environment variables |
| | 123 | 5. Run $MG_SRC/setenviroment[64].bat to set additional environment variables |
| | 124 | 6. Run $MG_OEM_LIBS/setenvironment $ARCH $CONFIG to set OEM component environment variables |
| | 125 | 7. Run $MG_SRC/build.bat to build to the MapGuide source |
| | 126 | 8. Run $MG_SRC/build.bat -a=install to build the MapGuide source and copy binaries to $MG_STAGING_AREA |
| | 127 | 9. Run $MG_INST_SRC/build.bat -a=prepare to add additional binaries to $MG_STAGING_AREA |
| | 128 | 10. Run $MG_INST_SRC/build.bat -a=generate to generate the installer file listing $MG_STAGING_AREA |
| | 129 | 11. Run $MG_INST_SRC/build.bat to build the windows installer using the files from $MG_STAGING_AREA |
| | 130 | |
| | 131 | Benefits gained: |
| | 132 | |
| | 133 | * No OEM checkout overhead |
| | 134 | * No OEM build overhead as the buildpack archive already contains the headers/libs/dlls |
| | 135 | * $MG_OEM_LIBS can stay for the duration of needing to build that particular release branch. It only needs to be replaced if building for a different release branch of MapGuide that has different required OEM component versions |
| | 136 | |
| | 137 | This workflow makes continuous integration feasible as it avoids the significant build/checkout overhead of having to build OEM components |
| | 138 | |
| | 139 | == Implications == |
| | 140 | |
| | 141 | This change will significantly change current developer workflows. Refer to any of the example workflows above that match yours and adjust as necessary. |
| | 142 | |
| | 143 | The Visual Studio IDE is tied to the environment variables set by the buildpack setenvironment.bat. It is not possible to switch Platform/Configuration and build. |
| | 144 | Switching Platform/Configuration would require exiting Visual Studio, invoking setenvironment.bat on the appropriate buildpack for the given Platform/Configuration, |
| | 145 | restart Visual Studio under the new environment variables, set Platform/Configuration to match and then build. |
| | 146 | |
| | 147 | Care must also be taken with regards to future OEM component changes/updates. Any such changes should ensure that the buildpacks produced by the OEM build scripts will contain the necessary headers and libraries of your updated OEM components. |
| | 148 | |
| | 149 | Although this RFC enables the possibility of replacing our OEM components with system-installed versions on Linux, the scope of this RFC will only extend to the point that the Linux build process can build off of a pre-existing OEM buildpack (and running a configure script `--with-oem` pointing to this buildpack location) |
| | 150 | |
| | 151 | == Test Plan == |
| | 152 | |
| | 153 | Ensure existing test suites pass under this new build workflow |
| | 154 | |
| | 155 | == Funding / Resources == |
| | 156 | |
| | 157 | Community |
