__TOC__ Here put examples on how to build the docs Logs of documentation writing: * Building the docs: ** http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-07-27.log ** http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-07-28.log * http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-07-31.log * http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-08-09.log * http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-08-10.log * http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-08-16.log * http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-08-17.log * generating overview: ** http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-08-24.log * Fixing issues: ** http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-09-05.log * Conf.py and locales (1/2): ** http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-09-06.log * locales (2/2) * next time * sphinx commands / transifex ** http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-09-13.log == Required softwares == * CMake (>= 2.8) * git * perl: ** 1 hour tutorial: https://www.youtube.com/watch?v=WEghIXs8F6c * sphinx (>=1.1) === Installing dependencies === On ubuntu 16.04 sudo apt install cmake git sudo pip install sphinx == Preparing the repository == === cloning the repository === === branching === == Build commands == === Simplest build === This build will only make the English version and will be faster so you can quickly visualize your changes. ## making the build folder (where builds will be put) mkdir build ## Getting into the build directory cd build # Preparing the build cmake -DHTML=ON .. # don't forget the .. at the end !! # Build html files make html And you're done ! You can go to build/doc/_build/ and open index.html in a web browser. === Main workflow === * make the build directory mkdir build * get into build/: cd build/ ==== Prepare the build with CMake ==== With cmake -D<build option>=ON .. with the option in uppercase: cmake -DSINGLEHTML=ON .. ==== Available options ==== Available options can be see with the command: cmake -L .. ===== Building options ===== There is several option available to the build: * HTML (plain HTML) * DIRHTML (HTML pages with internal links not working) * SINGLEHTML (All the docs in a single html page) * PICKLE * JSON * HTMLHELP * LATEX (LaTeX format, used to generate pdfs) ===== Languages options ===== As for now the supported languages are: * "ca" * "de" * "es" * "fr" * "it" * "ja" * "ko" * "pl" * "ru" * "hu", "el", "id" and "zh" are not available yet (bug fixe in progress) By default, english will be always made. You can add languages easily with the flag -D<Language code>=ON For example, to get spanish: cmake -DHTML=ON -DES=ON To use a specific language, use the cmake flag -D<language code in uppercase>=ON -D<another language code in uppercase>=ON You can pass no language flag (building only english) or as many flag as there is supported languages (and built all of them). To build all languages you can use the flag -DALL_LANG=ON The option (-DES=ON ... or -DALL_LANG=ON) are kept in the OSGeoLiveDoc_BUILD_LANGUAGES variable, which is automatically filled by CMake. ==== make the build ==== make <option> with option in lowercase: make singlehtml ==== about Sphinx ==== Sphinx generates the po files used by transifex. Sphinx needs locales, english is set by default. Other locales are generated from the languages in OSGeoLiveDoc_SUPPORTED_LANGUAGES variable in CMakeLists.txt. For more details, see above. == Projects management == Projects are loaded from the projects_info.csv file. It is the main entry point.
Note: Actually, it is manually build from this file: https://docs.google.com/spreadsheets/d/1Dem6RYkokaX61N8VcomebtqZvcUP-OOt14jt6GPL2TY/edit#gid=2122109332
To add a project or update project version, modify the projects_csv file: https://github.com//projects_info.csv Please indicate if there is a quickstart or not, an overview or not, an openHub account or not, the version, if it is an OSGeo project, or community or none of those. Overviews will be generated with the data from projects_info.csv, and links to quickstarts will be added automaticly. More information how CMake use this file are available above. == Debugging == There is 2 versions: regular and verbose. The first one can be called with this flag: -DOSGeoLiveDoc_DEBUG=ON For example: cmake -DHTML=ON -DES=ON -DOSGeoLiveDoc_DEBUG=ON .. The verbose debug is called with this flag: -DOSGeoLiveDoc_VERBOSE_DEBUG=ON For example: cmake -DHTML=ON -DES=ON -DOSGeoLiveDoc_VERBOSE_DEBUG=ON .. == Configuring CMake / Setting the build == CMake is use to build the repository instead of simply make because it can be finely tuned. CMake configuration is done by the CMakeLists.txt. CMake says how the build is made from that file. This file allows to set a few things: ==== Checking CMake version ==== cmake_minimum_required(VERSION 2.8 FATAL_ERROR) ==== Checking build folder ==== The first thing that CMake does is to check if you are in the folder called build. If it is not the case, CMake will return an error. 
if ( ${CMAKE_SOURCE_DIR} STREQUAL ${CMAKE_BINARY_DIR} )
message(FATAL_ERROR "In-source builds not allowed.
Please make a new directory (called a build directory) and run CMake from there.
You may need to remove CMakeCache.txt." )
endif()
==== Changing the version number ==== In CMakeLists.txt, you can change the OSGeo-Live version number with the following variables: 
## Major version
set(OSGeoLiveDoc_VERSION_MAJOR "12")

## minor version
set(OSGeoLiveDoc_VERSION_MINOR "0")

## Patch number
set(OSGeoLiveDoc_VERSION_PATCH "0")

## if it is a development version, a Release Candidate or a final release
set(OSGeoLiveDoc_VERSION_DEV "-dev")
==== Checking the sphinx version ==== To build, you sphinx 1.1 and above: set(SPHINX_MINIMUM_VERSION "1.1") If not a warning will be displayed. If you need a more recent version to build the doc, you can test it from here. ==== Checking Perl ==== Perl is needed at some point, so ==== Adding / removing a language ==== Add the 2 letters code of the new language: set(OSGeoLiveDoc_SUPPORTED_LANGUAGES "ca" "de" "es" "fr" "it" "ja" "ko" "pl" "ru") Then add a new INSERT_INTO_MAP command to add the language to the lang navigation bar (in index.html). For example: INSERT_INTO_MAP("fr" "Français") in the CMakeLists.txt. The line set(OSGeoLiveDoc_ENGLISH "en") is not supposed to change, it sets the default build in english. === Projects management === Projects are loaded from the projects_info.csv file. It is the main entry point.
Note: Actually, it is manually build from this file: https://docs.google.com/spreadsheets/d/1Dem6RYkokaX61N8VcomebtqZvcUP-OOt14jt6GPL2TY/edit#gid=2122109332
To add a project or update project version, modify the projects_csv file: https://github.com//projects_info.csv Please indicate there is a quickstart or not, an overview or not, an openHub account or not, the version, if it is an OSGeo project, or community or none of those. Overviews will be generated with the data from projects_info.csv, and links to quickstarts will be added automaticly. == reading the file == file(STRINGS projects_info.csv OSGeoLiveDoc_PROJECTS_VERSIONS_FILE) opens the csv file set (OSGeoLiveDoc_PROJECTS_VERSIONS "") is prepared to stock data from the file Data are extracted with a regular expression, which detect the column, but has to know the number of columns. Currently the file has 9 columns. The file is processed line by line. == Slicing the lines with a regex == set (OSGeoLiveDoc_CONFIGURATION_REGEXP "^(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)") So (.*)\\| has to be repeated 9 times. If a column is added in projects_info.csv, it is mandatory to add one more (.*)\\| in the regex to find the new column. Each column will extracted by group, for example, column 3 will be extracted as group "\3"; because it's a csv, it's kind of a 1 to 1 relationship between the group and the column. Quick explanation of the regex: ^ <--- beginning of line (.'') <---- any character = "." any number of times = "''" grouped "(" ")" \| <--- code for vertical line "|" The for each line in the file, the ones starting with '''#''' are treated as comments. Comments will be ignored and the whole line replaced by a #. This part only use the 5 first column. == Group use == For example, group "\3", will get the project version, and will be cleaned with STRIP: string(STRIP ${project_version} project_version) will strip the blanks spaces at the beginning and the end of the string. So " 4.2.1 " will return "4.2.1" and "4.2.2 beta " -> "4.2.2 beta". === Debugging === This treatment can be seen with the -DOSGeoLiveDoc_VERBOSE_DEBUG=ON flag. It will show something like that: -- Y 52nSOS 4.4.0 want_quickstart:Y want_overview:Y -- Y 52nWPS 3.6.1 want_quickstart:Y want_overview:Y -- N 52nWSS retired want_quickstart:N want_overview:N -- 52nWSS is not for documentation -- N atlasstyler retired want_quickstart:N want_overview:N -- atlasstyler is not for documentation -- N cartaro retired want_quickstart:N want_overview:N -- cartaro is not for documentation Active projects will get want_quickstart and want_overview set to Y; the retired projects will be set to N and won't be used for documentation. This process is always done, but won't be print unless the -DOSGeoLiveDoc_VERBOSE_DEBUG=ON flag is used. === Project version in conf.py === For each line (so active and retired projects), a new python command will be generated for inserting in conf.py. .. |version-${project_name}| replace:: ${project_version}\n" For example, for the ''4.4.0'' release of the ''52nSOS'' project, the string will be .. |version-52nSOS| replace:: 4.4.0. === Selecting project who needs on overview and a quickstart building === This code add the project name to lists for quickstart and overviews, if a Y is the columns/ groups 4 and 5. Some projects might have an overview but no quickstart, it is mostly libraries. if ("{want_quickstart}" MATCHES "Y") list(APPEND OSGeoLiveDoc_QUICKSTART "{project_name}") endif() if ("{want_overview}" MATCHES "Y") list(APPEND OSGeoLiveDoc_OVERVIEW "{project_name}") endif() = Standards = Standards are builded with this set of commandes: set(OSGeoLiveDoc_STANDARDS "csw" "gml" "sld" "wcs" "wms" "fe" "kml" "sensorml" "sos" "wfs" "wps" ) # Other settings There is several files that is need for the build, they are copied with the command configure_file() * settings.py * licences.csv * contributors.csv * translators.csv For example, configure_file("settings.py" "settings.py") will copy ''settings.py'' and '''substitute''' the variables defined in the cmake. Which is very handy. The csv files are copied but are not modified. They'll be loaded later in the html file with an another process. = Setting building variables = The line add_subdirectory(doc) will add the '''doc''' folder to the build and all the files in it will be set with the variables from cmake, by copying the CMakeLists.txt file in each folder and subfolder. Each copy = Comments in CMake = There is no comments in CMake, but there is a way to have code that won't be executed. if(FALSE) endif() = Included files and folder = CMake will use the files and folders in the main folder (aka OSGeoLive-doc) but only the ones present in the variables OSGeoLiveDoc_FILES and OSGeoLiveDoc_FOLDERS. In CMakeLists.txt they are set like that: 
#---------------------
# Files
#---------------------
set(OSGeoLiveDoc_FILES
#"disclaimer.rst"
"translators.rst"
"osgeo_contact.rst"
"contributors.rst"
"index.rst"
"contact.rst"
#"metrics.rst"
"sponsors.rst"
"welcome_message.txt"
"copyright.rst"
"download.rst"
"mac_installers.rst"
"presentation.rst"
"sponsors_osgeo.rst"
"prior_applications.rst"
"win_installers.rst"
)


#---------------------
# Directories
#---------------------
set(OSGeoLiveDoc_SUBDIRS
"WindowsInstallers"
"MacInstallers"
"presentation"
"quickstart"
"overview"
"standards"
)
So commented out or deleted lines won't be taken in account, and won't be translated. If you comment "overview", the overviews won't be built. = Directory processing = This process is managed by the command add_subdirectory() which tell '''CMake''' to look for a ''CMakeLists.txt'' in the subdirectory and execute it. For example, the last line of the main ''CMakeLists.txt'' is add_subdirectory(doc) so '''CMake''' will look into doc/ and execute the ''CMakeLists.txt'' in it. Several variables will be initialized like ${OSGeoLiveDoc_SUBDIRS} which contains the folders to look into. Then the following loop will be looking in each named subfolder to execute its ''CMakeLists.txt'' {bash} foreach (dir ${OSGeoLiveDoc_SUBDIRS}) message(STATUS " Adding directory ${dir}") add_subdirectory(${dir}) endforeach() For example, the proccessed files for the Overview folder are extract from ${OSGeoLiveDoc_OVERVIEW}) built in the previous process (from projects_info.csv). Files not listed in ${OSGeoLiveDoc_OVERVIEW}) won't be processed. == Local list == Some ''CMakeLists.txt'' have a local list, for example in quickstart, there is local list for non project quickstart. It is quickstarts non included in the ''projects_info.csv''. If a new project add its ''.rst'' file in the quickstart folder but forget to update ''projects_info.csv''; the quickstart won't show. For a non project quickstart like ''virtualization_quickstart'', which explain how to use OSGeo-Live in a virtual machine, it has to be set in the '''LOCAL''' variable of the quickstarts ''CMakeLists.txt''. == Quickstart and Overview folder == The CMakeLists.txt here to copy all projects listed in projetc_info.csv and transmitted with the variables ${OSGeoLiveDoc_QUICKSTART} and ${OSGeoLiveDoc_OVERVIEW}. Local CMakeLists.txt will be initialized with those variables. === Overview folder === in Overview, there is no local files. The Overview webpage (listing all projects) is generated with perl from data extracted of projects_info.csv. == Standards == Standards were not part of the documentation any more, they have been removed. The process is still there, in case they come back into the documentation. == Main files == Main files, present at the root of the project and website, are processed if listed in the variable OSGeoLiveDoc_FILES at the beginning of the ''CMakeLists.txt''. They can be added and commented out easily. = Images = The images folder is entirely copied to build/doc/images. All images are copied, even the one that are not used by the documentation any more. That need to be cleaned. = Scripts = Some files are build with perl scripts like overview.html or metrics. Those scripts will be detailed below. == overview.html == [https://live.osgeo.org/en/overview/overview.html overview.html] is the overview of all projects, and is generated from data in projects_info.csv. So retired project wouldn't appear in it. The script is there: https://github.com/cvvergara/OSGeoLive-doc/blob/cm_fix_header/doc/scripts/build_overview.pl First three lines are testing the shell and args. There is includes (pragmas) from perl library used by the script in line 6 to 8. Then three specials variables are declared: - ''name (filename) - ''dir (get the directory of the perl script) - *prune (get the path to the perl script) They are shortcuts for functions (*File::Find::name is a function returning the filename). Those varaibles are loaded with informations from the CMake: * $DEBUG = "@OSGeoLiveDoc_DEBUG@"; * $version = "@OSGeoLiveDoc_VERSION@"; * $projects_info_file = '@CMAKE_SOURCE_DIR@/projects_info.csv'; * $output_file = '@CMAKE_BINARY_DIR@/doc/overview/overview.rst'; ==== See if the projects_info.csv file exist, if don't the script stop and throws an error: ==== die "ERROR: Failed to find: '$projects_info_file'\n" unless -f $projects_info_file; === Get the configuration from projects_info.csv === This line parse the projects_info file and stores it in $configuration. If the flag -DOSGeoLiveDoc_DEBUG=ON is used, then a comment line will be print. my $configuration = read_and_parse_configuration($projects_info_file); ==== read_and_parse_configuration subroutine ==== This subroutine takes a file in parameter. It is stored in $file with the keyword shift. Then an empty Hash called %hash is initialized. It will store hashes extracted during the process. The script will try to open the file in parameter or die and throws an error. Then a while loop reads the file line by line and exit when the file is empty (operation becomes TRUE). During the while loop several actions are made: - ignores the commented lines; - clean the line by removing spaces surplus; - ignores project who are not subject to documentation (no overview and / or no quickstart) - get lines in the right bucket (section) $hash{$values[5]} For example, with the following line: Y | 52nSOS | 4.4.0 | Y | Y | Web Services | | Sensor Observation Service | SensorObservationService The instructions will put the line in the bucket that has the name ''Web Services''. At the end of the loop, the file is closed and the hash with the collected data is returned. In the script, the hash will be store in $configuration. === Building sections === Sections are stored in $sections. For each defined sections, there will be a call for the section in $configuration. It use the subroutine get_section. The result will be append to $sections. ==== get_sections() ==== This subroutine take two parameters: * the name of the section * the variable where the configuration is stored
Note: There is a lot of concatenation in this subroutine. It is short written with .=
get_sections() workflow: - prepare the section headers and toc tree - split values around the |and store it in an array (@values) - cleans the excess | and spaces in in each value - Test it ''overview'' is set to Y then handle the overview by adding it to the toctree and adding the bullet line - Test it ''quickstart'' is set to Y then handle the quickstart by adding it to the toctree and adding the bullet line between brackets - Write the comment / short description if provided. When all projects have been read, the toctree is added to the contents, then the bullets. The contents is returned. === Writing the file === When all the sections are build and added to $sections, the subroutine write_script() is called. It takes one parameter, $sections with all the sections, in the right order. # it tries to create the output file (or die if cannot) # write out the header and the commands to clean up the old extension # paste the content of $sections after the toc (.. contents:\n :local:) and before prior applications # close the file == Script exit == When the script is done, it exit returning 0. It is a mandatory for perl. = Metrics.pl = === Variables === The variables are loaded by CMake during the build from data in CMakeLists.txt. my $DEBUG = "@OSGeoLiveDoc_DEBUG@"; my $version = "@OSGeoLiveDoc_VERSION@"; my $projects_info_file = '@CMAKE_SOURCE_DIR@/projects_info.csv'; my $output_file = '@CMAKE_CURRENT_BINARY_DIR@/../metrics.rst'; = Useful commands = === Rename or move files and rename all occurences === ==== Moving the files to the right folder ==== git mv doc/images/projects/gvsigFOOOOOOOOOOOO doc/images/projects/gvsig # see the change git status ==== Changing all occurrences ==== To do that, we will use a one liner perl command to change from projects/gvsigFOOOOOOOOOOOO to projects/gvsig the / is special in perl so, \ is prepended in the command from projects/gvsigFOOOOOOOOOOOO to projects/gvsig ==== THE command: ==== perl -pi -e "s/projects\/gvsigFOOOOOOOOOOOO/projects\/gvsig/" $(git ls-files) * git ls-files : lists the files that git knows about * perl <--- use that * -pi <--- one liner * -e <-- do changes on the file and save them on the file * "s/projects/gvsigFOOOOOOOOOOOO/projects/gvsig/" <<< s/// substitute from to * $(git ls-files) <---- $() * git ls-files <--- generates filenames know to git = Check and remove unused images = When updating the docs, it can happen that some images are not removed. So the [https://github.com/cvvergara/OSGeoLive-doc/blob/cm_fix_header/bin/remove_unsused_images.sh remove_unsused_images.sh] script check all the projects (current and retired). = conf.py = Sphinx need a conf.py file with several data. With this project, it is built from [https://github.com/cvvergara/OSGeoLive-doc/blob/cm_fix_header/doc/conf.py.in conf.py.in] where @var@ are modified with data from CMakeLists.txt. The original [https://github.com/cvvergara/OSGeoLive-doc/blob/cm_fix_header/doc/conf.py conf.py] should be remove at the end of the project. This is done a the [https://github.com/cvvergara/OSGeoLive-doc/blob/cm_fix_header/doc/CMakeLists.txt#L76 L. 76 of CMakeLists.txt] = Theming = Themes are stock in the _theme subdirectory, they are loaded by the line [https://github.com/cvvergara/OSGeoLive-doc/blob/cm_fix_header/doc/CMakeLists.txt#L79 L. 79 of CMakeLists.txt] There is a small [https://github.com/cvvergara/OSGeoLive-doc/blob/cm_fix_header/doc/_themes/overview/CMakeLists.txt local CMakeLists.txt] for each theme. It is used to configure the language bar with the asked languages. = Locales = Locales are stocked in the (locales folder](https://github.com/cvvergara/OSGeoLive-doc/tree/cm_fix_header/locale). This folder is updated by 2 distinct processes: * first time, by sphinx, when building the strings to be translated (before pushing it to '''Transifex''') * second time, when the translated strings are retrieved from [https://www.transifex.com/osgeo/osgeolive Transifex] Waiting for [https://github.com/cvvergara/OSGeoLive-doc/issues/44 issue #44] to be fixed. = Sphinx commands = The [https://github.com/cvvergara/OSGeoLive-doc/blob/cm_fix_header/doc/CMakeLists.txt#L115 lines l.115 to l.156] are sphinx commands. Those commands are used to communicate with the [https://www.transifex.com/osgeo/osgeolive Transifex plateform].