wiki:building_the_docs

Context Navigation

Version 8 (modified by bakaniko, 7 years ago) ( diff )
--

TOC

Here put examples on how to build the docs

Logs of documentation writing:

Building the docs:

http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-07-27.log http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-07-28.log

http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-08-24.log

Fixing issues:

http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-09-05.log

Conf.py and locales (1/2):

http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-09-06.log

locales (2/2)
next time
sphinx commands / transifex

http://irclogs.geoapt.com/osgeolive/%23osgeolive.2017-09-13.log

Required softwares

CMake (>= 2.8)
git
perl:

1 hour tutorial: https://www.youtube.com/watch?v=WEghIXs8F6c

sphinx (>=1.1)

Installing dependencies

On ubuntu 16.04

sudo apt install cmake git

sudo -H pip install sphinx

Preparing the repository

cloning the repository

This is a temporary repository before merging into the main one

git clone https://github.com/cvvergara/OSGeoLive-doc

branching

git checkout develop

Python 2.7

This builds works with python 2.7.

If it is not standard into your environnement, create a virtual one.

This is an example with the anaconda distribution.

conda create -n docbuild27 python=2.7 anaconda
source activate docbuild27

To exit the environnement at the end:

source deactivate

Build commands

Simplest build

This build will only make the English version and will be faster so you can quickly visualize your changes.

## making the build folder (where builds will be put)
mkdir build

## Getting into the build directory
cd build

# Preparing the build
cmake -DHTML=ON ..       # don't forget the .. at the end !!

# Build html files
make html

And you're done ! You can go to build/doc/_build/ and open index.html in a web browser.

Main workflow

make the build directory mkdir build
get into build/: cd build/

Prepare the build with CMake

With cmake -D<build option>=ON .. with the option in uppercase:

cmake -DSINGLEHTML=ON ..

Available options

Available options can be see with the command:

cmake -L ..

Building options

There is several option available to the build:

HTML (plain HTML)
DIRHTML (HTML pages with internal links not working)
SINGLEHTML (All the docs in a single html page)
PICKLE
JSON
HTMLHELP
LATEX (LaTeX format, used to generate pdfs)

Languages options

As for now the supported languages are:

"ca"
"de"
"es"
"fr"
"it"
"ja"
"ko"
"pl"
"ru"
"hu", "el", "id" and "zh" are not available yet (bug fixe in progress)

By default, english will be always made. You can add languages easily with the flag -D<Language code>=ON

For example, to get spanish:

cmake -DHTML=ON -DES=ON

To use a specific language, use the cmake flag -D<language code in uppercase>=ON -D<another language code in uppercase>=ON

You can pass no language flag (building only english) or as many flag as there is supported languages (and built all of them).

To build all languages you can use the flag -DALL_LANG=ON

The option (-DES=ON ... or -DALL_LANG=ON) are kept in the OSGeoLiveDoc_BUILD_LANGUAGES variable, which is automatically filled by CMake.

make the build

make <option> with option in lowercase:

make singlehtml

about Sphinx

Sphinx generates the po files used by transifex.

Sphinx needs locales, english is set by default. Other locales are generated from the languages in OSGeoLiveDoc_SUPPORTED_LANGUAGES variable in CMakeLists.txt. For more details, see above.

Projects management

Projects are loaded from the projects_info.csv file. It is the main entry point.

<blockquote>Note: Actually, it is manually build from this file: https://docs.google.com/spreadsheets/d/1Dem6RYkokaX61N8VcomebtqZvcUP-OOt14jt6GPL2TY/edit#gid=2122109332 </blockquote> To add a project or update project version, modify the projects_csv file: https://github.com/<PATH TO FILE>/projects_info.csv

Please indicate if there is a quickstart or not, an overview or not, an openHub account or not, the version, if it is an OSGeo project, or community or none of those.

Overviews will be generated with the data from projects_info.csv, and links to quickstarts will be added automaticly.

More information how CMake use this file are available above.

Debugging

There is 2 versions: regular and verbose.

The first one can be called with this flag: -DOSGeoLiveDoc_DEBUG=ON

For example: cmake -DHTML=ON -DES=ON -DOSGeoLiveDoc_DEBUG=ON ..

The verbose debug is called with this flag: -DOSGeoLiveDoc_VERBOSE_DEBUG=ON

For example: cmake -DHTML=ON -DES=ON -DOSGeoLiveDoc_VERBOSE_DEBUG=ON ..

Configuring CMake / Setting the build

CMake is use to build the repository instead of simply make because it can be finely tuned. CMake configuration is done by the CMakeLists.txt. CMake says how the build is made from that file.

This file allows to set a few things:

Checking CMake version

cmake_minimum_required(VERSION 2.8 FATAL_ERROR)

Checking build folder

The first thing that CMake does is to check if you are in the folder called build. If it is not the case, CMake will return an error.

{{{if ( ${CMAKE_SOURCE_DIR} STREQUAL ${CMAKE_BINARY_DIR} ) message(FATAL_ERROR "In-source builds not allowed. Please make a new directory (called a build directory) and run CMake from there. You may need to remove CMakeCache.txt." ) endif()}}}

Changing the version number

In CMakeLists.txt, you can change the OSGeo-Live version number with the following variables:

{{{## Major version set(OSGeoLiveDoc_VERSION_MAJOR "12")

## minor version set(OSGeoLiveDoc_VERSION_MINOR "0")

## Patch number set(OSGeoLiveDoc_VERSION_PATCH "0")

## if it is a development version, a Release Candidate or a final release set(OSGeoLiveDoc_VERSION_DEV "-dev")}}}

Checking the sphinx version

To build, you sphinx 1.1 and above:

set(SPHINX_MINIMUM_VERSION "1.1")

If not a warning will be displayed. If you need a more recent version to build the doc, you can test it from here.

Checking Perl

Perl is needed at some point, so

Adding / removing a language

Add the 2 letters code of the new language:

set(OSGeoLiveDoc_SUPPORTED_LANGUAGES "ca" "de" "es" "fr" "it" "ja" "ko" "pl" "ru") Then add a new INSERT_INTO_MAP command to add the language to the lang navigation bar (in index.html). For example:

INSERT_INTO_MAP("fr" "Français") in the CMakeLists.txt.

The line set(OSGeoLiveDoc_ENGLISH "en") is not supposed to change, it sets the default build in english.

Projects management

Projects are loaded from the projects_info.csv file. It is the main entry point.

Please indicate there is a quickstart or not, an overview or not, an openHub account or not, the version, if it is an OSGeo project, or community or none of those.

Overviews will be generated with the data from projects_info.csv, and links to quickstarts will be added automaticly.

reading the file

file(STRINGS projects_info.csv OSGeoLiveDoc_PROJECTS_VERSIONS_FILE) opens the csv file

set (OSGeoLiveDoc_PROJECTS_VERSIONS "") is prepared to stock data from the file

Data are extracted with a regular expression, which detect the column, but has to know the number of columns. Currently the file has 9 columns.

The file is processed line by line.

Slicing the lines with a regex

set (OSGeoLiveDoc_CONFIGURATION_REGEXP "^(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)\\|(.*)")

So (.*)\\| has to be repeated 9 times. If a column is added in projects_info.csv, it is mandatory to add one more (.*)\\| in the regex to find the new column.

Each column will extracted by group, for example, column 3 will be extracted as group "\3"; because it's a csv, it's kind of a 1 to 1 relationship between the group and the column.

Quick explanation of the regex:

^{<--- beginning of line (.) <---- any character = "." any number of times = "" grouped "(" ")" \| <--- code for vertical line "|"}

The for each line in the file, the ones starting with # are treated as comments. Comments will be ignored and the whole line replaced by a #.

This part only use the 5 first column.

Group use

For example, group "\3", will get the project version, and will be cleaned with STRIP:

string(STRIP ${project_version} project_version) will strip the blanks spaces at the beginning and the end of the string. So " 4.2.1 " will return "4.2.1" and "4.2.2 beta " -> "4.2.2 beta".

Debugging

This treatment can be seen with the -DOSGeoLiveDoc_VERBOSE_DEBUG=ON flag. It will show something like that:

-- Y 52nSOS 4.4.0 want_quickstart:Y want_overview:Y -- Y 52nWPS 3.6.1 want_quickstart:Y want_overview:Y -- N 52nWSS retired want_quickstart:N want_overview:N -- 52nWSS is not for documentation -- N atlasstyler retired want_quickstart:N want_overview:N -- atlasstyler is not for documentation -- N cartaro retired want_quickstart:N want_overview:N -- cartaro is not for documentation

Active projects will get want_quickstart and want_overview set to Y; the retired projects will be set to N and won't be used for documentation.

This process is always done, but won't be print unless the -DOSGeoLiveDoc_VERBOSE_DEBUG=ON flag is used.

Project version in conf.py

For each line (so active and retired projects), a new python command will be generated for inserting in conf.py.

.. |version-${project_name}| replace:: ${project_version}\n"

For example, for the 4.4.0 release of the 52nSOS project, the string will be .. |version-52nSOS| replace:: 4.4.0.

Selecting project who needs on overview and a quickstart building

This code add the project name to lists for quickstart and overviews, if a Y is the columns/ groups 4 and 5. Some projects might have an overview but no quickstart, it is mostly libraries.

if ("<math>{want_quickstart}" MATCHES "Y") list(APPEND OSGeoLiveDoc_QUICKSTART "</math>{project_name}") endif() if ("<math>{want_overview}" MATCHES "Y") list(APPEND OSGeoLiveDoc_OVERVIEW "</math>{project_name}") endif()

Standards

Standards are builded with this set of commandes:

set(OSGeoLiveDoc_STANDARDS "csw" "gml" "sld" "wcs" "wms" "fe" "kml" "sensorml" "sos" "wfs" "wps" ) # Other settings There is several files that is need for the build, they are copied with the command configure_file()

settings.py
licences.csv
contributors.csv
translators.csv

For example, configure_file("settings.py" "settings.py") will copy settings.py and substitute the variables defined in the cmake. Which is very handy.

The csv files are copied but are not modified. They'll be loaded later in the html file with an another process.

Setting building variables

The line add_subdirectory(doc) will add the doc folder to the build and all the files in it will be set with the variables from cmake, by copying the CMakeLists.txt file in each folder and subfolder. Each copy

Comments in CMake

There is no comments in CMake, but there is a way to have code that won't be executed.

if(FALSE) <none executed code> endif()

Included files and folder

CMake will use the files and folders in the main folder (aka OSGeoLive-doc) but only the ones present in the variables OSGeoLiveDoc_FILES and OSGeoLiveDoc_FOLDERS.

In CMakeLists.txt they are set like that:

{{{#--------------------- # Files #--------------------- set(OSGeoLiveDoc_FILES #"disclaimer.rst" "translators.rst" "osgeo_contact.rst" "contributors.rst" "index.rst" "contact.rst" #"metrics.rst" "sponsors.rst" "welcome_message.txt" "copyright.rst" "download.rst" "mac_installers.rst" "presentation.rst" "sponsors_osgeo.rst" "prior_applications.rst" "win_installers.rst" )

#--------------------- # Directories #--------------------- set(OSGeoLiveDoc_SUBDIRS "WindowsInstallers" "MacInstallers" "presentation" "quickstart" "overview" "standards" )}}} So commented out or deleted lines won't be taken in account, and won't be translated. If you comment "overview", the overviews won't be built.

Directory processing

This process is managed by the command add_subdirectory() which tell CMake to look for a CMakeLists.txt in the subdirectory and execute it.

For example, the last line of the main CMakeLists.txt is add_subdirectory(doc) so CMake will look into doc/ and execute the CMakeLists.txt in it. Several variables will be initialized like ${OSGeoLiveDoc_SUBDIRS} which contains the folders to look into.

Then the following loop will be looking in each named subfolder to execute its CMakeLists.txt

{bash} foreach (dir ${OSGeoLiveDoc_SUBDIRS}) message(STATUS " Adding directory ${dir}") add_subdirectory(${dir}) endforeach()

For example, the proccessed files for the Overview folder are extract from ${OSGeoLiveDoc_OVERVIEW}) built in the previous process (from projects_info.csv). Files not listed in ${OSGeoLiveDoc_OVERVIEW}) won't be processed.

Local list

Some CMakeLists.txt have a local list, for example in quickstart, there is local list for non project quickstart. It is quickstarts non included in the projects_info.csv.

If a new project add its .rst file in the quickstart folder but forget to update projects_info.csv; the quickstart won't show.

For a non project quickstart like virtualization_quickstart, which explain how to use OSGeo-Live in a virtual machine, it has to be set in the LOCAL variable of the quickstarts CMakeLists.txt.

Quickstart and Overview folder

The CMakeLists.txt here to copy all projects listed in projetc_info.csv and transmitted with the variables ${OSGeoLiveDoc_QUICKSTART} and ${OSGeoLiveDoc_OVERVIEW}.

Local CMakeLists.txt will be initialized with those variables.

Overview folder

in Overview, there is no local files. The Overview webpage (listing all projects) is generated with perl from data extracted of projects_info.csv.

Standards

Standards were not part of the documentation any more, they have been removed. The process is still there, in case they come back into the documentation.

Main files

Main files, present at the root of the project and website, are processed if listed in the variable OSGeoLiveDoc_FILES at the beginning of the CMakeLists.txt. They can be added and commented out easily.

Images

The images folder is entirely copied to build/doc/images. All images are copied, even the one that are not used by the documentation any more. That need to be cleaned.

Scripts

Some files are build with perl scripts like overview.html or metrics. Those scripts will be detailed below.

overview.html

overview.html is the overview of all projects, and is generated from data in projects_info.csv. So retired project wouldn't appear in it.

The script is there: https://github.com/cvvergara/OSGeoLive-doc/blob/cm_fix_header/doc/scripts/build_overview.pl

First three lines are testing the shell and args.

There is includes (pragmas) from perl library used by the script in line 6 to 8.

Then three specials variables are declared: - name (filename) - dir (get the directory of the perl script) - *prune (get the path to the perl script)

They are shortcuts for functions (*File::Find::name is a function returning the filename).

Those varaibles are loaded with informations from the CMake: * $DEBUG = "@OSGeoLiveDoc_DEBUG@"; * $version = "@OSGeoLiveDoc_VERSION@"; * $projects_info_file = '@CMAKE_SOURCE_DIR@/projects_info.csv'; * $output_file = '@CMAKE_BINARY_DIR@/doc/overview/overview.rst';

See if the `projects_info.csv` file exist, if don't the script stop and throws an error:

die "ERROR: Failed to find: '$projects_info_file'\n" unless -f $projects_info_file;

Get the configuration from projects_info.csv

This line parse the projects_info file and stores it in $configuration. If the flag -DOSGeoLiveDoc_DEBUG=ON is used, then a comment line will be print.

my $configuration = read_and_parse_configuration($projects_info_file);

read_and_parse_configuration subroutine

This subroutine takes a file in parameter. It is stored in $file with the keyword shift.

Then an empty Hash called %hash is initialized. It will store hashes extracted during the process.

The script will try to open the file in parameter or die and throws an error.

Then a while loop reads the file line by line and exit when the file is empty (operation becomes TRUE). During the while loop several actions are made: - ignores the commented lines; - clean the line by removing spaces surplus; - ignores project who are not subject to documentation (no overview and / or no quickstart) - get lines in the right bucket (section) $hash{$values[5]}

For example, with the following line:

Y | 52nSOS | 4.4.0 | Y | Y | Web Services | | Sensor Observation Service | SensorObservationService

The instructions will put the line in the bucket that has the name Web Services.

At the end of the loop, the file is closed and the hash with the collected data is returned.

In the script, the hash will be store in $configuration.

Building sections

Sections are stored in $sections.

For each defined sections, there will be a call for the section in $configuration. It use the subroutine get_section.

The result will be append to $sections.

get_sections()

This subroutine take two parameters: * the name of the section * the variable where the configuration is stored

<blockquote>Note: There is a lot of concatenation in this subroutine. It is short written with .= </blockquote> get_sections() workflow: - prepare the section headers and toc tree - split values around the |and store it in an array (@values) - cleans the excess | and spaces in in each value - Test it overview is set to Y then handle the overview by adding it to the toctree and adding the bullet line - Test it quickstart is set to Y then handle the quickstart by adding it to the toctree and adding the bullet line between brackets - Write the comment / short description if provided.

When all projects have been read, the toctree is added to the contents, then the bullets. The contents is returned.

Writing the file

When all the sections are build and added to $sections, the subroutine write_script() is called.

It takes one parameter, $sections with all the sections, in the right order.

# it tries to create the output file (or die if cannot) # write out the header and the commands to clean up the old extension # paste the content of $sections after the toc (.. contents:\n :local:) and before prior applications # close the file

Script exit

When the script is done, it exit returning 0. It is a mandatory for perl.

Metrics.pl

Variables

The variables are loaded by CMake during the build from data in CMakeLists.txt.

{{{my $DEBUG = "@OSGeoLiveDoc_DEBUG@"; my $version = "@OSGeoLiveDoc_VERSION@"; my $projects_info_file = '@CMAKE_SOURCE_DIR@/projects_info.csv'; my $output_file = '@CMAKE_CURRENT_BINARY_DIR@/../metrics.rst';}}}

Useful commands

Rename or move files and rename all occurences

Moving the files to the right folder

{{{git mv doc/images/projects/gvsigFOOOOOOOOOOOO doc/images/projects/gvsig

# see the change git status}}}

Changing all occurrences

To do that, we will use a one liner perl command to change from projects/gvsigFOOOOOOOOOOOO to projects/gvsig

the / is special in perl so, \ is prepended in the command from projects/gvsigFOOOOOOOOOOOO to projects/gvsig

THE command:

perl -pi -e "s/projects\/gvsigFOOOOOOOOOOOO/projects\/gvsig/" $(git ls-files)

git ls-files : lists the files that git knows about
perl <--- use that
-pi <--- one liner
-e <-- do changes on the file and save them on the file
"s/projects/gvsigFOOOOOOOOOOOO/projects/gvsig/" <<< s/<from>/<to>/ substitute from to
$(git ls-files) <---- $(<command>)
git ls-files <--- generates filenames know to git

Check and remove unused images

When updating the docs, it can happen that some images are not removed. So the remove_unsused_images.sh script check all the projects (current and retired).

conf.py

Sphinx need a conf.py file with several data. With this project, it is built from conf.py.in where @var@ are modified with data from CMakeLists.txt.

The original conf.py should be remove at the end of the project. This is done a the L. 76 of CMakeLists.txt

Theming

Themes are stock in the _theme subdirectory, they are loaded by the line L. 79 of CMakeLists.txt

There is a small local CMakeLists.txt for each theme. It is used to configure the language bar with the asked languages.

Locales

Locales are stocked in the (locales folder](https://github.com/cvvergara/OSGeoLive-doc/tree/cm_fix_header/locale). This folder is updated by 2 distinct processes:

first time, by sphinx, when building the strings to be translated (before pushing it to Transifex)
second time, when the translated strings are retrieved from Transifex

Waiting for issue #44 to be fixed.

Sphinx commands

The lines l.115 to l.156 are sphinx commands. Those commands are used to communicate with the Transifex plateform.

Note: See TracWiki for help on using the wiki.

Download in other formats:

Plain Text

Context Navigation

Required softwares

Installing dependencies

Preparing the repository

cloning the repository

branching

Python 2.7

Build commands

Simplest build

Main workflow

Prepare the build with CMake

Available options

Building options

Languages options

make the build

about Sphinx

Projects management

Debugging

Configuring CMake / Setting the build

Checking CMake version

Checking build folder

Changing the version number

Checking the sphinx version

Checking Perl

Adding / removing a language

Projects management

reading the file

Slicing the lines with a regex

Group use

Debugging

Project version in conf.py

Selecting project who needs on overview and a quickstart building

Standards

Setting building variables

Comments in CMake

Included files and folder

Directory processing

Local list

Quickstart and Overview folder

Overview folder

Standards

Main files

Images

Scripts

overview.html

See if the projects_info.csv file exist, if don't the script stop and throws an error:

Get the configuration from projects_info.csv

read_and_parse_configuration subroutine

Building sections

get_sections()

Writing the file

Script exit

Metrics.pl

Variables

Useful commands

Rename or move files and rename all occurences

Moving the files to the right folder

Changing all occurrences

THE command:

Check and remove unused images

conf.py

Theming

Locales

Sphinx commands

Download in other formats:

See if the `projects_info.csv` file exist, if don't the script stop and throws an error: