Building Cloe¶

Note

Before you follow any of the steps described below, please make sure that you complete all steps described on the Installation page. Please also read the sections on Understanding Cloe Packages first.

Suppose you intend to modify the content of one of Cloe’s multiple Conan packages. Below you can find some hints on how to build and test your package of interest without exporting it to the Conan local cache every time.

Build System¶

If you are developing Cloe itself, it is necessary to understand how the build system is composed. In essence, we are using GNU Make as a wrapper for Conan, and Conan as a wrapper for CMake. The following files are present in the root directory of every Cloe package (e.g. in plugins/basic/):

Makefile

Contains a set of targets that can be run in this directory. For most packages, the Makefile.package from the top-level directory is included to provide Make targets for the most common Conan commands. Refer to the output of make help in the respective package directory for a list of the available targets (here: make -C ./models help):

make: Entering directory '/home/docs/checkouts/readthedocs.org/user_builds/cloe/checkouts/stable/models'
Usage: make target

The following targets define common operations with this package in
editable (local in-source) and uneditable (in the Conan cache) modes.

Available targets:
  help                  show this help                                
  status                show status of package                        
  info                  show detailed package info                    
  smoketest-deps        build smoketest dependencies for package      
  smoketest             run smoketests for package (requires built packages) 

  export                export recipe and sources                     [conan-cache]
  download              download or create package                    [conan-cache]
  package               create package with build policy              [conan-cache]
  list                  list installed package files                  [conan-cache]
  purge                 remove package from cache                     [conan-cache]

  editable              instruct Conan to use in-source build         
  uneditable            instruct Conan to use local cache             

  all                   build the package                             [in-source]
  configure             install dependencies and configure package    [in-source]
  test                  run CMake tests if they are available         [in-source]
  export-pkg            export build artifacts to Conan cache         [in-source]
  clean                 remove build directory                        [in-source]

  Configuration:
    LOCKFILE_SOURCE = 
    SOURCE_DIR      = .
    BUILD_DIR       = build
    BUILD_POLICY    = outdated
    BUILD_TYPE      = 
    CONAN_OPTIONS   = 

  Package information:
    PACKAGE_NAME    = cloe-models
    PACKAGE_VERSION = 0.25.0-1-gb9b2f3e-dirty
    PACKAGE_CHANNEL = cloe/develop
    PACKAGE_FQN     = cloe-models/0.25.0-1-gb9b2f3e-dirty@cloe/develop
    PACKAGE_EDITABLE= not-editable
    PACKAGE_ID      = 
    PACKAGE_DIR     = 
    PACKAGE_DATE    = 
    GIT_COMMIT_HASH = b9b2f3e

make: Leaving directory '/home/docs/checkouts/readthedocs.org/user_builds/cloe/checkouts/stable/models'

conanfile.py

Conan package recipe. To gain a thorough understanding of the Conan methods and tools used therein, the Conan CMake integration and Conan recipe reference documentation is probably a good starting point.

CMakeLists.txt

Contains cross-platform instructions for how to build, test, and install the package. The configuration is done via Conan. You may come across CMake Cloe-specific CMake functions such as cloe_add_plugin, which are defined in runtime/cmake/CloePluginSetup.cmake in the top-level directory. Conan-specific declarations are contained in conanbuildinfo.cmake, which will be generated after conan install is invoked.

In-Source Builds¶

Suppose you are making changes to the basic controller plugin and you would like to build your modified package in your local working directory. To do this, you may build Conan packages in editable mode. Try it out:

cd plugins/basic/
make editable

Verify that the cloe-plugin-basic package is currently in editable mode:

make status

Note

Positive-Example:

editable : cloe-plugin-basic/0.18.0-rc5-3-g53c80db@cloe/develop

Negative-Example:

ok       : cloe-plugin-basic/0.18.0-rc5-3-g53c80db@cloe/develop

Now, you can build the package binaries in your local working directory:

make clean all

The next time when Conan needs the package cloe-plugin-basic in this version, it will resolve the include and library directories to this local build. It is important to understand that you as a developer are now responsible for ABI compatibility!!

Note

Conan can build packages with any constellation of dependencies that you may require. This means that it is necessary to build an individual package in a way that is compatible with the final composition.

For example, it may be that the entire set of packages as defined by the cloe-meta package require boost/1.65.1. When building the basic plugin as in this example, it has no way of knowing that this version of Boost will be used when building cloe-engine. Therefore Conan will use the latest version of the boost package it can find, such as boost/1.78.0.

In normal non-editable builds, Conan tracks these potential incompatibilities and prevents you from incorrect combinations. In editable mode however, you are responsible. Combining code linked to two different versions of Boost is undefined behavior and will lead to segfaults or worse!

The solution to this dilemma is to let Conan know when making a local build to use the final composition configuration for resolving dependency configurations. This can be done by generating a lockfile first of the final composition, and using this lockfile when building a part locally.

This is common enough that there is a simple mechanism baked into the Makefiles to use a Conan recipe for automatically generating and using a lockfile:

make clean all LOCKFILE_SOURCE=${CLOE_ROOT}/conanfile-meta.py

where ${CLOE_ROOT} is the path to the repository root; for the basic plugin, this is ../...

Since the package is in editable mode, the binaries will be created in the ./build/ directory in your package sub-directory.

When you finalize work on your package, remember to remove the editable mode:

make uneditable

Verify the package status:

make status

If you execute the latter command from the top-level directory, you will see the status of all Cloe packages.

Practical Example¶

Let’s apply the above to a very practical example involving cloe-engine. Assume the following situation: I checkout a develop branch, such as develop, with the intention of modifying the cloe-engine package.

First, because I am going to make changes, I disable the use of git describe for versioning by explicitely setting a version:

echo "0.99.0-develop" > VERSION

Then I make sure the entire project is exported:

make export-all

Note

This is roughtly equivalent to:

( cd fable && conan export conanfile.py fable/0.99.0-develop@cloe/develop)
( cd runtime && conan export conanfile.py cloe-runtime/0.99.0-develop@cloe/develop)
...

If there are any changes in other packages, I want to pick up those as well. I let Conan know that I want to use cloe-engine in editable mode:

cd engine
make editable

Note

This is equivalent to:

cd engine
conan editable add conanfile cloe-engine/0.99.0-develop@cloe/develop

TODO: This section

Now, I need to choose a configuration that I want to use for testing the entire set of packages. I can use a configuration in the tests/ directory, such as tests/conanfile_deployment.py. I use this when building cloe-engine as the source for creating a lockfile:

cd engine
make clean all LOCKFILE_SOURCE=../tests/conanfile_deployment.py

This will automatically build any missing dependencies that are necessary for building cloe-engine, after which it will build cloe-engine locally.

Before running any tests, I may need to make sure any additional dependencies not required by cloe-engine but required for the test execution are built:

cloe-launch prepare tests/conanfile_deployment.py

Note

This is approximately equivalent to:

conan install ../tests/conanfile_deployment.py --build=outdated --build=cascade

Once this is complete, we can launch into a correctly configured shell, with PATH and LD_LIBRARY_PATH set so that the shell can find cloe-engine and it can find required libraries and plugins.

cloe-launch shell -c tests/conanfile_deployment.py

And at this point we are done and can run tests, make modifications to the editable cloe-engine package, rebuild, and run tests again:

bats tests
$EDITOR engine/src/main.cpp
make -C engine all
bats tests

Superbuild¶

The practical example of in-source builds above was one of the easiest configurations we could choose. It becomes more arduous when we want to edit packages that other packages in turn depend on, because then we need to compile multiple packages by hand in the correct order.

To side-step all of this, we have the cloe package, which is a super-build of all other packages.

You can build in the Conan cache with:

make package

You can build it locally with:

make all

You can then launch a virtual environment with cloe-launch:

make editable
cloe-launch shell tests/conanfile_superbuild.py