A CMake implementation of the build system

Hi everyone,

I recently completed writing an implementation of the Octave build system using CMake (3.17+). You can check it out at https://sourceforge.net/p/gnu-octave-cmake-build/code.

The reason I wrote this is because I was finding it difficult to figure out a set up where I would be able to use code analysis and assistance features. Hence, this CMake implementation makes it easier to set up a development environment in various IDEs like CLion, Eclipse, CodeBlocks, etc. I use CLion as my daily driver, so support will be best for CLion.

This implementation also allows for native development, building, and packaging on Windows (ie. without using an MSYS shell). However, it still does require an installation of MSYS2.

Usage of Autotools is not completely eliminated because of the usage of gnulib. However, the bootstrap, configuration, and build are all handled automatically at build-time in parallel with the rest of the build.

In general, configuration performance is better than when using Autotools, and build performance is improved through the optional usage of unity builds and precompile headers as well as the Ninja build system. A lot of the configuration performance is lost though the attempted emulation of the format in which Autotools configures some files like build-env.cc. Such code could be targeted for optimization in the future.

Relatedly, there are a few areas which are very hacky because of the need to emulate the output of Autotools to ensure that various systems like mkoctfile are usable. It is also quite difficult to export a full build environment when using CMake, so the generation of some files is very complex and fragile. I don’t really have a better solution than what is in there at the moment, but I’m just going to acknowledge that the code is kind of gross.

Other features:

  • Sphinx documentation for CMake commands, variables, and modules.
  • Release distribution and NSIS Installer generation using CPack.
  • An experimental version of mkoctfile using CMake.
  • Experimental parallel testing using CTest (currently slower than the existing testing framework because of the startup time).
  • The script run-octave-env which will set up environment variables when sourced. This can make it easier to attach a debugger in various IDEs like CLion, which allows a file to be sourced prior to running the executable, or if you just want to use an executable directly from the command line.
  • Support for spaces in the source and build directory paths.
  • Multi-config support (CMake 3.20+).
  • Easier usage of lld and gold.
  • Unity builds.
  • Precompile headers.
  • Fewer reconfigurations required prior to rebuilding after modifying script files.
  • Easy creation of Octfile projects when using the CMake config module.
  • Lots of bugs, probably.

Tested systems (sub-bullets are cross-configuration targets):

  • Alpine Linux (x86_64-alpine-linux-musl)
  • Arch Linux (x86_64-pc-linux-gnu)
    • aarch64-linux-gnu
    • i686-pc-linux-gnu
    • i686-w64-mingw32
    • riscv64-linux-gnu
    • x86_64-w64-mingw32
  • Cygwin (x86_64-pc-cygwin)
  • Debian 11 (x86_64-linux-gnu)
  • Fedora 34 (x86_64-redhat-linux)
    • i686-redhat-linux
  • FreeBSD 13.0 (x86_64-unknown-freebsd13.0)
  • MSYS2 (x86_64-pc-msys)
  • MSYS2 MinGW64 (x86_64-w64-mingw32)
  • MSYS2 MinGW32 (i686-w64-mingw32)
  • MSYS2 UCRT64 (x86_64-w64-mingw32)
  • macOS Big Sur (x86_64-apple-darwin20.1.0)
  • openSUSE Leap 15.3 (x86_64-suse-linux)
  • Windows 10 (x86_64-w64-mingw32)
  • Windows Subsystem for Linux - Debian 11 (x86_64-linux-gnu)

Hopefully this is helpful to some people.

1 Like

Thank you for offering this, and I’m willing to try it out. I’ve had good experiences with Cmake in general. When I visited your link, I found it included the whole Octave repository along with the Cmake files and directories. Is there a way to get only the Cmake files and drop them in an existing clone of the Octave repository? Also, is it possible to use Cmake in parallel with autotools analogous to having parallel build directories with different configuration settings, or does it need a completely separate clone of Octave?

Is there a way to get only the Cmake files and drop them in an existing clone of the Octave repository?

Yes, you should be able to simply pull from the repository since the histories are related. The following should do the trick:

hg pull http://gharveymn@hg.code.sf.net/p/gnu-octave-cmake-build/code
hg update cmake

Also, is it possible to use Cmake in parallel with autotools analogous to having parallel build directories with different configuration settings, or does it need a completely separate clone of Octave?

No, you should be just fine as long as you’re not using the same build directory for both systems. The CMake branch does not touch the existing files written for Autotools.

My overall experience is good. A few minor things:

  1. When I tried your ssh link it asked for a password but it worked when I changed ssh to http.

  2. The first time I created a build directory for Cmake and did “cmake …” it unexpectedly ran bootstrap.sh again. But it worked after that. The messages while building were a surprising mix of Cmake’s messages and a configure script running. But it built properly.

I then deleted the Cmake build directory and created it again, then did “cmake …” and “make -jN all” where N was the number of processor threads. It took less than 20 seconds to do the “cmake …” step which is the equivalent of configure, which normally takes about 65 to 70 seconds for me. The time to do “make -jN all” was about 3 minutes 42 seconds, a full 75 seconds faster than doing it with autotools.

Rebuilds with autotools typically take me at least 10 seconds even if nothing has changed. Rebuilds with Cmake take less than a second, so the wait time is reduced a lot.

Overall I’m happy with the performance of what you’ve built. My congratulations to you.

Since I’ve not used a project that had both autotools and Cmake in parallel, do you know how to translate configure options like --disable-java etc to Cmake? Typically I’ve seen the user of -Dxxx=yyy defines during the first invocation but it didn’t guide me this time.

  1. When I tried your ssh link it asked for a password but it worked when I changed ssh to http.

Ah yeah, I forgot about the whole password thing when using ssh. I’ll edit that for posterity.

  1. The first time I created a build directory for Cmake and did “cmake …” it unexpectedly ran bootstrap.sh again. But it worked after that. The messages while building were a surprising mix of Cmake’s messages and a configure script running. But it built properly.

This is actually all expected behavior, so I’m glad that it’s working properly! The CMake system actually needs to perform its own bootstrap so that it can run Autotools on some stripped down scripts used to build gnulib. As you can guess, this also does mean that we still have to use a configure script to build gnulib, but this is all handled automatically during the build, so that’s what you were seeing there.

Note that the configuration for this bootstrap is still derived from the bootstrap.conf at the top level directory, so if you need to add or remove modules, you can still just edit that file and affect both systems.

do you know how to translate configure options like --disable-java etc to Cmake?

There are a few ways to get a list of variables affecting the build configuration. Probably the easiest way is to use cmake-gui or ccmake which will present you a categorized and interactive list of variables and their documentation.

You are also presented with a (non-comprehensive) list of variables which summarize your build configuration at the end if your first configuration. You can make this display every time with -D OCTAVE_PRINT_CONFIG=ON.

Finally, if you have Sphinx installed, you can build the octave.cmake.help target to build HTML documentation for the variables, commands, and modules which you can view using your favorite browser.

Specifically for that example, you should be able to disable Java (both the Java Native Interface and usage of javac) with -D OCTAVE_ENABLE_JAVA=OFF. You can disable just the Java Native Interface with -D OCTAVE_USE_JNI=OFF.

As a quick overview, usage of every one of the external libraries is controlled through variables of the pattern OCTAVE_USE_<NAME>. So if you didn’t want to use ARPACK, you can use -D OCTAVE_USE_ARPACK=OFF. You can also control where to search for each of the external libraries through variables of the pattern OCTAVE_USE_<NAME>_ROOT. So, again for ARPACK, you could use -D OCTAVE_USE_ARPACK_ROOT=/opt/arpack, for example.

I haven’t tested your implementation of a cmake build system for Octave yet. But it looks very interesting.
I also am quite positively impressed from my limited experience with using cmake in other projects. I am a little bit worried about the additional maintenance that adopting an additional build system would mean.
I’m not sure if we would like to drop the autotools build system at this point. If I understand some of the comments in the CMakeLists.txt file correctly, some (obscure) configurations are only supported with autotools (currently).
Having two build systems in parallel might carry the risk that they diverge from each other, or one of them rods in the future. Yet, using a more “modern” build system might come with some advantages.
@jwe: What are your thoughts on this.

I also noticed that you’ve applied a bunch of changes that aren’t directly related to the cmake build rules. Those look very interesting, too.
In particular, I’m just stumbling in the dark when it comes to symbol visibility (across platforms). Some of those patches already look helpful. I’d also appreciate any further hints and help very much!

1 Like

I’m not interested in having two distinct build systems in the Octave source tree because it will mean additional developer effort spent attempting to maintain two separate sets of files that do essentially the same job.

I’m also not interested in switching to CMake, primarily because I don’t understand how to fix problems with it. If the build works that’s great, but if something fails, I have no idea how to debug and fix it.

As for the speed of running configure, if you are repeatedly building Octave on the same system, you should see significant improvement in performance if you use either the --config-cache or the --cache-file=FILE option.

I cherry-picked a few of the changes on your fork (unrelated to cmake) and pushed them to the main repository here:
octave: cd4e03781ceb (gnu.org)
octave: e1f876747650 (gnu.org)
octave: 18f83b84897d (gnu.org)
octave: 898318e5c456 (gnu.org)
octave: ab00b8b7355f (gnu.org)
octave: 7bf5bee84c1f (gnu.org)
octave: 5822bd9d59f1 (gnu.org)
octave: 9c2c0aa0e03f (gnu.org)

Those are the “most obvious ones” (to me) that haven’t been fixed already in the meantime.
There a probably a few more that should be taken. But I’ll need a little more time to evaluate.

2 Likes