Rsync boosted Buildbots

community
#1

Using rsync instead of Buildbot’s own file transfers, which are known to be slow, significantly reduced the file transfer time between the Buildbot Workers and the Buildbot Master from 25.5 hours to about 18 minutes (-98%). This improvement, and a few others, enables a single “strong” Worker to build and publish Octave and all MS Windows installers within 24 hours or with four parallel Workers within 6 hours.

This is a companion discussion topic for the original entry at https://siko1056.github.io/blog/2020/11/08/octave-buildbot-boosted.html
1 Like
#2

Wrt ccache: I use ccache on the machine that I use for compiling Octave and MXE Octave. Intermittently checking ccache -s, it looks like it rarely uses the last about 8-10 GiB of the assigned cache size for me.
It looks like your buildbots perform about 20 cache cleanups for each build job. That probably means that the build time could be reduced even further by assigning a larger cache. But that is a tradeoff between build time and used disk space.
Alternatively, an admin could decide to assign builders to fixed workers. But - like you already wrote elsewhere - that would reduce coverage of the effect that different hardware might have for different builders.

Anyway, good job setting those buildbots up and having them perform pretty good. Well done!

1 Like
#3

Thanks for the hints with ccache @mmuetzel. I did not regard the cleanups and just thought the cache size is sufficient :man_facepalming: I continue the experiment with 50 GB for the .ccache folder of each worker and reset the ccache statistics to better see what is going on starting from this change.

#4

The ccache size still seems to hardly touch 8 GiB and multiple clean ups are performed per run.
I wonder why that is…

Is there enough free space for the cache?

#5

@siko1056: Could you please run g++ -v on any of the workers? I wonder if this contains any paths or something else that might change for each run…

#6

Sorry, I forgot to answer on this thread :sweat: Yes 50 GB exist for sure, it must have to do with the compiler hash as you say.

# g++ -v
Using built-in specs.
COLLECT_GCC=/usr/bin/g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.5.0-3ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)
#7

Sorry. The command I gave you queries the system wide g++.
What I intended to do was to get the string from the cross compiler that is compiled by MXE Octave.
Do you clean up the MXE repository after compilation? If not, could you please cd to that directory of a 64bit target and execute the following?

./usr/bin/x86_64-w64-mingw32-g++ -v

(Or adapt the prefix if you want to check for a 32bit target.)

#8

I see, here for a 32 bit mxe

# /buildbot/octave-mxe-w32/build/usr/bin/i686-w64-mingw32-g++ -v
Using built-in specs.
COLLECT_GCC=/buildbot/octave-mxe-w32/build/usr/bin/i686-w64-mingw32-g++
COLLECT_LTO_WRAPPER=/buildbot/octave-mxe-w32/build/usr/libexec/gcc/i686-w64-mingw32/9.3.0/lto-wrapper
Target: i686-w64-mingw32
Configured with: /buildbot/octave-mxe-w32/build/tmp-build-gcc/gcc-9.3.0/configure --prefix=/buildbot/octave-mxe-w32/build/usr --enable-languages=c,c++,fortran --disable-libsanitizer --enable-version-specific-runtime-libs --with-gcc --with-gnu-ld --with-gnu-as --disable-nls --without-x --disable-win32-registry --enable-threads=posix --libdir=/buildbot/octave-mxe-w32/build/usr/lib --with-native-system-header-dir=/include --disable-sjlj-exceptions --with-specs='%{!mfpmath:-mfpmath=sse} %{!msse:%{!mno-sse:-msse}} %{!msse2:%{!mno-sse2:-msse2}}' --target=i686-w64-mingw32 --build=x86_64-pc-linux-gnu --with-as=/buildbot/octave-mxe-w32/build/usr/bin/i686-w64-mingw32-as --with-ld=/buildbot/octave-mxe-w32/build/usr/bin/i686-w64-mingw32-ld --with-nm=/buildbot/octave-mxe-w32/build/usr/bin/i686-w64-mingw32-nm --disable-multilib --with-host-libstdcxx=-lstdc++ --with-system-zlib --enable-shared --disable-static --disable-libgomp --with-cloog=/buildbot/octave-mxe-w32/build/usr --with-gmp=/buildbot/octave-mxe-w32/build/usr --with-isl=/buildbot/octave-mxe-w32/build/usr --with-mpc=/buildbot/octave-mxe-w32/build/usr --with-mpfr=/buildbot/octave-mxe-w32/build/usr
Thread model: posix
gcc version 9.3.0 (GCC)

and for a 64 bit mxe:

# /buildbot/octave-mxe-w64/build/usr/bin/x86_64-w64-mingw32-g++ -v
Using built-in specs.
COLLECT_GCC=/buildbot/octave-mxe-w64/build/usr/bin/x86_64-w64-mingw32-g++
COLLECT_LTO_WRAPPER=/buildbot/octave-mxe-w64/build/usr/libexec/gcc/x86_64-w64-mingw32/9.3.0/lto-wrapper
Target: x86_64-w64-mingw32
Configured with: /buildbot/octave-mxe-w64/build/tmp-build-gcc/gcc-9.3.0/configure --prefix=/buildbot/octave-mxe-w64/build/usr --enable-languages=c,c++,fortran --disable-libsanitizer --enable-version-specific-runtime-libs --with-gcc --with-gnu-ld --with-gnu-as --disable-nls --without-x --disable-win32-registry --enable-threads=posix --target=x86_64-w64-mingw32 --build=x86_64-pc-linux-gnu --with-as=/buildbot/octave-mxe-w64/build/usr/bin/x86_64-w64-mingw32-as --with-ld=/buildbot/octave-mxe-w64/build/usr/bin/x86_64-w64-mingw32-ld --with-nm=/buildbot/octave-mxe-w64/build/usr/bin/x86_64-w64-mingw32-nm --disable-multilib --with-host-libstdcxx=-lstdc++ --with-system-zlib --enable-64bit --disable-32bit --enable-fully-dynamic-string --enable-shared --disable-static --disable-libgomp --with-cloog=/buildbot/octave-mxe-w64/build/usr --with-gmp=/buildbot/octave-mxe-w64/build/usr --with-isl=/buildbot/octave-mxe-w64/build/usr --with-mpc=/buildbot/octave-mxe-w64/build/usr --with-mpfr=/buildbot/octave-mxe-w64/build/usr
Thread model: posix
gcc version 9.3.0 (GCC)
#9

Thanks.
Nothing suspicious (like time stamps, …) that is bound to change on every run in those strings. AFAICT, those strings are ok for fingerprinting the compiler…

Is the storage for the cache persistent between runs?

Resetting the cache might also help. (“Did you try turning it off and on again.” :wink: )

ccache -C -z
#10

Yes, the .ccache directory is located in a persistent volume, untouched from image upgrades, etc.

I get another idea from the manual. For the “Docker image system compiler” (/usr/bin/g++) we create the symlink way of using ccache and this works flawlessly

On the other hand, are the right actions performed (i.e. the prefix method), when --with-ccache is given to MXE Octave? Or we also need some symlinks here?

#11

Well, it seems to be working for my local setup without any additional preparation.
You could try and see if the links were created in ./usr/bin/ccache in the MXE Octave build tree.

My ccache.conf contains the following:

max_size = 80.0G
hash_dir = false
compiler_check = %compiler% -v

I don’t remember now why exactly I added hash_dir = false. I believe I intended to make the cache match for multiple MXE trees with similar configuration.

The ccache documentation has the following to say about it:

hash_dir ( CCACHE_HASHDIR or CCACHE_NOHASHDIR , see Boolean values above)

If true (which is the default), ccache will include the current working directory (CWD) in the hash that is used to distinguish two compilations when generating debug info (compiler option -g with variations). Exception: The CWD will not be included in the hash if base_dir is set (and matches the CWD) and the compiler option -fdebug-prefix-map is used. See also the discussion under COMPILING IN DIFFERENT DIRECTORIES.

The reason for including the CWD in the hash by default is to prevent a problem with the storage of the current working directory in the debug info of an object file, which can lead ccache to return a cached object file that has the working directory in the debug info set incorrectly.

You can disable this option to get cache hits when compiling the same source code in different directories if you don’t mind that CWD in the debug info might be incorrect.

Is the cwd different for each run?
Maybe it is worth trying with that setting?