Rsync boosted Buildbots

Blogs
#1

Using rsync instead of Buildbot’s own file transfers, which are known to be slow, significantly reduced the file transfer time between the Buildbot Workers and the Buildbot Master from 25.5 hours to about 18 minutes (-98%). This improvement, and a few others, enables a single “strong” Worker to build and publish Octave and all MS Windows installers within 24 hours or with four parallel Workers within 6 hours.

This is a companion discussion topic for the original entry at https://siko1056.github.io/blog/2020/11/08/octave-buildbot-boosted.html
1 Like
#2

Wrt ccache: I use ccache on the machine that I use for compiling Octave and MXE Octave. Intermittently checking ccache -s, it looks like it rarely uses the last about 8-10 GiB of the assigned cache size for me.
It looks like your buildbots perform about 20 cache cleanups for each build job. That probably means that the build time could be reduced even further by assigning a larger cache. But that is a tradeoff between build time and used disk space.
Alternatively, an admin could decide to assign builders to fixed workers. But - like you already wrote elsewhere - that would reduce coverage of the effect that different hardware might have for different builders.

Anyway, good job setting those buildbots up and having them perform pretty good. Well done!

1 Like
#3

Thanks for the hints with ccache @mmuetzel. I did not regard the cleanups and just thought the cache size is sufficient :man_facepalming: I continue the experiment with 50 GB for the .ccache folder of each worker and reset the ccache statistics to better see what is going on starting from this change.

#4

The ccache size still seems to hardly touch 8 GiB and multiple clean ups are performed per run.
I wonder why that is…

Is there enough free space for the cache?