As our builds are currently done in a non-incremental fashion, there are many files that have to be needlessly rebuilt. ccache (https://ccache.dev/) is a compiler wrapper that primarily wraps C and C++ compilers such that only files that have changed are rebuilt (while the others are read cache). To see how it may impact build times, I packaged ccache for cenbuild and observed the build times being roughly cut in half for the client and slightly reduced (~20% reduction) for the server [1]. Since we initially only want ccache in our local builds (and not on the build server) we have two (or possibly more) alternatives: - Depending on cendio-build-ccache in one of the top-level cenbuild meta-packages but disabling ccache by setting the environment variable `CCACHE_DISABLE=1' on Jenkins. - Not depending on cendio-build-ccache in any of the meta packages and manually installing ccache on the workstations. [1] This was measured using ccache 3.7.12 (the last release with major number 3) instead of the latest version (4.5.1) since the latter of these managed to segfault our compiler when building the ccache rpm. There have been quite a few notable improvements since 3.7.12 so we can likely expect an even better improvement if we manage to get the latest ccache to build with our current compiler infrastructure.
It seems like ccache version 4.4 is the first that crashes our g++ 5.5.0. The line that makes the compiler segfault is pretty heavy on relatively new C++ features, so let's just stick with ccache 4.3 until the next gcc bump.
Regarding the statement made in comment 1, it seems like we still have to go with ccache 3.7.12 since the macOS assembler are missing some of the new and fancy ISA extensions used by zstd (a ccache dependency introduced in 4.0). Once we bump the macOS compiler infrastructure a upgrade to at least ccache 4.3 should be viable. I have taken a look at the release notes and, from what I can tell, 3.7.12 seem to be production-worthy.
Ccache has now been disabled for both the Jenkins client and server bundle builds. This is done as follows: > # Disable ccache and clear its cache > export CCACHE_DISABLE=1 > cbrun i386 ccache -C || true > > make <targets> > > # If ccache is available on the system, verify that the cache is still empty. > if cbrun i386 ccache --version > /dev/null 2>&1 > then > echo "Verifying that ccache was disabled during the build" > cbrun i386 ccache --print-stats | awk '/files_in_cache\t0/' | grep . > /dev/null > fi Thus, we now explicitly disable ccache, clear the cache before the actual build, as well as verify that the cache is still empty afterwards.
(In reply to William Sjöblom from comment #3) > > cbrun i386 ccache -C || true Worth noting is that the cache is located in the user home directory that is shared between all of cbruns arches, so doing these steps for i386 is sufficient.
Ccache have now been deployed in cenbuild and should provide a notable speedup out of the box for all builds done using cenbuild. Though, an even further speedup can be obtained by placing the cache on the local machine and not the NFS mount: > mv ~/.ccache/local/home/$USER/.ccache > ln -s /local/home/$USER/.ccache/home/$USER/.ccache Performance characterization ════════════════════════════ Following are some performance measurements for building the client and server with and without ccache installed. Without ccache: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Component Time ─────────────────────────────────── server-bundle 8 min 12 sec client-bundle 30 min 57 sec client-linux-rpm64 4 min 03 sec ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ With ccache: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Component Time Speedup Cache hits Cache misses Hit rate ──────────────────────────────────────────────────────────────────────────────── server-bundle 5 min 53 sec 1.58 2637 17 99.36 % client-bundle 20 min 32 sec 1.51 12733 46 99.64 % client-linux-rpm64 2 min 43 sec 1.96 1857 6 99.68 % ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Worth noting is that the bundle builds perform many steps that does not benefit from ccache, thus the rather lacking speedup for these two. Good to knows ═════════════ Clear compiler cache: > cbrun i386 ccache -C Show ccache statistics: > cbrun i386 ccache -s Clear ccache statistics: > cbrun i386 ccache -z Ccache can be disabled by setting the environment variable `CCACHE_DISABLE'. Similarly, `CCACHE_NODISABLE' can be set to trump this flag.
We should probably check that the cenbuild bootstraps correctly before marking this bug as resolved.
The job for building TigerVNC was missed when disabling ccache for Jenkins builds. Thus, while concurrently building TigerVNC and the client/server bundle the guard checking that the ccache was empty would fail. I have now both disabled ccache and added a similar guard to the TigerVNC job as well.
We have now successfully bootstrapped cenbuild with ccache in place. This bug is thus ready for testing.
There are some gains to be seen here, but as noted the bundles don't get a massive increase in actual time. However the compilation seems to be massively sped up if you dig a bit deeper: The server takes 9 minutes to build without ccache on my workstation, using 21 minutes of CPU time during that build. With ccache, and warm caches, the build drops to 7.5 minutes. Not a massive change. But the spent CPU time drops to 12 minutes, which is almost half of what was used without ccache. So for the large builds this only gives a minor improvement. But when building components it seems to help a lot. So I also tested TigerVNC: Without ccache it takes 1.5 minutes and uses about 6 minutes of CPU time. With ccache that drops to just 20 seconds build time, with 1 minute of CPU time. So for active development in one corner of the product the gains are massive.
I checked the client build, server build and tigervnc build on Jenkins and from what I can tell ccache is properly disabled there. It is also disabled in repo/rebuild for building cenbuild packages. The symlinks created to intercept calls to gcc look correct and match the actual names we have for our different gcc's. The handling to modify $PATH is a bit magical, but it works and it is documented and modular so it is probably as good as it gets. The spec file looks nice and clean. I think the cmake and sysroot dependencies are not needed though and should probably be removed.
Comment 16 is correct. The CMake and sysroot dependencies were leftovers from earlier experimentation. These have now been removed. Awaiting build system bootstrapping before marking as resolved again.
Build system bootstrapped. Marking as RESOLVED.
Everything looks good now: > - Ccache should provide a speed-up for ThinLinc builds on developer workstations Yup. See numbers above. > - Ccache should be disabled in all other cases (e.g. Jenkins builds and when building cenbuild packages) Yup. See checks above.
