Bug 7827 - Builds are non-incremental and can benefit from ccache
Summary: Builds are non-incremental and can benefit from ccache
Alias: None
Product: ThinLinc
Classification: Unclassified
Component: Build system (show other bugs)
Target Milestone: 4.15.0
Assignee: William Sjöblom
Keywords: ossman_tester, prosaic
Depends on:
Reported: 2022-01-31 12:37 CET by William Sjöblom
Modified: 2022-05-17 15:52 CEST (History)
0 users

See Also:
Acceptance Criteria:
- Ccache should provide a speed-up for ThinLinc builds on developer workstations - Ccache should be disabled in all other cases (e.g. Jenkins builds and when building cenbuild packages)


Description William Sjöblom cendio 2022-01-31 12:37:14 CET
As our builds are currently done in a non-incremental fashion, there are many files that have to be needlessly rebuilt. ccache (https://ccache.dev/) is a compiler wrapper that primarily wraps C and C++ compilers such that only files that have changed are rebuilt (while the others are read cache).

To see how it may impact build times, I packaged ccache for cenbuild and observed the build times being roughly cut in half for the client and slightly reduced (~20% reduction) for the server [1]. 

Since we initially only want ccache in our local builds (and not on the build server) we have two (or possibly more) alternatives:

- Depending on cendio-build-ccache in one of the top-level cenbuild 
  meta-packages but disabling ccache by setting the environment variable
  `CCACHE_DISABLE=1' on Jenkins.
- Not depending on cendio-build-ccache in any of the meta packages and 
  manually installing ccache on the workstations.

[1] This was measured using ccache 3.7.12 (the last release with major number 3) instead of the latest version (4.5.1) since the latter of these managed to segfault our compiler when building the ccache rpm. There have been quite a few notable improvements since 3.7.12 so we can likely expect an even better improvement if we manage to get the latest ccache to build with our current compiler infrastructure.
Comment 1 William Sjöblom cendio 2022-01-31 15:53:35 CET
It seems like ccache version 4.4 is the first that crashes our g++ 5.5.0. The line that makes the compiler segfault is pretty heavy on relatively new C++ features, so let's just stick with ccache 4.3 until the next gcc bump.
Comment 2 William Sjöblom cendio 2022-02-15 10:27:03 CET
Regarding the statement made in comment 1, it seems like we still have to go with ccache 3.7.12 since the macOS assembler are missing some of the new and fancy ISA extensions used by zstd (a ccache dependency introduced in 4.0). Once we bump the macOS compiler infrastructure a upgrade to at least ccache 4.3 should be viable. I have taken a look at the release notes and, from what I can tell, 3.7.12 seem to be production-worthy.
Comment 3 William Sjöblom cendio 2022-02-16 10:47:50 CET
Ccache has now been disabled for both the Jenkins client and server bundle builds.
This is done as follows:
> # Disable ccache and clear its cache
> cbrun i386 ccache -C || true
> make <targets>
> # If ccache is available on the system, verify that the cache is still empty.
> if cbrun i386 ccache --version > /dev/null 2>&1
> then
>     echo "Verifying that ccache was disabled during the build"
>     cbrun i386 ccache --print-stats | awk '/files_in_cache\t0/' | grep . > /dev/null
> fi
Thus, we now explicitly disable ccache, clear the cache before the actual build, as well as verify that the cache is still empty afterwards.
Comment 4 William Sjöblom cendio 2022-02-16 10:53:41 CET
(In reply to William Sjöblom from comment #3)
> > cbrun i386 ccache -C || true
Worth noting is that the cache is located in the user home directory that is shared between all of cbruns arches, so doing these steps for i386 is sufficient.
Comment 9 William Sjöblom cendio 2022-02-16 17:06:56 CET
Ccache have now been deployed in cenbuild and should provide a notable
speedup out of the box for all builds done using cenbuild. Though, 
an even further speedup can be obtained by placing the cache on the 
local machine and not the NFS mount: 
> mv ~/.ccache/local/home/$USER/.ccache 
> ln -s /local/home/$USER/.ccache/home/$USER/.ccache

Performance characterization

  Following are some performance measurements for building the client
  and server with and without ccache installed.

  Without ccache:
   Component           Time          
   server-bundle       8 min 12 sec  
   client-bundle       30 min 57 sec 
   client-linux-rpm64  4 min 03 sec  
  With ccache:
   Component           Time           Speedup  Cache hits  Cache misses  Hit rate 
   server-bundle       5 min 53 sec      1.58        2637            17  99.36 %  
   client-bundle       20 min 32 sec     1.51       12733            46  99.64 %  
   client-linux-rpm64  2 min 43 sec      1.96        1857             6  99.68 %  

  Worth noting is that the bundle builds perform many steps that does
  not benefit from ccache, thus the rather lacking speedup for these

Good to knows

  Clear compiler cache: 
>   cbrun i386 ccache -C 
  Show ccache statistics: 
>   cbrun i386 ccache -s 
  Clear ccache statistics: 
>   cbrun i386 ccache -z

  Ccache can be disabled by setting the environment variable
  `CCACHE_DISABLE'.  Similarly, `CCACHE_NODISABLE' can be set to trump
  this flag.
Comment 10 William Sjöblom cendio 2022-02-16 17:11:30 CET
We should probably check that the cenbuild bootstraps correctly before marking this bug as resolved.
Comment 13 William Sjöblom cendio 2022-03-07 10:33:24 CET
The job for building TigerVNC was missed when disabling ccache for Jenkins builds. Thus, while concurrently building TigerVNC and the client/server bundle the guard checking that the ccache was empty would fail. I have now both disabled ccache and added a similar guard to the TigerVNC job as well.
Comment 14 William Sjöblom cendio 2022-03-28 09:58:50 CEST
We have now successfully bootstrapped cenbuild with ccache in place. This bug is thus ready for testing.
Comment 15 Pierre Ossman cendio 2022-03-29 10:47:45 CEST
There are some gains to be seen here, but as noted the bundles don't get a massive increase in actual time. However the compilation seems to be massively sped up if you dig a bit deeper:

The server takes 9 minutes to build without ccache on my workstation, using 21 minutes of CPU time during that build.

With ccache, and warm caches, the build drops to 7.5 minutes. Not a massive change. But the spent CPU time drops to 12 minutes, which is almost half of what was used without ccache.

So for the large builds this only gives a minor improvement. But when building components it seems to help a lot.

So I also tested TigerVNC:

Without ccache it takes 1.5 minutes and uses about 6 minutes of CPU time. With ccache that drops to just 20 seconds build time, with 1 minute of CPU time.

So for active development in one corner of the product the gains are massive.
Comment 16 Pierre Ossman cendio 2022-03-29 15:39:09 CEST
I checked the client build, server build and tigervnc build on Jenkins and from what I can tell ccache is properly disabled there.

It is also disabled in repo/rebuild for building cenbuild packages.

The symlinks created to intercept calls to gcc look correct and match the actual names we have for our different gcc's.

The handling to modify $PATH is a bit magical, but it works and it is documented and modular so it is probably as good as it gets.

The spec file looks nice and clean. I think the cmake and sysroot dependencies are not needed though and should probably be removed.
Comment 18 William Sjöblom cendio 2022-03-30 15:58:13 CEST
Comment 16 is correct. The CMake and sysroot dependencies were leftovers from earlier experimentation. These have now been removed. 

Awaiting build system bootstrapping before marking as resolved again.
Comment 19 William Sjöblom cendio 2022-03-31 14:23:46 CEST
Build system bootstrapped. Marking as RESOLVED.
Comment 20 Pierre Ossman cendio 2022-03-31 14:34:12 CEST
Everything looks good now:

> - Ccache should provide a speed-up for ThinLinc builds on developer workstations

Yup. See numbers above.

> - Ccache should be disabled in all other cases (e.g. Jenkins builds and when building cenbuild packages)

Yup. See checks above.

Note You need to log in before you can comment on or make changes to this bug.