Bug 8300 - Missing Google's Cloud KMS Library in cenbuild
Summary: Missing Google's Cloud KMS Library in cenbuild
Status: NEW
Alias: None
Product: ThinLinc
Classification: Unclassified
Component: Build system (show other bugs)
Version: trunk
Hardware: PC Unknown
: P2 Normal
Target Milestone: MediumPrio
Assignee: Bugzilla mail exporter
Depends on:
Blocks: 8172
  Show dependency treegraph
Reported: 2024-01-30 16:35 CET by Alexander Zeijlon
Modified: 2024-02-23 16:14 CET (History)
2 users (show)

See Also:
Acceptance Criteria:
MUST: The KMS library should be added to x86_64 Cenbuild - The library (libkmsp11.so file) should be usable in Cenbuild SHOULD: x86_64 Cenbuild should be able to build the KMS Library - Building this library generates libkmsp11.so


Description Alexander Zeijlon cendio 2024-01-30 16:35:50 CET
Break out from bug 8172.

As a part of automated signing of Windows binaries with osslsigncode, we need to add Google's KMS library [1]. More specifically, the file we need is libkmsp11.so.

The library itself is build Bazel [2], which in turn requires Java. Hopefully we can use we can use OpenJDK [3].

[1] https://github.com/GoogleCloudPlatform/kms-integrations
[2] https://github.com/bazelbuild/bazel
[3] https://github.com/openjdk/jdk
Comment 1 Alexander Zeijlon cendio 2024-01-30 16:42:42 CET
It looks like some parts of the KMS library are written in Go, which we do not have support for in cenbuild at the moment.

Hopefully, those parts are not required for the components we need to build.
Comment 3 Linn cendio 2024-02-15 15:22:06 CET
The KMS library is built using Bazel, and Bazel does not behave as a package we want into cenbuild.

The probably biggest issue is that it requires an internet connection, which is something we do not want in our cenbuild environment. We know some of the files Bazel downloads, and in theory it would be possible for us to download these ourselves. However, Bazel also downloads its own dependencies automagically without informing what these dependencies are.

As such, including Bazel directly in cenbuild does not currently seem like an appropriate way forward. Instead, one possible option is to build KMS ourselves, e.g. by using CMake.
Comment 4 Linn cendio 2024-02-15 15:44:49 CET
To build KMS ourselves, we need to find all its dependencies manually. Ideally, we would also want to find another project that have converted from building with Bazel to CMake.

Regarding conversion from Bazel to CMake, we found Abseil (which is a dependency of KMS), which can build with both of these languages:

Regarding the dependencies, we started mapping these out but it has quickly started snowballing. There are several dependencies that we do not have in cenbuild. Some of them seem pretty straightforward to include, but e.g. the Google specific lib "boringssl" requires Go to build. That would require upgrading gcc, which is a big task only in itself. 

Boringssl is based on OpenSSL so in theory, we might get away with using that. It would, however, still be a big task to get to a stage far enough to confirm if KMS would be possible to include in cenbuild or not.
Comment 5 Linn cendio 2024-02-15 15:58:02 CET
There are two other way forward that may be more realistic the way things currently are:

1) Some variant of building the KMS binary ourselves and including it in cenbuild.

2) Building an external service, in which KMS is a piece in the puzzle to automatically sign our builds.

Looking into which solution that seems like a good forward.
Comment 6 Linn cendio 2024-02-20 14:39:14 CET
(In reply to Linn from comment #4)
> To build KMS ourselves, we need to find all its dependencies manually.
> Ideally, we would also want to find another project that have converted from
> building with Bazel to CMake.
> ...
In addition to build KMS with CMake ourselves, for this option we also have to make sure that none of its dependencies uses Bazel to build.

Unfortunately, KMS depends on project Google APIs [1], which only has Bazel as an option to build. Google APIs is a quite large project, and converting it from Bazel to CMake would likely be a big task. Note that it is likely that we only use a small part of Google APIs, but in that case we would still need to extract the relevant parts and make sure they build correctly.

Seeing that the ecosystem KMS is in generally is fond of Bazel, there may be additional dependencies that only build with Bazel. Even if the projects to convert are small, it feels like another potential timesink.

1: https://github.com/googleapis/googleapis

If we in the future need to find all dependencies of KMS, Bazel seems to list it dependencies quite clearly in files named WORKSPACE. This should simplify the process of finding them.
Comment 8 Adam Halim cendio 2024-02-20 16:14:52 CET
We made an attmept to build libkms the correct way in Cenbuild, which quickly

A lot of dependencies were required to build KMS, of one which being Bazel,
which has problems described in comment #3 and comment #4. Bazel also requires
Java to build, which is quite a large project.

Some of the obstacles we encountered along way were:
 * /dev/fd/ missing in Cenbuild (bug 8307)
 * Debug stripping doesn't work  in Cenbuild. Initially, we found that the
   documentation was incorrect, which has now been fixed [1].
   This fix does not work Cenbuild, instead, we have to do the following:
> %define _cendio_debug_package %{nil}
 * When building OpenJDK, Java insisted on using ccache during configure, which
   gave errors. To fix this, we simply hardcoded the CC/CXX paths to gcc/g++.
 * Various small patches to make things compile in Cenbuild.

In the end, we were able to build libkmsp11.so for x86_64 and we encountered
some errors when building for win32 which we did not investigate further. The
resulting libkmsp11.so file, built for x86_64, did not work together with
osslsigncode in cbrun. We tested this using the following command:

> GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials.json" \
> KMS_PKCS11_CONFIG="/path/to/config.yaml" \
> PKCS11_MODULE_PATH="/usr/lib64/libkmsp11.so" \
> osslsigncode sign \
> -pkcs11engine /usr/lib64/engines-3/pkcs11.so \
> -certs /path/to/cert.pem  \
> -key 'key' \
> -n "ThinLinc Client" -i "https://www.cendio.com/" -comm \
> -ts 'http://rfc3161timestamp.globalsign.com/advanced' \
> -in client-windows.exe -out signed-client-windows.exe
Running the command on Fedora 38/39 works fine, but running it with cbrun
x86_64 resulted in it hanging indefinitely:
> $ cbrun x86_64 osslsigncode sign <...>
>   Engine "pkcs11" set.
Even though at this stage we're not adding any of the dependencies to Cenbuild,
the .spec files have been attached for future reference. See attachment 1181 [details].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2264161
Comment 9 Adam Halim cendio 2024-02-20 17:18:47 CET
Note that before adding all dependencies to Cenbuild, we tried to
build some of our dependencies "by hand" using cbrun. In this approach, we
noticed that newer versions of Bazel (e.g. 7.0.2) required gcc 7 to build,
which we don't support. After some experimentation, we were able to
successfully build Bazel v3.5.0 using cbrun.

Using Bazel 3.5.0, we were unsuccessful in building (using cbrun) a version of
KMS newer than v0.9. While the successfully compiled binary (v0.9) was
usable with  outside Cenbuild, it resulted in a deadlock just as in 
comment #8 when used with cbrun.

Unfortunately, v1.0 of KMS is the first General Availability version of the KMS
library (later versions seem to be incremental updates as yet). Locking
ourselves into using an outdated Bazel version, and in turn an old KMS version, is
not a robust solution.

We also looked at any precompiled binaries we could use in Cenbuild, but the
only ones available seem to require glibc-2.17, which is newer than we can
currently use. This forced us to build it ourselves.
Comment 10 Alexander Zeijlon cendio 2024-02-21 10:06:09 CET
We also looked at using the server/client functionality [1] in p11-kit, that enables remote sharing of pkcs11-modules. This was done using the official release version of libkmsp11.so [2], which is compiled for 64-bit systems.

[1] https://p11-glue.github.io/p11-glue/p11-kit/manual/remoting.html
[2] https://github.com/GoogleCloudPlatform/kms-integrations/releases/tag/pkcs11-v1.3

Both the server and the client needed some setup to get things working.

Server side configuration

Make libkmsp11.so discoverable by p11-kit:

1. Place libkmsp11.so in /usr/lib64/pkcs11/
2. Run: echo "module: /usr/lib64/pkcs11/libkmsp11.so" >/usr/share/p11-kit/modules-kms.module

Start p11-kit server outside cenbuild:

1. Run: export GOOGLE_APPLICATION_CREDENTIALS=<path to credentials file>
2. Run: export KMS_PKCS11_CONFIG=<path to config file>
3. Find libkmsp11 token URI: p11-kit list-modules
4. Run: p11-kit server --name=$HOME/p11-kit-socket <token URI>
5. The server outputs two environment variables, P11_KIT_SERVER_ADDRESS and P11_KIT_SERVER_PID, where the first one needs to be set on the client side for it to find the server socket.

Client side configuration

Make p11-kit in cenbuild function as a client:

1. Run [3]: echo "module: /usr/lib64/pkcs11/p11-kit-client.so" >.config/pkcs11/modules/p11-kit-client.module
2. Run: export P11_KIT_SERVER_ADDRESS=$HOME/p11-kit-socket
3. Run: cbrun <arch> p11-kit list-objects <token URI>
4. Verify that the output looks to have valid data, e.g. that it doesn't show data like "0xFF8C40E0 (unknown)".

[3] Note that $HOME/.config can be seen both in- and outside cenbuild, but the path to p11-kit-client.so in cenbuild may vary between arches. E.g. in win64 the path is /usr/lib/pkcs11/p11-kit-client.so.
Comment 12 Alexander Zeijlon cendio 2024-02-21 10:16:46 CET
Once the remote p11-kit server was set up, we tried signing our client exe file, see attachment 1182 [details].

Signing outside cenbuild worked well with server/client, but inside cenbuild we encountered two different issues: 

1. For some reason, p11-kit-client.so in arches that are compiled for 32-bit, we get the following output when running osslsigncode:
> Engine "pkcs11" set.
> Unable to enumerate private keys
> Unable to enumerate private keys
> The private key was not found at: pkcs11:object=ThinLincKey
> PKCS11_get_private_key returned NULL
> Failed to load private key pkcs11:object=ThinLincKey
> Failed to read key or certificates
> C0E6D7F7:error:43000030:PKCS#11 module:ERR_CKR_error:Device error:p11_key.c:583:
> C0E6D7F7:error:13000080:engine routines:ENGINE_load_private_key:failed loading private key:crypto/engine/eng_pkey.c:79:
> Failed
2. If we instead try to run osslsigncode in e.g. arch x86_64, it hangs indefinitely. We investigated the reason for this with gdb, and it looks like the issue is in function pkcs11_get_session() in libp11 (the engine we are pointing to in osslsigncode). There is a call to pthread_cond_wait() that deadlocks/never gets fulfilled. This is likely the same hang as mentioned in comment 8.
Comment 13 Alexander Zeijlon cendio 2024-02-21 10:31:32 CET
Follow up on comment 10.

While testing, we used p11-kit 0.25.3, both in cenbuild and locally.
Comment 14 Linn cendio 2024-02-22 08:45:33 CET
It was decided to stop pursuing this for now. We need to prioritize other things for 4.17.0. We will continue to sign Windows binaries manually until we decide to revisit this.

For ideas of other signing solutions that does not include KMS, see bug 8172 comment 17 and bug 8172 comment 18.
Comment 15 Alexander Zeijlon cendio 2024-02-22 14:27:42 CET
An issue has been created at upstream libp11 for the issue regarding indefinite hanging/deadlocking: https://github.com/OpenSC/libp11/issues/525
Comment 16 Linn cendio 2024-02-23 16:14:51 CET
Reported the issue with "p11-kit list-objects <token URI>" outputting invalid token data in win64 environment of cenbuild:

One possible reason is due to different data models on windows and linux, see the quoted answer from the bug below:
> While in theory the protocol should work with both 32-bit and 64-bit systems, it
> might need to assume the same data model, in which GNU/Linux (LP64) and Windows
> (LLP64) have some differences. That's one of the reasons why we don't enable
> building p11-kit-client.dll on Windows by default.

Note You need to log in before you can comment on or make changes to this bug.