Break out from bug 8172. As a part of automated signing of Windows binaries with osslsigncode, we need to add Google's KMS library [1]. More specifically, the file we need is libkmsp11.so. The library itself is build Bazel [2], which in turn requires Java. Hopefully we can use we can use OpenJDK [3]. [1] https://github.com/GoogleCloudPlatform/kms-integrations [2] https://github.com/bazelbuild/bazel [3] https://github.com/openjdk/jdk
It looks like some parts of the KMS library are written in Go, which we do not have support for in cenbuild at the moment. Hopefully, those parts are not required for the components we need to build.
The KMS library is built using Bazel, and Bazel does not behave as a package we want into cenbuild. The probably biggest issue is that it requires an internet connection, which is something we do not want in our cenbuild environment. We know some of the files Bazel downloads, and in theory it would be possible for us to download these ourselves. However, Bazel also downloads its own dependencies automagically without informing what these dependencies are. As such, including Bazel directly in cenbuild does not currently seem like an appropriate way forward. Instead, one possible option is to build KMS ourselves, e.g. by using CMake.
To build KMS ourselves, we need to find all its dependencies manually. Ideally, we would also want to find another project that have converted from building with Bazel to CMake. Regarding conversion from Bazel to CMake, we found Abseil (which is a dependency of KMS), which can build with both of these languages: https://github.com/abseil/abseil-cpp Regarding the dependencies, we started mapping these out but it has quickly started snowballing. There are several dependencies that we do not have in cenbuild. Some of them seem pretty straightforward to include, but e.g. the Google specific lib "boringssl" requires Go to build. That would require upgrading gcc, which is a big task only in itself. Boringssl is based on OpenSSL so in theory, we might get away with using that. It would, however, still be a big task to get to a stage far enough to confirm if KMS would be possible to include in cenbuild or not.
There are two other way forward that may be more realistic the way things currently are: 1) Some variant of building the KMS binary ourselves and including it in cenbuild. 2) Building an external service, in which KMS is a piece in the puzzle to automatically sign our builds. Looking into which solution that seems like a good forward.
(In reply to Linn from comment #4) > To build KMS ourselves, we need to find all its dependencies manually. > Ideally, we would also want to find another project that have converted from > building with Bazel to CMake. > > ... In addition to build KMS with CMake ourselves, for this option we also have to make sure that none of its dependencies uses Bazel to build. Unfortunately, KMS depends on project Google APIs [1], which only has Bazel as an option to build. Google APIs is a quite large project, and converting it from Bazel to CMake would likely be a big task. Note that it is likely that we only use a small part of Google APIs, but in that case we would still need to extract the relevant parts and make sure they build correctly. Seeing that the ecosystem KMS is in generally is fond of Bazel, there may be additional dependencies that only build with Bazel. Even if the projects to convert are small, it feels like another potential timesink. 1: https://github.com/googleapis/googleapis If we in the future need to find all dependencies of KMS, Bazel seems to list it dependencies quite clearly in files named WORKSPACE. This should simplify the process of finding them.
We made an attmept to build libkms the correct way in Cenbuild, which quickly snowballed. A lot of dependencies were required to build KMS, of one which being Bazel, which has problems described in comment #3 and comment #4. Bazel also requires Java to build, which is quite a large project. Some of the obstacles we encountered along way were: * /dev/fd/ missing in Cenbuild (bug 8307) * Debug stripping doesn't work in Cenbuild. Initially, we found that the documentation was incorrect, which has now been fixed [1]. This fix does not work Cenbuild, instead, we have to do the following: > %define _cendio_debug_package %{nil} * When building OpenJDK, Java insisted on using ccache during configure, which gave errors. To fix this, we simply hardcoded the CC/CXX paths to gcc/g++. * Various small patches to make things compile in Cenbuild. In the end, we were able to build libkmsp11.so for x86_64 and we encountered some errors when building for win32 which we did not investigate further. The resulting libkmsp11.so file, built for x86_64, did not work together with osslsigncode in cbrun. We tested this using the following command: > GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials.json" \ > KMS_PKCS11_CONFIG="/path/to/config.yaml" \ > PKCS11_MODULE_PATH="/usr/lib64/libkmsp11.so" \ > osslsigncode sign \ > -pkcs11engine /usr/lib64/engines-3/pkcs11.so \ > -certs /path/to/cert.pem \ > -key 'key' \ > -n "ThinLinc Client" -i "https://www.cendio.com/" -comm \ > -ts 'http://rfc3161timestamp.globalsign.com/advanced' \ > -in client-windows.exe -out signed-client-windows.exe Running the command on Fedora 38/39 works fine, but running it with cbrun x86_64 resulted in it hanging indefinitely: > $ cbrun x86_64 osslsigncode sign <...> > Engine "pkcs11" set. > > Even though at this stage we're not adding any of the dependencies to Cenbuild, the .spec files have been attached for future reference. See attachment 1181 [details]. [1] https://bugzilla.redhat.com/show_bug.cgi?id=2264161
Note that before adding all dependencies to Cenbuild, we tried to build some of our dependencies "by hand" using cbrun. In this approach, we noticed that newer versions of Bazel (e.g. 7.0.2) required gcc 7 to build, which we don't support. After some experimentation, we were able to successfully build Bazel v3.5.0 using cbrun. Using Bazel 3.5.0, we were unsuccessful in building (using cbrun) a version of KMS newer than v0.9. While the successfully compiled binary (v0.9) was usable with outside Cenbuild, it resulted in a deadlock just as in comment #8 when used with cbrun. Unfortunately, v1.0 of KMS is the first General Availability version of the KMS library (later versions seem to be incremental updates as yet). Locking ourselves into using an outdated Bazel version, and in turn an old KMS version, is not a robust solution. We also looked at any precompiled binaries we could use in Cenbuild, but the only ones available seem to require glibc-2.17, which is newer than we can currently use. This forced us to build it ourselves.
We also looked at using the server/client functionality [1] in p11-kit, that enables remote sharing of pkcs11-modules. This was done using the official release version of libkmsp11.so [2], which is compiled for 64-bit systems. [1] https://p11-glue.github.io/p11-glue/p11-kit/manual/remoting.html [2] https://github.com/GoogleCloudPlatform/kms-integrations/releases/tag/pkcs11-v1.3 Both the server and the client needed some setup to get things working. Server side configuration ######################### Make libkmsp11.so discoverable by p11-kit: 1. Place libkmsp11.so in /usr/lib64/pkcs11/ 2. Run: echo "module: /usr/lib64/pkcs11/libkmsp11.so" >/usr/share/p11-kit/modules-kms.module Start p11-kit server outside cenbuild: 1. Run: export GOOGLE_APPLICATION_CREDENTIALS=<path to credentials file> 2. Run: export KMS_PKCS11_CONFIG=<path to config file> 3. Find libkmsp11 token URI: p11-kit list-modules 4. Run: p11-kit server --name=$HOME/p11-kit-socket <token URI> 5. The server outputs two environment variables, P11_KIT_SERVER_ADDRESS and P11_KIT_SERVER_PID, where the first one needs to be set on the client side for it to find the server socket. Client side configuration ######################### Make p11-kit in cenbuild function as a client: 1. Run [3]: echo "module: /usr/lib64/pkcs11/p11-kit-client.so" >.config/pkcs11/modules/p11-kit-client.module 2. Run: export P11_KIT_SERVER_ADDRESS=$HOME/p11-kit-socket 3. Run: cbrun <arch> p11-kit list-objects <token URI> 4. Verify that the output looks to have valid data, e.g. that it doesn't show data like "0xFF8C40E0 (unknown)". [3] Note that $HOME/.config can be seen both in- and outside cenbuild, but the path to p11-kit-client.so in cenbuild may vary between arches. E.g. in win64 the path is /usr/lib/pkcs11/p11-kit-client.so.
Once the remote p11-kit server was set up, we tried signing our client exe file, see attachment 1182 [details]. Signing outside cenbuild worked well with server/client, but inside cenbuild we encountered two different issues: 1. For some reason, p11-kit-client.so in arches that are compiled for 32-bit, we get the following output when running osslsigncode: > Engine "pkcs11" set. > Unable to enumerate private keys > Unable to enumerate private keys > The private key was not found at: pkcs11:object=ThinLincKey > PKCS11_get_private_key returned NULL > Failed to load private key pkcs11:object=ThinLincKey > Failed to read key or certificates > C0E6D7F7:error:43000030:PKCS#11 module:ERR_CKR_error:Device error:p11_key.c:583: > C0E6D7F7:error:13000080:engine routines:ENGINE_load_private_key:failed loading private key:crypto/engine/eng_pkey.c:79: > Failed 2. If we instead try to run osslsigncode in e.g. arch x86_64, it hangs indefinitely. We investigated the reason for this with gdb, and it looks like the issue is in function pkcs11_get_session() in libp11 (the engine we are pointing to in osslsigncode). There is a call to pthread_cond_wait() that deadlocks/never gets fulfilled. This is likely the same hang as mentioned in comment 8.
Follow up on comment 10. While testing, we used p11-kit 0.25.3, both in cenbuild and locally.
It was decided to stop pursuing this for now. We need to prioritize other things for 4.17.0. We will continue to sign Windows binaries manually until we decide to revisit this. For ideas of other signing solutions that does not include KMS, see bug 8172 comment 17 and bug 8172 comment 18.
An issue has been created at upstream libp11 for the issue regarding indefinite hanging/deadlocking: https://github.com/OpenSC/libp11/issues/525
Reported the issue with "p11-kit list-objects <token URI>" outputting invalid token data in win64 environment of cenbuild: https://github.com/p11-glue/p11-kit/issues/623 One possible reason is due to different data models on windows and linux, see the quoted answer from the bug below: > While in theory the protocol should work with both 32-bit and 64-bit systems, it > might need to assume the same data model, in which GNU/Linux (LP64) and Windows > (LLP64) have some differences. That's one of the reasons why we don't enable > building p11-kit-client.dll on Windows by default.
