The requirements on code signing certificates have recently changed, and it is no longer allowed to have the private key stored as a plain file: https://knowledge.digicert.com/generalinformation/new-private-key-storage-requirement-for-standard-code-signing-certificates-november-2022.html This was previously a requirement for EV certificates, but is now a requirement for every type. This complicates things for us, as we've tried to make everything self-contained within the build system. It would now have to be tied to some hardware device that severely limits where we can do signed builds. I'm also concerned about how well-supported the different hardware dongles are on Linux, given how mixed support for Linux is in the smart card ecosystem. Fortunately, there appears to be the option of storing the private key "in the cloud". The signature operations are then done via an API. Multiple CAs provide this as an add-on service, but Google also has an independent service called "Cloud Key Management". And multiple are providing PKCS#11 libraries, with Linux support, as a way to access these. Putting the key in a network resource is not an issue as we already required network access for signing as we need to access public time signing servers. So, the problem then becomes whether those PKCS#11 libraries will work in our build environment. The best odds for that should be Google's library, as it is open source. It does require Bazel to build, though.
For reference, Google's library lives in this repo: https://github.com/GoogleCloudPlatform/kms-integrations
I compared the outcome of the behavior when trying to install the ThinLinc client on Windows 10 and Windows 11. Test1. I downloaded the client-bundle.zip using sftp to the Windows 10 and Windows 11 machines in the lab | Build | Signed | |------------------------------|---------------------- | 1| Jenkins tl-4.14.0post | Yes, but invalid cert | |------------------------------|---------------------- | 2| alex tl-4.14.0post | Unsigned | |------------------------------|-----------------------| 3| tl-4.14.0-client-windows.exe*| Yes. Expired cert. | 1 & 2: Windows10/11 popups "Do you want to allow this app from an unknown publisher to make changes to your device?" with yellow header, indicating that this app comes from an uknown publisher. Providing the admin user and password let's the user continue with the installation 3: Same as above, but the publisher is known, and the phrasing is a bit different in the popup: "Do you want to allow this app to make changes to your device". The popup also shows information about the publisher: "Verified publisher: Cendio Aktiebolag". Also the header is blue. Providing the admin user and password let's the user continue with the installation. Test2 Downloading the same bundles but over the web. This is how the users will obtain the client bundle. 1 & 2: Windows10/Windows11 know presents smartscreen popup: "Windows protected your PC. Microsoft Defender SmartScreen prevented an unrecognized app from starting. Running this app might put your PC at risk. More info" The user is presented with a single button, "Don't run" unless clicking "More info" and then clicking "Run anyway". Then it continues with presenting the same dialogs as 1 & 2 in Test 1
Created attachment 1130 [details] UAC_Yellow
Created attachment 1131 [details] SmartScreen
Created attachment 1132 [details] UAC_Blue_Verified_Publisher
One hope we had was that, by uploading our client to Microsoft Store, that client would automatically be signed by Microsoft. Unfortunately, that is not the case. We get the yellow smart screen warning for the Microsoft Store client as well.
Right now, we are able to do the signing of our Windows client installer manually. We want to be able to do signing automatically as a part of our build process, and there are a couple of things that need to be in place before we can do this: * osslsigncode: * cmake (>= 3.17) * openssl-pkcs11: * p11-kit-trust: * This is possibly already part of our p11-kit build. * kms integrations: * https://github.com/GoogleCloudPlatform/kms-integrations/tree/master * bazel (build system) * Java: * Possibly https://github.com/openjdk/jdk/blob/master/doc/building.md
I created a couple of bugs for the major dependencies we have to solve before we can use osslsigncode in cenbuild: * Bug 8298 (Upgrade CMake) * Bug 8299 (Build openssl-pkcs11) * Bug 8300 (Build libkmsp11)
The osslsigncode package has now been upgraded to version 2.6. I verified that our old method for signing is still working in the new version.
I've created a ticket upstream about the built-in tests not being compatible with our cenbuild Python version. The patch we apply to the build right now simply removes the entire test suite. We should concider patching the tests. See the diff in the upstream issue [1]. [1] https://github.com/mtrojnar/osslsigncode/issues/349
Aside from the certificate, we need a private key to sign our Windows binaries. As described in comment 0, this can no longer be stored locally. We are now subscribed to a service from Google that allows us to store this key in the cloud. The service is “Cloud HSM” (Hardware-Security Module) and is available as part of “Google Cloud KMS” (Key Management Service). The certificate, which we bought from GlobalSign, was expensive. The certificate is linked to the key hosted on KMS. However, they have said that it is possible to “re-issue” a new key. That means we are probably not locked to using Google's KMS.
It seems very challenging to include Google's KMS client library into cenbuild (see bug 8300). However, one option we still have, that would allow us to potentially continue using Google's KMS, is to write our own client library. The overall process is described well here: https://icedev.pl/posts/setting-up-ev-code-signing-google-hsm-fips-140-2/ Hopefully, it would not require too much work. From what we can understand, we need to be able to do two things: * Enumerate the keys on Google's HSM * Execute the cryptographic operation that will give us the signature It is possible that we don't even need to write a PKCS#11 module here, we might be able to create an OpenSSL engine and use that interface instead. If we're writing our own client library, we have full control on requirements and would ensure that we build something that fits perfectly in our environments. Google's implementation can be found here: https://github.com/GoogleCloudPlatform/kms-integrations Code showing how to use Google's KMS client library from, for example Python, can be found here: https://cloud.google.com/kms/docs/create-validate-signatures#kms-sign-asymmetric-python
Adam found a project which is a PKCS#11 provider which interfaces with AWS KMS (yes — Amazon, not Google): https://github.com/JackOfMostTrades/aws-kms-pkcs11 It could be useful as inspiration if we decide to write our own library for interfacing with Google's KMS. Alternatively, if we decide to abandon Google's service, we could look into using Amazon's counterpart. In that case, we could investigate if aws-kms-pkcs11 is easier to integrate into cenbuild (from a glance it seems to require an AWS SDK).
It was decided to stop pursuing this for now. We need to prioritize other things for 4.17.0. We will continue to sign Windows binaries manually until we decide to revisit this.
I have set up a virtual machine where manual signing has been streamlined a bit. Our wiki page for osslsigncode has been updated to reflect this. Keep in mind that we see this as a temporary fix that can be used until we have automatic signing in place as a part of the build procedure.
