We have run into a problem where programs that require CA-certificates will not work from inside our build environment. This is because we do not have any cendio-build-packages that provide CA-certificates inside the chrooted environment. One example of where this issue can be seen is with the new Transifex tx cli tool that will be added to cenbuild in bug 7995. The cli tool will exit with the error: > Error while fetching resource: Get "https://...": x509: > certificate signed by unknown authority if no CA-certificates can be found.
Fedora and Ubuntu provide ca-certs through their respective ca-certificates packages. Both distributions use the file certdata.txt from Mozilla as their source of trusted certificates, but Fedora uses a more modern approach that uses p11-kit to generate certificate bundles for different purposes from a "trust store" source file. We will follow this approach when generating our own ca-certificates package.
If we do not consider everything related to legacy certificates, then Fedora's procedure is as follows when generating their ca-certificates package (simplified): 1. Use the script certdata2pem.py [A] to generate p11-kit-files [B] for each individual certificate in certdata.txt [A]. 2. Append all generated p11-kit-files into one file called ca-bundle.trust.p11-kit (This file is referred to as a trust store by p11-kit after install). 3. Package ca-bundle.trust.p11-kit together with the shell script update-ca-trust [A]. [A] Filename exists in the source for Fedora's ca-certificates package. [B] In addition to actual certificates, p11-kit-files also contain metadata. When the ca-certificates package is installed, the following happens (simplified): 1. ca-bundle.trust.p11-kit is installed to the path /usr/share/pki/ca-trust-source/ [C] update-ca-trust is installed to /usr/bin/ 2. update-ca-trust is run as a post-install script by the package itself. This is what generates the actual certificate files, that for example OpenSSL and curl needs when verifying the validity of external sites. [C] This path needs to match the trust-path, configured in p11-kit. Otherwise, all certificates generated by update-ca-trust will be empty.
If we do not consider everything related to legacy certificates, then Fedora's procedure is as follows when generating their ca-certificates package (simplified): 1. Use the script certdata2pem.py [A] to generate p11-kit-files [B] for each individual certificate in certdata.txt [A]. 2. Append all generated p11-kit-files into one file called ca-bundle.trust.p11-kit (This file is referred to as a trust store by p11-kit after install). 3. Package ca-bundle.trust.p11-kit together with the shell script update-ca-trust [A]. [A] Filename exists in the source for Fedora's ca-certificates package. [B] In addition to actual certificates, p11-kit-files also contain metadata. When the ca-certificates package is installed, the following happens (simplified): 1. ca-bundle.trust.p11-kit is installed to /usr/share/pki/ca-trust-source/ [C] 2. update-ca-trust is installed to /usr/bin/ 3. update-ca-trust is run as a post-install script by the package itself. This is what generates the actual certificate files, that for example OpenSSL and curl needs when verifying the validity of external sites. [C] This path needs to match the trust-path, configured in p11-kit. Otherwise, all certificates generated by update-ca-trust will be empty.
Our ca-certificates package will depend on p11-kit, which we also need to package for our build environment. For our use case, p11-kit has a these dependencies: +----------+------------------------------+ | Lib name | Status | +----------+------------------------------+ | libtasn1 | Already packaged in cenbuild | | libffi | Already packaged in cenbuild | | gettext | Already packaged in cenbuild | +----------+------------------------------+
Note that Fedora's reason for installing the script update-ca-trust, instead of running its contents directly as a post-install script [1], is to allow users/admins to add their own root certificates after the ca-certificates package has been installed. This is not something we are interested in doing in our build environment right now, which means that we can run the equivalent of update-ca-trust directly as a post-install script. [1] The post-install script is defined by the %post-macro directly in rpmbuild's spec-files, instead of us having to create a separate shell script.
When trying to add the ca-certificates package to the build system, we noticed that it has a circular dependency with the OpenSSL package. 1. ca-certificates has a build requirement on OpenSSL. 2. OpenSSL has an installation requirement on ca-certificates. This means that building ca-certificates, requires us to install OpenSSL, which in turn requires ca-certificates to be installed. Resolving this is difficult in our build system since in it, there are no previous package versions of ca-certificates that can be used as indirect build requirements. When Fedora builds their packages, they already have the build requirements available in the form of previous versions of the packages that they are trying to build. Hence, they get around this circular dependency issue, since they are already in a bootstrapped state.
As a workaround for comment 6, we decided to put the requirement for ca-certificates in clientdeps and serverdeps. This works since there is no package in the build system that requires ca-certificates during the building stage of the build system. Currently, the only time ca-certificates is needed is during manual work, e.g. when using the Transifex CLI tool. Because of this, we can still get things working the way we want without the dependencies being fully "correct". In the future, we should ideally fix the dependency structure so OpenSSL and ca-certificates can have their proper requirements fulfilled, without using clientdeps or serverdeps as a proxy.
For Fedora's ca-certificates package, a bunch of pem-files are generated in /etc/pki/ca-trust/extracted/directory-hash/ by update-ca-trust. These pem-files are difficult to take ownership of in the ca-certificates package. This is problematic since we don't know the name of these files during the package build, which is required by rpmbuild's %ghost macro to properly take ownership of files created post-install. Because of this, the pem-files are currently not generated by our ca-certificates package. OpenSSL and GnuTLS seem to be working fine without them.
Testing: OpenSSL was verified to correctly work with ca-certificate bundles by running the follwing command: > echo "Q" | cbrun <arch> openssl s_client -connect google.com:443 -brief which without ca-certificates results in an output that contains the message: > Verification error: unable to get local issuer certificate When OpenSSL finds the certificates, it instead contains the message: > Verification: OK GnuTLS was verified to correctly work with ca-certificate bundles by running the follwing command: > echo "Q" | cbrun <arch> gnutls-cli google.com:443 which without ca-certificates results in an output that contains the message: > - Status: The certificate is NOT trusted. The certificate issuer is unknown. > *** PKI verification of server certificate failed... > *** Fatal error: Error in the certificate. When GnuTLS finds the certificates, it instead contains the message: > - Status: The certificate is trusted. Overview of tested configurations: > |--------+----------------------+------------+---------------------+-----------| > | arch | OpenSSL[1] (no cert) | OpenSSL[1] | GnuTLS[2] (no cert) | GnuTLS[2] | > |--------+----------------------+------------+---------------------+-----------| > | x86_64 | Doesn't work | Works | Doesn't work | Works | > | i386 | Doesn't work | Works | Doesn't work | Works | > | armhf | Doesn't work | Works | [3] | N/A | > | osx64 | Doesn't work | Works | [3] | N/A | > | win32 | Doesn't work | Works | [3] | N/A | > | win64 | Doesn't work | Works | [3] | N/A | > |--------+----------------------+------------+---------------------+-----------| [1] The package was modified such that it sets OPENSSLDIR to point to /etc/pki/tls/ where the ca-certificates package now stores certificate bundles. [2] The package was modified, where the build flag --with-default-trust-store-file was added that now points to one of the certificate bundles that is generated by the ca-certificates package. [3] The package does not contain the command gnutls-cli since there is nothing that requires native GnuTLS in this arch. Since the context we are testing in is always native, the expectation would be that the result for this arch would be the same as for x86_64 and i386.
Additionally, for OpenSSL we have tested that other packages are still working after we have redirected its OPENSSLDIR from /usr/ssl/ to /etc/pki/tls/. We tested this by running the follwing python3 script [1]: > import socket > import ssl > > hostname = 'www.python.org' > context = ssl.create_default_context() > > with socket.create_connection((hostname, 443)) as sock: > with context.wrap_socket(sock, server_hostname=hostname) as ssock: > print(ssock.version()) [1] Found at https://docs.python.org/3/library/ssl.html. We also verified our osslsigncode command by downloading a signed windows client from jenkins and then runnign: > cbrun win64 osslsigncode verify tl-4.14.0post-client-windows.exe
The initial problem mentioned in comment 0 have now been tested as follows: 1. Make sure that the new Transifex client (v. 1.6.7) is installed in our build environment. 2. Go to /ctc-git/client/metainfo/ 3. Run: cbrun i386 tx push --source 4. If tx asks for a token, then add some random combination of letters. You should then see this error: > thinlinc.tlclient-metainfo-pot - Error while fetching resource: 401, Unauthorized: > Invalid token. Even if the above is an error message. It shows that the tx client was able to connect over TLS to Transifex's site, which means that it is able to find the certificates we are installing with the ca-certificates package.
This bug is now solved.
There was a mistake made around synched rpm-packages where some dependencies are not correct. Reopening.
We found a bug when testing reinstallation of ca-cert package due to how the symlinks were created in the post install script. This script has been updated to only create the symlinks if they do not already exist. We also found that Fedora's package creates symlinks in /etc/ssl/ for compatibility purposes, so creation of those symlinks were also added to our ca-cert package.
(In reply to Alexander Zeijlon from comment #21) > We found a bug when testing reinstallation of ca-cert package due to how the > symlinks were created in the post install script. This script has been > updated to only create the symlinks if they do not already exist. Tested by reinstalling ca-cert package, and we no longer get errors when it runs the post install script. > We also found that Fedora's package creates symlinks in /etc/ssl/ for > compatibility purposes, so creation of those symlinks were also added to our > ca-cert package. Checked that /etc/ssl/ and all its content were created on install and removed on uninstall. The created symlinks were also referencing the corresponding files. Note that the referenced files for all created symlinks also exists locally on Fedora. To see that they were correctly installed, check the references relative to each arch's root directory (i.e. relative to /opt/cendio-build/arch/<arch_name>/).
(In reply to Alexander Zeijlon from comment #15) > There was a mistake made around synched rpm-packages where some dependencies > are not correct. Reopening. The dependency on ca-cert was previously attached to transifex-client package. While this technically is correct, the dependency has been moved to the everything package to better signal that ca-cert currently has a workaround for a circular dependency in place (see comment 7 for context).
We also noticed that some package were not synced between our rpm repo and our svn repo. After some investigation, all the out-of-sync package have had their release number incremented to accomplish a synced state.
Testing p11-kit: Since ca-cert uses p11-kit to generate certificate bundles, we know that p11-kit works if those bundles exist and contain certificates. In order for ca-cert to work, its "trust store" source file also need to be installed where p11-kit is configured to look for it. We tested this by checking that the files generated by the ca-cert post install script (that uses p11-kit), located in /etc/pki/ca-trust/, contain certificates.
With that, all parts of this bug have been tested. MUST: ✅ Programs in our build environment that need to, should be able to access external sites over TLS. > Tested by running openssl, gnutls-cli and tx on their respective architectures > in cbrun. See comment 11 and comment 13 for more info. Also tested openssl > through Python, see comment 12. > > Note that transifex needs a .transifexrc file to work. ✅ Installed CA-certificates must be fetched from a credible source. > They are. Note that certificates have to be manually retrieved from Fedora's > site, since their certdata.txt is modified to also contain Microsoft objsign > certificates. Also added a README with a link to their package. ✅ The RPM:s should install and uninstall without errors. > They do. Tested by installing, uninstalling and reinstalling ca-certificates. > This is the only new RPM that contains a post install script generating files, > which are not actually in the package themselves. Post install script can > contain bugs not apparent during package build. SHOULD: ✅ The chosen location for installed certificates should follow best practices of, e.g. up-to-date Linux distributions. > Our certificates are installed in the same locations relative to each > cbrun arch-root as where Fedora install their certs.
Removed packages six3 and slugify, that became obsolete after the Transifex client was updated.
Removed unidecode package that became obsolete for the same reason as the packages in comment 29.
Ownership and creation of some compatibility related symlinks have been moved from our ca-certificates package to our OpenSSL package, since the files they referenced are already owned by OpenSSL.
We are now using Mozilla instead of Fedora as our upstream for certdata.txt and nssckbi.h. This has simplified keeping our source files in synch with upstream, since new upstream releases do not have to be merged with ca-certs from Microsoft (this was the case for Fedora's ca-certificates package). This also means that we are not creating object-sign-certificates, since these are a part of the ca-certs from Microsoft.
General =================== Tested this bug on a Fedora38 where several facets were explored, including ✅ Packages build successfully • transifex-client ◦ i386 • ca-certificates ◦ i386 ◦ x86_64 ◦ win32 ◦ win64 ◦ osx64 ◦ armhf • p11-kit ◦ i386 ◦ x86_64 ◦ win32 ◦ win64 ◦ osx64 ◦ armhf ✅ Updated versions and releases in Cendio repo ✅ Install / uninstall without errors • p11-kit • ca-certificates • transifex-client ✅ Paths • ca-certificates and p11-kit trust paths synced • OPENSSLDIR • GnuTLS trust store • Installed certificate paths follow modern practice, e.g. Fedora38 • Compatibility symlinks ✅ Certdata from credible source • Mozilla: https://hg.mozilla.org/releases/mozilla-release/raw-file/default/security/nss/lib/ckfw/builtins/ ✅ Generated certificates are non-empty ✅ A selection of certificate dependent packages working as intended • OpenSSL ◦ As suggested by comment 11 ◦ Via Python script as suggested by comment 12 • GnuTLS ◦ As suggested by 11 • osslsigncode ◦ As suggested by comment 12 • Transifex client ◦ As suggested by comment 13 Acceptance criteria =================== MUST: ✅ Programs in our build environment that need to, should be able to access external sites over TLS. Confirmed. ✅ Installed CA-certificates must be fetched from a credible source. Confirmed. ✅ The RPM:s should install and uninstall without errors. Confirmed. SHOULD: ✅ The chosen location for installed certificates should follow best practices of, e.g. up-to-date Linux distributions. Confirmed. Conclusion =================== Certificates are working as intended, implementation looks good, and the acceptance criteria are fulfilled. Closing.
