Bug 7809 - Server RPMs cannot be installed on FIPS enforcing system
Summary: Server RPMs cannot be installed on FIPS enforcing system
Status: CLOSED FIXED
Alias: None
Product: ThinLinc
Classification: Unclassified
Component: Server Installer (show other bugs)
Version: trunk
Hardware: PC Unknown
: P2 Normal
Target Milestone: 4.17.0
Assignee: Adam Halim
URL:
Keywords: relnotes, samuel_tester
Depends on: 2960 5435
Blocks:
  Show dependency treegraph
 
Reported: 2021-12-16 16:39 CET by Pierre Ossman
Modified: 2024-04-08 08:53 CEST (History)
4 users (show)

See Also:
Acceptance Criteria:
MUST * Our RPMs should install on most FIPS enforcing systems (RHEL, SLES, Fedora)


Attachments

Description Pierre Ossman cendio 2021-12-16 16:39:02 CET
If you enable "FIPS mode" on Red Hat systems it activates some extra checks in rpm that causes our installation to fail:

> 2021-12-16 16:25:46,370: Installation of packages failed:
> 2021-12-16 16:25:46,371:     ########################################
> 2021-12-16 16:25:46,371:     ########################################
> 2021-12-16 16:25:46,371:        package thinlinc-tlmisc-libs-4.13.0post-2376.x86_64 does not verify: no digest
> 2021-12-16 16:25:46,371:        package thinlinc-tlprinter-4.13.0post-2376.noarch does not verify: no digest
> 2021-12-16 16:25:46,371:        package thinlinc-tlmisc-4.13.0post-2376.x86_64 does not verify: no digest
> 2021-12-16 16:25:46,371:        package thinlinc-vsm-4.13.0post-2376.x86_64 does not verify: no digest
> 2021-12-16 16:25:46,371:        package thinlinc-webaccess-4.13.0post-2376.noarch does not verify: no digest
> 2021-12-16 16:25:46,371:        package thinlinc-vnc-server-4.13.0post-2376.x86_64 does not verify: no digest
> 2021-12-16 16:25:46,371:        package thinlinc-tladm-4.13.0post-2376.x86_64 does not verify: no digest
> 2021-12-16 16:25:46,372:        package thinlinc-tlmisc-libs32-4.13.0post-2376.i686 does not verify: no digest

Unfortunately bypassing the signature check is insufficient:

> $ sudo rpm -Uvh --nodigest --noverify *.rpm
> Preparing...                          ################################# [100%]
> Updating / installing...
>    1:thinlinc-tlmisc-libs-4.13.0post-2################################# [ 13%]
> error: unpacking of archive failed on file /etc/passwdaliases;61bb5bb6: cpio: Digest mismatch
> error: thinlinc-tlmisc-libs-4.13.0post-2376.x86_64: install failed
>    2:thinlinc-tlprinter-4.13.0post-237################################# [ 25%]
> error: unpacking of archive failed on file /opt/thinlinc/libexec/add_nearest_printer.sh;61bb5bb6: cpio: Digest mismatch
> error: thinlinc-tlprinter-4.13.0post-2376.noarch: install failed
>    3:thinlinc-tlmisc-4.13.0post-2376  ################################# [ 38%]
> error: unpacking of archive failed on file /etc/cron.d/tl-statistics-cron;61bb5bb6: cpio: Digest mismatch
> error: thinlinc-tlmisc-4.13.0post-2376.x86_64: install failed
>    4:thinlinc-vsm-4.13.0post-2376     ################################# [ 50%]
> error: unpacking of archive failed on file /etc/logrotate.d/thinlinc-vsm-agent;61bb5bb6: cpio: Digest mismatch
> error: thinlinc-vsm-4.13.0post-2376.x86_64: install failed
>    5:thinlinc-webaccess-4.13.0post-237################################# [ 63%]
> error: unpacking of archive failed on file /etc/logrotate.d/thinlinc-webaccess;61bb5bb6: cpio: Digest mismatch
> error: thinlinc-webaccess-4.13.0post-2376.noarch: install failed
>    6:thinlinc-tladm-4.13.0post-2376   ################################# [ 75%]
> error: unpacking of archive failed on file /etc/logrotate.d/thinlinc-tlwebadm;61bb5bb6: cpio: Digest mismatch
> error: thinlinc-tladm-4.13.0post-2376.x86_64: install failed
>    7:thinlinc-vnc-server-4.13.0post-23################################# [ 88%]
> error: unpacking of archive failed on file /opt/thinlinc/bin/setxkbmap;61bb5bb6: cpio: Digest mismatch
> error: thinlinc-vnc-server-4.13.0post-2376.x86_64: install failed
>    8:thinlinc-tlmisc-libs32-4.13.0post################################# [100%]
> error: unpacking of archive failed on file /lib/libnss_passwdaliases.so.2;61bb5bb6: cpio: Digest mismatch
> error: thinlinc-tlmisc-libs32-4.13.0post-2376.i686: install failed

There is some extra check for the files as well, which you fortunately can also bypass:

> $ sudo rpm -Uvh --nodigest --nofiledigest *.rpm
> Preparing...                          ################################# [100%]
> Updating / installing...
>    1:thinlinc-tlmisc-libs-4.13.0post-2################################# [ 13%]
>    2:thinlinc-tlprinter-4.13.0post-237################################# [ 25%]
>    3:thinlinc-tlmisc-4.13.0post-2376  ################################# [ 38%]
>    4:thinlinc-vsm-4.13.0post-2376     ################################# [ 50%]
>    5:thinlinc-webaccess-4.13.0post-237################################# [ 63%]
>    6:thinlinc-tladm-4.13.0post-2376   ################################# [ 75%]
>    7:thinlinc-vnc-server-4.13.0post-23################################# [ 88%]
>    8:thinlinc-tlmisc-libs32-4.13.0post################################# [100%]

So it seems like it's not just package signatures that are missing, but something more. This thread also confirms that signing the packages is not enough:

https://access.redhat.com/discussions/5043121

Also worth noting is that us signing our packages is not sufficient to get pass the signature check. The key must also be trusted, which is unclear how that should accomplished in a secure way. Also discussed on bug 7637.
Comment 2 Patrik Pira 2021-12-16 20:10:05 CET
You will also need to build the packages with SHA256 file digests. This is taken from /usr/lib/rpm/macros on RHEL 8. 

#	Algorithm to use for generating file checksum digests on build.
#	If not specified or 0, MD5 is used.
#	WARNING: non-MD5 is backwards incompatible with rpm < 4.6!
#	The supported algorithms may depend on the underlying crypto
#	implementation but generally at least the following are supported:
#	1	MD5
#	2	SHA1
#	8	SHA256 (default)
#	9	SHA384
#	10	SHA512
#
%_source_filedigest_algorithm	8
%_binary_filedigest_algorithm	8
Comment 3 Ken Bass 2022-06-23 21:13:53 CEST
Is there any possibility your build process can create the RPM packages using RPM version 4.14 or newer? My understanding is that will fix this issue and will also be backwards compatible with older RHEL/Centos versions.
Comment 4 Pierre Ossman cendio 2022-06-27 08:38:27 CEST
Not at the moment, no. We'd need to check exactly what an upgrade means for the various distributions we try to support.

For now, I'm afraid you'll have to use the appropriate flags to rpm when installing on a FIPS enforcing system.
Comment 5 Ken Bass 2022-06-27 18:30:36 CEST
I did spend some effort researching this before posting here, so I think using RPM 4.14 will fix the issue and will also work on other older RPM platforms. 

https://fedoraproject.org/wiki/Changes/RPM-4.14 indicates they changed RPM to use SHA256 by default and they declare it is backwards compatible. 

From other research I did, there is an indication that building your package with  RPM 4.14 or newer will work on RHEL6, RHEL7, and RHEL8 (and their Centos equivalents) due to the backwards compatibility.

Maybe give it a try when you have a chance? Not sure what other RPM based systems you need to support, but many older ones are past or nearing EOL.
Comment 10 Adam Halim cendio 2024-03-26 15:56:15 CET
In our quest to allow our RPMs to be installed on FIPS systems, we had to
change our file digests to use SHA256 instead of MD5. SHA256 digests have been
supported in RPM since version 4.6, and has been the default since 4.14. 

This means that with this change, we technically raise our minimum server
requirements to have an RPM version of >= 4.6. In practice, this should not
affect anyone as version 4.6 is very old. We had a quick look at some distros
and the oldest and found that SLES 12 has version 4.11, and even CentOS 6 uses
version 4.8.0.
Comment 12 Adam Halim cendio 2024-03-27 16:06:25 CET
Tested build 3537, which is signed and uses SHA256 file digests.

The following distos were tested, with FIPS mode enabled:
* SLES 12   [1]
* SLES 15   [1]
* CentOS 6  [2]
* RHEL 7    [3]
* RHEL 8    [4]
* RHEL 9    [5]

We wanted to test whether signing our RPMs has any impact on installation on
FIPS systems. It seems like this does not matter for the distros that we
tested, see below:

* SLES 12
    ✅ Signed RPM, not trusted key
    ✅ Unsigned RPM
    ✅ MD5 file digest (tl-4.16.0)
        - Interestingly enough, the RPM installed successfully even though the
          RPM had MD5 digests.

* SLES 15
    ✅ Signed RPM, not trusted key
        - We get a warning if the key is not trusted, but the RPM is installed
          successfully.
> warning: tl-4.16.0post-server/packages/thinlinc-server-4.16.0post-3537.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 5e02d204: NOKEY 
    ✅ Unsigned RPM
    ✅ MD5 file digest (tl-4.16.0)
        - Interestingly enough, the RPM installed successfully even though the
          RPM had MD5 digests.

* CentOS 6
    ✅ Signed RPM, not trusted key
    - We did not enable FIPS on this system, but the installation went fine.

* RHEL 7
    ✅ Signed RPM, not trusted key
    ✅ Unsigned RPM
    ✅ MD5 file digest (tl-4.16.0)
        - Interestingly enough, the RPM installed successfully even though the
          RPM had MD5 digests.

* RHEL 8
    ✅ Signed RPM, not trusted key
        - We get a warning if the key is not trusted, but the RPM is installed
          successfully.
> warning: tl-4.16.0post-server/packages/thinlinc-server-4.16.0post-3537.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 5e02d204: NOKEY
    ✅ Unsigned RPM
        - No warning.
    ❌ MD5 file digest (tl-4.16.0)
> [root@lab-59 ~]# rpm -ivh tl-4.16.0-server/packages/thinlinc-server-4.16.0-3491.x86_64.rpm 
> Verifying...                          ################################# [100%]
> Preparing...                          ################################# [100%]
>     package thinlinc-server-4.16.0-3491.x86_64 does not verify: no digest

* RHEL 9
    ✅ Signed RPM, not trusted key
        - We get a warning if the key is not trusted, but the RPM is installed
          successfully.
> warning: tl-4.16.0post-server/packages/thinlinc-server-4.16.0post-3537.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 5e02d204: NOKEY
    ✅ Unsigned RPM
        - No warning.
    ❌ MD5 file digest (tl-4.16.0)
> [root@lab-23 ~]# rpm -ivh tl-4.16.0-server/packages/thinlinc-server-4.16.0-3491.x86_64.rpm 
> Verifying...                          ################################# [100%]
> Preparing...                          ################################# [100%]
>     package thinlinc-server-4.16.0-3491.x86_64 does not verify: no digest

Note that it was only the newer RHEL distros (8 and 9) that failed to install
our RPMs if they had MD5 file digests.

This is how FIPS mode was enabled on the distros:
[1] https://www.suse.com/support/kb/doc/?id=000019432
[2] * We did not succeed in enabling FIPS on CentOS 6.
[3] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/chap-federal_standards_and_regulations
[4] fips-mode-setup --enable
[5] fips-mode-setup --enable
Comment 14 Adam Halim cendio 2024-03-28 09:13:34 CET
Tested client build 3435 on SLES 15 and RHEL 9 with FIPS mode enabled.

The client installed successfully on both platforms and worked without issue.
Comment 15 Samuel Mannehed cendio 2024-03-28 16:31:38 CET
Verified that build 3541 of the ThinLinc server, and build 3438 of the client works well on fips-enabled CentOS 8 and fips-enabled Ubuntu 22.04:

Client tests:

✓ Install client
✓ Launch client
✓ Sound redirection
✓ Local drive redirection

Server tests:

✓ Install server
✓ tl-setup
✓ Connect using Web Access
✓ Browsing different pages in Web Admin
✓ Sound redirection using native client
✓ Local printing using native client
✗(*) Local drive redirection using native client

(*) Didn't work on CentOS 8, see bug 8331. Worked well on Ubuntu.

To enable fips on Ubuntu [1], I needed to get a Ubuntu Pro token.

[1]: https://ubuntu.com/security/certifications/docs/2204/fips
Comment 16 Samuel Mannehed cendio 2024-04-04 12:45:25 CEST
I verified that fips was indeed enabled on the Ubuntu 22.04 machine mentioned in the previous comment:

> cendio@ubuntu-22:~$ cat /proc/sys/crypto/fips_enabled 
> 1
> cendio@ubuntu-22:~$ cat /proc/cmdline 
> BOOT_IMAGE=/boot/vmlinuz-5.15.0-101-fips root=UUID=66891745-0aec-4830-a1b6-18035a54e613 ro quiet splash fips=1
Comment 17 Samuel Mannehed cendio 2024-04-04 12:45:46 CEST
The code changes also look good. Closing.

Note You need to log in before you can comment on or make changes to this bug.