Bug 7764 - Upgrade OpenSSH and OpenSSL
Summary: Upgrade OpenSSH and OpenSSL
Status: CLOSED FIXED
Alias: None
Product: ThinLinc
Classification: Unclassified
Component: Client (show other bugs)
Version: trunk
Hardware: PC Unknown
: P2 Normal
Target Milestone: 4.14.0
Assignee: William Sjöblom
URL:
Keywords: linma_tester, relnotes, samuel_tester
Depends on: 7766
Blocks:
  Show dependency treegraph
 
Reported: 2021-09-09 15:57 CEST by William Sjöblom
Modified: 2022-07-05 09:33 CEST (History)
4 users (show)

See Also:
Acceptance Criteria:


Attachments

Description William Sjöblom cendio 2021-09-09 15:57:00 CEST
We are currently using OpenSSH 8.3p1. We should upgrade OpenSSH to the latest version 8.7p1, and in turn upgrade OpenSSL.
Comment 1 Pierre Ossman cendio 2021-09-13 12:29:29 CEST
For OpenSSL we are currently on 1.1.1g, whilst 1.1.1l is the latest in that series, but there is also a 3.0.0. Will need to check if OpenSSH supports the new stable series.
Comment 2 Pierre Ossman cendio 2021-09-13 12:59:18 CEST
OpenSSL CVEs:

 * CVE-2021-3712: Not relevant as we don't use ASN.1 in OpenSSH
 * CVE-2021-3711: Not relevant as OpenSSH doesn't support SM2¹
 * CVE-2021-3450: Not relevant for our version of OpenSSL
 * CVE-2021-3449: Not relevant as OpenSSH doesn't use TLS
 * CVE-2021-23841: Not relevant as OpenSSH doesn't use X.509
 * CVE-2021-23840: Doesn't seem to be relevant as OpenSSH doesn't use any of the mentioned functions.
 * CVE-2020-1971: Not relevant as OpenSSH doesn't use X.509

¹https://bugzilla.mindrot.org/show_bug.cgi?id=3337
Comment 6 William Sjöblom cendio 2021-09-14 11:08:29 CEST
Here are the most notable changes in OpenSSH since our last
upgrade. These include all security related issues with OpenSSH as a
whole and potentially incompatible changes made to the components of
OpenSSH that we are actively using.

• 8.4p1
  • Various potentially incompatible- and security changes related to
    FIDO authentication, which we do not use.
• 8.5p1
  • Changed first preference signature algorithm from ECDSA to ED25519.
  • Fixed double-free vulnerability in ssh-agent, which we do not use.
• 8.6p1
  • Fixed low-privilege sandbox escape in sshd, which we do not use.
• 8.7p1
  • OpenSSH will disable the "ssh-rsa" signature scheme in the next
    release.
  • The configuration parser has been made stricter and will reject
    syntactically questionable configuration files that were previously
    accepted.
Comment 8 Pierre Ossman cendio 2021-09-15 10:06:17 CEST
Our version of OpenSC doesn't support OpenSSL 3.0 unfortunately. However the latest version (0.22.0) has mild support, and they are working on fully migrating:

https://github.com/OpenSC/OpenSC/issues/2308
Comment 9 Pierre Ossman cendio 2021-09-15 10:08:25 CEST
Unfortunately OpenSC 0.22.0 requires some functions on Windows that we lack (e.g. gmtime_s()). They are conditional in our version of mingw, and are disabled by default. I'm not getting a clear picture here, but it seems they were an optional add-on to Windows until Windows Vista or Windows 7, which may be why they aren't always enabled.

So we'll have to rebuild our mingw with those enabled, but bug 7766 is currently in the way.
Comment 16 William Sjöblom cendio 2021-09-16 14:34:34 CEST
Tested Kerberos authentication on a number of platforms with Jenkins
client build #2199 against two different ThinLinc servers (both running
Jenkins server build #2274). The SLES12 server have OpenSSH 7.2p2 and
OpenSSL 1.0.2. The Ubuntu 20.10 Desktop server have OpenSSH 7.9p1 and
OpenSSL 1.1.1f.

━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Client Distribution │ SLES12  Ubuntu 20.10 Desktop 
─────────────────────┼──────────────────────────────
 Fedora 34           │ ok      ok                   
 Debian 9            │ ok      ok                   
─────────────────────┼──────────────────────────────
 Windows 10 64-bit   │ ok      ok                   
─────────────────────┼──────────────────────────────
 macOS 11.5.2        │ ok      ok                   
━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Comment 17 Pierre Ossman cendio 2021-09-16 16:41:07 CEST
Tested that the Windows installer signature is accepted by Windows 10, so osslsigncode seems to work fine now.
Comment 18 William Sjöblom cendio 2021-09-16 17:00:39 CEST
Tested public key authentication on a number of platforms with Jenkins
client build #2200 against two different ThinLinc servers (both running
Jenkins server build #2274).
━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Client Distribution │ SLES12  Ubuntu 20.10 Desktop 
─────────────────────┼──────────────────────────────
 Fedora 34           │ ok      ok                   
 Debian 9            │ ok      ok                   
─────────────────────┼──────────────────────────────
 Windows 10          │ ok      ok                   
─────────────────────┼──────────────────────────────
 macOS 11.5.2        │ ok      ok                   
━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Comment 20 William Sjöblom cendio 2021-09-17 13:46:11 CEST
A bug was found when testing OTP where the message displayed in the OTP dialog was not displaying correctly. This has now been corrected. I have tested a local client build running on Fedora 34 and it now seems to work as expected both when doing OTP and when the server administrator has required the password to be changed.
Comment 21 Niko Lehto cendio 2021-09-17 14:44:16 CEST
Servers:
 - SLES 12
 - Ubuntu 20.10

Client build 2199
Server build 2274

Cards:
 - Inetra, valid: 2018.03.22 
 - Karlstads kommun, valid: 2017-03-10
 - Aventra 4.5*
 - Aventra 3.3*

* Aventra cards needs to be configured manually,
  but we have an helper script for that here:
    http://intranet.lkpg.cendio.se/ThinLinc/Aventra

Client on Windows 10: 					

| Card     | SLES12 | Ubuntu |
+----------+--------+--------+
| Karlstad |  x**   |   x**  |
+----------+--------+--------+
| Inetra   |  OK    |	OK   |
+----------+--------+--------+
| Avtr 4.5 |  OK    |   OK   |
+----------+--------+--------+
| Avtr 3.3 |  OK    |   OK   |
+----------+--------+--------+


Client on macOS 11.5:

| Card     | SLES12 | Ubuntu |
+----------+--------+--------+
| Karlstad |  x**   | 	x**  |
+----------+--------+--------+
| Inetra   |  OK    |  	OK   |
+----------+--------+--------+
| Avtr 4.5 |  OK    |   OK   |
+----------+--------+--------+
| Avtr 3.3 |  OK    |   OK   |
+----------+--------+--------+


Client on Ubuntu 20.04:

| Card     | SLES12 | Ubuntu |
+----------+--------+--------+
| Karlstad |  x**   | 	x**  |
+----------+--------+--------+
| Inetra   |  OK    | 	OK   |
+----------+--------+--------+
| Avtr 4.5 |  OK    | 	OK   |
+----------+--------+--------+
| Avtr 3.3 |  OK    | 	OK   |
+----------+--------+--------+

** This card gets the error:
     "Smart card malfunction. Check your hardware"
   To test this we propably need to configure testserver
   ssh and disable newer public key algoritms that will deny this card.
  
Also tested automatic disconnect on all plattforms mentioned above. Works good.
We also propably should test "Automatic username" for smart cards.
Comment 22 William Sjöblom cendio 2021-09-20 09:26:45 CEST
Tested interactive authentication on a number of platforms with Jenkins
client build #2204 against two different ThinLinc servers (both running
Jenkins server build #2284). Since SLES12 does not have the
google-authenticator PAM module, these tests diverge from my previous
setup and use RHEL7 instead.
━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Client Distribution │ RHEL7  Ubuntu 20.10 Desktop 
─────────────────────┼─────────────────────────────
 Fedora 34           │ ok     ok                   
 Debian 9            │ ok     ok                   
─────────────────────┼─────────────────────────────
 Windows 10          │ ok     ok                   
─────────────────────┼─────────────────────────────
 macOS 11.5.2        │ ok     ok                   
━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Comment 23 William Sjöblom cendio 2021-09-20 09:34:52 CEST
To summarize, I have tested the following features and scenarios (in comment 16, comment 18, and comment 22 respectively):
• Feature: Kerberos Authentication
  • Scenario: Authentication
• Feature: Public Key Authentication
  • Scenario: No Password
  • Scenario: With Password
  • Scenario: Unauthorized Key
• Feature: Interactive Authentication
  • Scenario: Simple Password
  • Scenario: One-time Password
Comment 24 Pierre Ossman cendio 2021-09-20 11:25:16 CEST
Did a very basic performance test by playing video at 1024x768 and checking ssh CPU usage. 

 * Fedora 34: 5-7% for both new version and 4.13.0, a few points more for 4.10.0
 * CentOS 7 (RPi): 25-30% for both new and older versions
 * Windows 10: 2-5% for both new and older versions
 * macOS 11: 2-3% for both new and older versions.

So no substantial changes in performance by this upgrade.

Server was a RHEL 8 system.
Comment 25 William Sjöblom cendio 2021-09-21 10:18:01 CEST
Tested smart card automatic username using Jenkins client build #2204
running on Fedora 34 against a Ubuntu 20.10 Desktop server running
Jenkins server build #2274.
━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━
 Client Platform │ Inera  Karlstad 
─────────────────┼─────────────────
 Fedora 34       │ ok     ok       
 Debian 9        │ ok     ok       
─────────────────┼─────────────────
 Windows 10      │ ok     n/a      
─────────────────┼─────────────────
 macOS 11.5.2    │ ok     ok       
━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━

I tested the card that was left untested in comment 21 (see bug 7599 for
further details about the problem). There was an attempt was made to get
it working by forbidding SHA512 pubkeys in server-side sshd
configuration without success [1]. I instead managed to get it working
by forbidding SHA512 pubkeys by creating a `~/.thinlinc/config'
containing the line:

┌────
│ PubkeyAcceptedKeyTypes=-rsa-sha2-512
└────

━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Client Platform │ SLES12  Ubuntu 20.10 Desktop 
─────────────────┼──────────────────────────────
 Fedora 34       │ ok      ok                   
 Debian 9        │ ok      ok                   
─────────────────┼──────────────────────────────
 macOS 11.5.2    │ ok      ok                   
━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

I also tested automatic usernames on the above Ubuntu 20.10 Desktop
server using the server and client version without any issues.
━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━
 Client Platform │ Inera  Karlstad 
─────────────────┼─────────────────
 Fedora 34       │ ok     ok       
 Debian 9        │ ok     ok       
─────────────────┼─────────────────
 Windows 10      │ ok     n/a      
─────────────────┼─────────────────
 macOS 11.5.2    │ ok     ok       
━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━


• [1] https://github.com/openssh/openssh-portable/blob/e1a596186c81e65a34ce13076449712d3bf97eb4/kex.c#L437
Comment 26 William Sjöblom cendio 2021-09-21 10:24:37 CEST
Everything seems to have held together nicely after the upgrade. Marking as resolved.
Comment 27 Linn cendio 2021-09-28 11:01:40 CEST
Tested everything below, no issues found. Kerberos still remains to be tested.

  Dist    | ver SSH | ver SSL
----------+---------+--------
Fedora 34 |  8.5p1  | 1.1.1k
CentOS 7  | 6.6.1p1 | 1.0.2k


Public key authentication
  ✓ No password
  ✓ With password
  ✓ Unauthorized key
  
Interactive authentication
  ✓ Simple Password
  ✓ One time password
  
Smart card authentication
  ✓ Simple authentication
    ✓ Card type Inera
    ✓ Card type Karlstad
Comment 28 Linn cendio 2021-09-30 09:28:50 CEST
To clarify comment 27, all tests were performed on client build 2207 with clients on Fedora 33, Windows 10 and macOS 11.6.
Comment 29 Linn cendio 2021-09-30 09:41:50 CEST
Tested Kerberos on server build 2288 on Fedora 34 and CentOS 7, seems to be working well. 
Used client build 2211 with clients on Fedora 33, Windows 10 and macOS 11.6.

  Dist    | ver SSH | ver SSL
----------+---------+--------
Fedora 34 |  8.5p1  | 1.1.1k
CentOS 7  | 6.6.1p1 | 1.0.2k


Kerberos authentication | Fed33 | Win10 | macOS11.6 |
  Authentication            ✓       ✓         ✓
Comment 30 Frida Flodin cendio 2021-09-30 14:10:11 CEST
Found a regression when batch testing. The session is no longer disconnected when the smart card reader is disconnected. Tested against server build 2291 on Fedora 34.

Tested the nightly client build 2211 on both Windows 10 and Fedora 34 and the session continues to live when smart card reader is pulled out. 

I also tested 4.13.0 client and then the session disconnects when I pull out the smart card reader. This was first fixed in bug 7253.
Comment 31 Pierre Ossman cendio 2021-10-01 10:30:11 CEST
This looks like some bug in OpenSC. I'm comparing traces from OpenSC (using OPENSC_DEBUG=9) in ThinLinc 4.13.0 and the current build. It looks like the new OpenSC clears all events when it notices that the reader is gone as part of the C_GetSlotList() call. Hence it no longer states that things have changed when we then call C_WaitForSlotEvent().

However that is wrong according to our current understanding of the PKCS#11 specification. Calling C_GetSlotList() should cause OpenSC to check if the slots have changed, and only queue the events for subsequent calls to C_WaitForSlotEvent().
Comment 32 Pierre Ossman cendio 2021-10-01 12:45:07 CEST
Issue reported upstream:

https://github.com/OpenSC/OpenSC/issues/2415
Comment 38 Frida Flodin cendio 2021-10-06 11:16:22 CEST
Should be fixed now. Tested on Fedora 34, Windows 10 and macOS 11.6.

Tested the following by looking at the list of smartcards in tlclient login
window.
✓ Remove smart card
✓ Remove reader with card in
✓ Insert card
✓ Insert reader with card in

In the same poll interval*
✓ Remove one reader with card, insert another
✓ Two readers, remove one card, insert another card in other reader
✓ First remove card then remove the reader
✓ First insert reader then card
✓ Two readers, insert card in reader A, remove reader B with card in it


Tested the following when logged in with smartcard and with automatic
disconnect turned on. Tested against 4.13.0 server:
✓ Remove card, disconnects
✓ Remove reader with card in it, disconnects
✓ Doing nothing, no disconnect

In the same poll interval*:
✓ Remove card then insert it again, no disconnect
✓ Remove reader then insert it again, no disconnect
✓ Remove reader insert the card in another reader, no disconnect
✓ Add new reader and move card to it, no disconnect
✓ One reader, change to another card in the reader, disconnects**

  * The normal poll interval is 1 second. For testing purposes I changed this to
    10 seconds in order to have the time to do the smart card maneuvers.
 ** Does not work on macOS, I have tested with and old build and it is not a
    regression. I will investigate it further and probably report it upstream.
Comment 39 Samuel Mannehed cendio 2021-10-07 17:35:29 CEST
Tested using server build 2295 on Fedora 34, and connecting using client build 2215. I tested using Fedora 34, Windows 10 (32-bit) and macOS 11.6 clients:

 ✓ All certificates on a card are shown when no filters are active
 ✓ All certificates from both cards are shown when two cards are inserted
 ✓ Certificates disappear properly when the card is removed
 ✓ Filters work properly both with regards to "Issuer" and "Key usage"
 ✓ Filters are disabled after Ctrl+Shift+F8 is pressed
 ✓ Automatic login doesn't trigger when multiple certificates are available

 ✓ Regular login with username + smartcard
 ✓ Login doesn't work with the incorrect PIN
 ✓ Login doesn't work with a different smartcard

 ✓ Automatic login when smartcard is inserted
 ✓ Disconnect by removing smartcard
 ✓ Disconnect by removing card reader

 ✓ Session is not disconnected when disconnecting a different smartcard
 ✓ Session is not disconnected when disconnecting a different card reader
 ✓ Session is not disconnected when disconnecting and quickly re-inserting the card in the same reader
 ✓ Session is not disconnected when disconnecting and quickly re-inserting the card in a different reader
 ✓ Session IS disconnected when disconnecting and quickly re-inserting a different card the same reader

 ✓ No unwanted things are written to the client log files
 ✓ Code changes look good
Comment 40 Samuel Mannehed cendio 2021-10-08 10:08:03 CEST
>  ✓ Session IS disconnected when disconnecting and quickly re-inserting a different card the same reader

Frida mentioned that this did not work as intended on macOS, as mentioned in comment #38. I re-tested and concluded the same - quickly re-inserting a different card in the same reader will leave the user connected. The setting "Disconnect when smart card is removed" was enabled.

The trick is to switch the cards within the same polling interval.

I must have been "lucky" and hit a poll in the middle, making the software detect that no card was present, before I had managed to insert the new card.

Another trick that can be used to more easily reproduce this is by running tlclient from a terminal, and "pausing" it by pressing Ctrl+Z in the terminal when making the card switch.

Note You need to log in before you can comment on or make changes to this bug.