Something is seriously amiss with the recently released RHEL 9. If I try to connect using the native client, then it fails on the agent step and gives me: > Couldn't set up secure tunnel to ThinLinc agent. > (Couldn't establish SSH tunnel, SSH terminated.) The client log file doesn't say much: > ... > 2022-05-20T09:04:56: SSH pid is 1936919 > 2022-05-20T09:04:57: ssh[E]: CONFIRM HOST KEY: 10.48.2.96 10.48.2.96 22 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFa/LrDZODnAJAdcNuRuALWrivATmKcuVxlLMDPF4zjOluADcgJgB1kRn4syTSVA/waWVVLKU+9YciHrwCGmCSE= ECDSA > 2022-05-20T09:04:57: Given host key matches one in the acceptable list. > 2022-05-20T09:04:57: ssh[E]: NEXT AUTHMETHOD: none > 2022-05-20T09:04:57: ssh[E]: AUTH FAILURE > 2022-05-20T09:04:57: ssh[E]: NEXT AUTHMETHOD: password > 2022-05-20T09:04:57: ssh[E]: PASSWORD: cendio 10.48.2.96 > 2022-05-20T09:05:01: ssh[E]: AUTH SUCCESS > 2022-05-20T09:05:01: ssh[E]: CONNECTED > 2022-05-20T09:05:01: ssh[E]: Connection to 10.48.2.96 closed by remote host. > 2022-05-20T09:05:04: Process 1936886 exited with code 0 > 2022-05-20T09:05:04: Process 1936919 exited with code 255 If I compare that with a working system, the next few lines should be a host key sync: > ... > 2022-05-20T09:10:13: ssh[E]: NEXT AUTHMETHOD: password > 2022-05-20T09:10:13: ssh[E]: PASSWORD: cendio 10.48.2.111 > 2022-05-20T09:10:13: ssh[E]: AUTH SUCCESS > 2022-05-20T09:10:13: ssh[E]: CONNECTED > 2022-05-20T09:10:13: ssh[E]: UPDATE HOST KEYS: 2 10.48.2.111 10.48.2.111 22 > 2022-05-20T09:10:13: ssh[E]: UPDATED HOST KEY: AAAAB3NzaC1yc2EAAAADAQABAAABgQCrFAA8xj617fokeaVq/yMTKF/w/2X5zASlVq+dCMuCv+sLAEufs13Bq9ZlPJ1yEYwnYfXsSKgtIWg+cfLjHW2w+gI6RW2cSuv3bS7NQl9mCqbMcYHgIUfcGtm2cHmRvvUjYXNthU6PH4KwI1yG/0Ce8HZvwX+2itw/jsjZdec3yeywVMPfhWSED4CERbnVmvhF7gEKHFs1sfOLGDAV4PHFq1tF4W6Rv68xyym0LXDQfvox50ITc7hG0vv8LpiFzeIlx/J8pFd6WFZWwcl9Jf7+mQaznbg1+yk1mtJaHhYWZluBRBojhR/MphLdmRPUEUocLK2Cnyj3F9rCyUePoU2uYVlYqa4Lx7dK9Ku/X5Uhuq3/CkReMgTwG+YrZmDlvHUQJyJwS2oZQI5xD+0vc1taVCplt98AmVwGfwhKdEsgMomdBNBh17RI6sr3TwBFsTF61Y7gplyQ1pZsNVsjbfwRZNCqQfV6fOZwXRNHh6S/K558o9skzhXEd5stUO3sfr0= > 2022-05-20T09:10:13: ssh[E]: UPDATED HOST KEY: AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMKN0L7y77XTTGo5hkOjN2Gv/iOWzdDQMAfGCb09tayr4GLxX6wkkn6PC2yD90gJlKOWptvkMF7cnc2+2H7C7CE= > 2022-05-20T09:10:13: ssh[E]: THINLINC-LOGIN: HELLO 4.14.0post > 2022-05-20T09:10:13: ssh[E]: > 2022-05-20T09:10:13: ssh[E]: COMMAND_EXITSTATUS: 0 On the server, I see this in the journal: > May 20 09:13:37 lab-96.lkpg.cendio.se sshd[15387]: Accepted password for cendio from 10.47.1.240 port 43536 ssh2 > May 20 09:13:37 lab-96.lkpg.cendio.se systemd-logind[872]: New session 20 of user cendio. > May 20 09:13:37 lab-96.lkpg.cendio.se systemd[1]: Started Session 20 of User cendio. > May 20 09:13:37 lab-96.lkpg.cendio.se sshd[15387]: pam_unix(sshd:session): session opened for user cendio(uid=1000) by (uid=0) > May 20 09:13:37 lab-96.lkpg.cendio.se sshd[15387]: fatal: mm_answer_sign: sign: error in libcrypto > May 20 09:13:37 lab-96.lkpg.cendio.se sshd[15387]: pam_unix(sshd:session): session closed for user cendio > May 20 09:13:37 lab-96.lkpg.cendio.se sshd[15391]: fatal: mm_request_send: write: Broken pipe > May 20 09:13:37 lab-96.lkpg.cendio.se systemd[1]: session-20.scope: Deactivated successfully. > May 20 09:13:37 lab-96.lkpg.cendio.se systemd-logind[872]: Session 20 logged out. Waiting for processes to exit. > May 20 09:13:37 lab-96.lkpg.cendio.se systemd-logind[872]: Removed session 20. I'm not getting any useful information if I google those errors. A strace also shows that it is doing something related to host key sync when it fails: > [pid 15348] write(5, "\0\0\0\0\0\0\1\340\0\0\0\35hostkeys-prove-00@op"..., 496 <unfinished ...> > [pid 15337] getpid( <unfinished ...> > [pid 15348] <... write resumed>) = 496 > [pid 15337] <... getpid resumed>) = 15337 > [pid 15348] getpid( <unfinished ...> > [pid 15337] read(8, <unfinished ...> > [pid 15348] <... getpid resumed>) = 15348 > [pid 15337] <... read resumed>"\0\0\1\361", 4) = 4 > [pid 15348] getpid( <unfinished ...> > [pid 15337] read(8, <unfinished ...> > [pid 15348] <... getpid resumed>) = 15348 > [pid 15337] <... read resumed>"\6\0\0\0\0\0\0\1\340\0\0\0\35hostkeys-prove-00@o"..., 497) = 497 > [pid 15348] getpid( <unfinished ...> > [pid 15337] getpid( <unfinished ...> > [pid 15348] <... getpid resumed>) = 15348 > [pid 15337] <... getpid resumed>) = 15337 > [pid 15348] read(5, <unfinished ...> > [pid 15337] getpid() = 15337 > [pid 15337] getpid() = 15337 > [pid 15337] getpid() = 15337 > [pid 15337] getpid() = 15337 > [pid 15337] getpid() = 15337 > [pid 15337] sendto(3, "<82>May 20 09:05:01 sshd[15337]:"..., 80, MSG_NOSIGNAL, NULL, 0) = 80 >
Reported to Red Hat: https://bugzilla.redhat.com/show_bug.cgi?id=2088750
So Red Hat gave a clue as to what the issue is. It seems sshd explodes if the client supports the older "ssh-rsa" algorithm, instead of the newer "rsa-sha2-256" or "rsa-sha2-512". The problem seems to be the corner case that all three of these use the same key, just different hash algorithms. The issue cannot be seen for other deprecated algorithms, e.g. "ssh-dss". This also exposes a bug in the client. When connecting to an agent, it doesn't properly advertise support for the newer algorithms. Moving that as a separate issue to bug 7935.
Also note that we are about to drop support for ssh-rsa anyway (bug 7535). So this issue might resolve itself by the mere fact that we will no longer support older systems that need ssh-rsa.
A platform specific note has been added with workarounds until we can have an updated client out.
Works fine now that bug 7535 is in place. A release note might still be prudent, even if we technically didn't fix anything, since this was a user visible issue.
Tested bug using client builds #2641 (pre-fix) and #2653 (post-fix) on Fedora36 and server build #2750 on RHEL9. pre-fix connect results: [X] crypto policy: DEFAULT [✔] crypto policy: DEFAULT:SHA1 post-fix connect results: [✔] crypto policy: DEFAULT [✔] crypto policy: DEFAULT:SHA1 Acceptance criteria: [✔] The client can connect to a default configure RHEL 9 system Evidently, the fix introduced in bug 7535 eliminates the connecting issue. Closing.
