Bug 7716

Summary: tl-setup installs random sendmail package on apt systems
Product: ThinLinc Reporter: Samuel Mannehed <samuel>
Component: Server InstallerAssignee: Samuel Mannehed <samuel>
Status: CLOSED FIXED    
Severity: Normal CC: nikle
Priority: P2 Keywords: nikle_tester, relnotes
Version: trunk   
Target Milestone: 4.13.0   
Hardware: PC   
OS: Linux Ubuntu   
See Also: https://bugzilla.cendio.com/show_bug.cgi?id=8530
Acceptance Criteria:
Bug Depends on: 5980    
Bug Blocks:    

Description Samuel Mannehed cendio 2021-05-27 16:44:35 CEST 
In order to ensure that /usr/sbin/sendmail is present on the server we check for packages that provide the virtual package called "mail-transport-agent".

Depending on the state of the cache, the list of providing packages is ordered differently. We always install the first package in that list. That means tl-setup installs different packages at random.

When running 'sudo apt install mail-transport-agent' you always get the same order of possible packages:

> Package mail-transport-agent is a virtual package provided by:
>   postfix 3.4.13-0ubuntu1
>   exim4-daemon-light 4.93-13ubuntu1.5
>   exim4-daemon-heavy 4.93-13ubuntu1.5
>   ssmtp 2.64-8.1ubuntu1
>   sendmail-bin 8.15.2-18
>   qmail-run 2.0.2+nmu1
>   opensmtpd 6.6.4p1-1
>   nullmailer 1:2.2-3
>   msmtp-mta 1.8.6-1
>   masqmail 0.3.4-1build1
>   esmtp-run 1.2-17
>   dma 0.12-1
>   courier-mta 1.0.6-1build2
>   citadel-server 917-4
>   lsb-invalid-mta 11.1.0ubuntu2
> You should explicitly select one to install.


Debian has a github mirror of the apt code, and the relevant code that does the printing above can be found here:

https://github.com/Debian/apt/blob/766b24b7f7484751950c76bc66d3d6cdeaf949a5/apt-private/private-cacheset.cc

I can't see any obvious sorting being done however, perhaps it's arbitrary.
Comment 1 Samuel Mannehed cendio 2021-05-27 17:35:13 CEST 
The 'python3-apt' package contains more low level information as part of the apt_pkg module. Here's documentation for that:

https://apt-team.pages.debian.net/python-apt/library/apt_pkg.html

Using that module we can find the same list of packages that 'apt install' finds (with some duplicates):

>>> import apt_pkg
>>> apt_pkg.init()
>>> c = apt_pkg.Cache()
Reading package lists... Done
Building dependency tree       
Reading state information... Done
>>> for (target, ver, pkg_ver) in c["mail-transport-agent"].provides_list:
...     print(pkg_ver.parent_pkg.name)
... 
postfix
exim4-daemon-light
exim4-daemon-heavy
ssmtp
sendmail-bin
qmail-run
opensmtpd
nullmailer
msmtp-mta
masqmail
esmtp-run
dma
courier-mta
citadel-server
postfix
lsb-invalid-mta
exim4-daemon-light
exim4-daemon-heavy
Comment 2 Samuel Mannehed cendio 2021-06-01 14:11:21 CEST 
It's worth noting that we found this bug since one time when installing 'mail-transport-agent' the randomness fell on 'courier-mta' which apparently is broken:

2021-05-26 14:22:10,785: '/usr/sbin/sendmail' not found
2021-05-26 14:22:29,982: Selecting candidate "courier-mta" as provider for requested package "mail-transport-agent"
2021-05-26 14:23:31,281: Signing certificate...
2021-05-26 14:23:31,281: Job for courier-msa.service failed because the control process exited with error code.
2021-05-26 14:23:31,281: See "systemctl status courier-msa.service" and "journalctl -xe" for details.
2021-05-26 14:23:31,281: invoke-rc.d: initscript courier-msa, action "start" failed.
2021-05-26 14:23:31,281: ESC[0;1;31m●ESC[0m courier-msa.service - LSB: Courier MSA server
2021-05-26 14:23:31,281: Loaded: loaded (ESC]8;;file://ubuntu2004/etc/init.d/courier-msa^G/etc/init.d/courier-msaESC]8;;^G; generated)
2021-05-26 14:23:31,281: Active: ESC[0;1;31mfailedESC[0m (Result: exit-code) since Wed 2021-05-26 14:23:25 UTC; 12ms ago
2021-05-26 14:23:31,281: Docs: ESC]8;;man:systemd-sysv-generator(8)^Gman:systemd-sysv-generator(8)ESC]8;;^G
2021-05-26 14:23:31,281: Process: 4099 ExecStart=/etc/init.d/courier-msa start ESC[0;1;31m(code=exited, status=2)ESC[0m
2021-05-26 14:23:31,281: 
2021-05-26 14:23:31,282: May 26 14:23:25 ubuntu2004 systemd[1]: Starting LSB: Courier MSA server...
2021-05-26 14:23:31,282: May 26 14:23:25 ubuntu2004 courier-msa[4099]:  * Starting Courier MSA server esmtpd-msa
2021-05-26 14:23:31,282: May 26 14:23:25 ubuntu2004 systemd[1]: ESC[0;1;39mESC[0;1;31mESC[0;1;39mcourier-msa.service: Control process exited, code=exited, status=2/INVALIDARGUMENTESC[0m
2021-05-26 14:23:31,282: May 26 14:23:25 ubuntu2004 systemd[1]: ESC[0;1;38;5;185mESC[0;1;39mESC[0;1;38;5;185mcourier-msa.service: Failed with result 'exit-code'.ESC[0m
2021-05-26 14:23:31,282: May 26 14:23:25 ubuntu2004 systemd[1]: ESC[0;1;31mESC[0;1;39mESC[0;1;31mFailed to start LSB: Courier MSA server.ESC[0m
2021-05-26 14:23:31,282: dpkg: error processing package courier-mta (--configure):
2021-05-26 14:23:31,282: installed courier-mta package post-installation script subprocess returned error exit status 1
2021-05-26 14:23:31,282: Setting up expect (5.45.4-2build1) ...
2021-05-26 14:23:31,282: Processing triggers for systemd (245.4-4ubuntu3.6) ...
2021-05-26 14:23:31,282: Processing triggers for man-db (2.9.1-1) ...
2021-05-26 14:23:31,282: Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
2021-05-26 14:23:31,282: Errors were encountered while processing:
2021-05-26 14:23:31,283: courier-mta
2021-05-26 14:23:31,288: failed to install packages with reason: installArchives() failed
Comment 3 Samuel Mannehed cendio 2021-06-01 14:13:29 CEST 
We have found no evidence that the list of packages providing 'mail-transport-agent' is actually manually or intentionally sorted:

https://www.debian.org/doc/debian-policy/ch-binary.html#s-virtual_pkg
https://www.debian.org/doc/debian-policy/ch-relationships.html#s-virtual

Looking at the list from the virtual package called 'x-terminal-emulator', it doesn't seem particularly sorted either:

> Package x-terminal-emulator is a virtual package provided by:
>   xterm 353-1ubuntu1.20.04.2
>   gnome-terminal 3.36.2-1ubuntu1~20.04
>   xiterm+thai 1.10-2build1
>   xfce4-terminal 0.8.9.1-1
>   tilix 1.9.3-4build3
>   termit 3.1-1
>   terminology 1.6.0-2
>   terminator 1.91-4ubuntu1
>   stterm 0.8.2-1
>   sakura 3.7.0-1
>   rxvt-unicode 9.22-6build3
>   qterminal 0.14.1-0ubuntu2
>   pterm 0.73-2
>   mlterm-tiny 3.8.9-1build1
>   mlterm 3.8.9-1build1
>   mate-terminal 1.24.0-2ubuntu1
>   lxterminal 0.3.2-1
>   lilyterm 0.9.9.4+git20150208.f600c0-5
>   konsole 4:19.12.3-0ubuntu1
>   kitty 0.15.0-1build1
>   eterm 0.9.6-6build1
>   deepin-terminal 5.0.0+ds1-1
Comment 4 Samuel Mannehed cendio 2021-06-01 14:18:39 CEST 
But we can side-step the whole question of sorting the list - there is a different virtual package called 'default-mta'. One package chosen by the dist will provide 'default-mta'.

On Ubuntu 18.04, 20.04 and 21.04 'default-mta' is provided by 'postfix'.

On Debian 9 and 10 'default-mta' is provided by 'exim4-daemon-light'.
Comment 6 Samuel Mannehed cendio 2021-06-01 16:50:50 CEST 
Fixed now by switching to default-mta.
Comment 10 Niko Lehto cendio 2021-06-04 13:45:32 CEST 
Tested this on build 2113 on following:

Default Postfix:
✔ Ubuntu 18.04
✔ Ubuntu 20.04

Default Exim:
✔ Debian 9
✔ Debian 10

Tested that each of these did not have any sendmail before, and after tl-setup we had sendmail provided by the default mta. Also verified that our tl-setup log stated that we installed it. Each of these were tested twice.

Release notes looks good!
Comment 11 Pierre Ossman cendio 2021-07-13 16:00:22 CEST 
This got implemented with a flawed assumption that we always need to install sendmail. However if something is already installed then tl-setup will also try to install postfix, which is likely not what the admin wants.
Comment 12 Pierre Ossman cendio 2021-07-20 12:44:02 CEST 
Fixed again by virtue of bug 5980 being fixed.