Bug 4290 - SysV services (might) not work on modern systems
Summary: SysV services (might) not work on modern systems
Status: CLOSED FIXED
Alias: None
Product: ThinLinc
Classification: Unclassified
Component: Other (show other bugs)
Version: 3.3.0
Hardware: PC Unknown
: P2 Normal
Target Milestone: 4.12.0
Assignee: Niko Lehto
URL:
Keywords: relnotes
Depends on:
Blocks: 4765 5163 5594 6235 7467 7478
  Show dependency treegraph
 
Reported: 2012-05-10 10:45 CEST by Aaron Sowry
Modified: 2020-06-16 13:12 CEST (History)
4 users (show)

See Also:
Acceptance Criteria:


Attachments

Description Aaron Sowry cendio 2012-05-10 10:45:23 CEST
For better or worse (okay, mostly for worse), it seems that many distributions are switching to systemd. This includes popular "community" distributions such as openSuSE and Fedora, which although are not officially recommended and tested by us, do technically meet our minimum requirements for ThinLinc.

While most services are still shipping SysVinit scripts for backwards compatibility, these will eventually be phased out, and some already have been - for example, there is no longer an init script for sshd on Fedora 17; the service must be controlled using systemctl instead. Our installer will therefore fail to complete with a running SSH server on such systems.

We should also take a look at implementing vsmserver, vsmagent and tlwebadm as systemd services.
Comment 1 Aaron Sowry cendio 2012-06-01 11:07:00 CEST
Perhaps also worth noting is that there is no longer an init script for the iptables service in Fedora.
Comment 2 Aaron Sowry cendio 2013-08-09 12:55:42 CEST
We've made some progress in supporting systemd (e.g. bug #4634), but we can probably do better. For example, see bug #4765 which causes problems on systemd systems.

In general, some parts of ThinLinc seem to assume that SysV init scripts are beung used. We should add compatibility with systemd, and probably start shipping our own systemd service definitions as well. Re-assigning for discussion.
Comment 3 Karl Mikaelsson cendio 2017-06-12 17:57:51 CEST
Adding fuel to the flame war: our services as systemd unit files

http://git.cendio.se/cgit/~derfian/thinlinc.git/commit/?h=bug4290&id=7efefc9371c3ba1d7b10d65244e934eb04e3f1f1

They work just fine even with the SysV init scripts installed in parallel (Fedora 24). Migrating from SysV to systemd requires a bit of thought, systemd didn't recognize that the services were already running. We'd probably want to stop and disable the SysV scripts before enabling the systemd services.
Comment 4 Karl Mikaelsson cendio 2017-10-18 09:36:39 CEST
(In reply to comment #3)
> http://git.cendio.se/cgit/~derfian/thinlinc.git/commit/?h=bug4290&id=7efefc9371c3ba1d7b10d65244e934eb04e3f1f1

Rebased onto SVN r32822 and moved all unit scripts to /usr/lib/systemd/user, as suggested by "pkgconfig systemd --variable=systemduserunitdir" on RHEL 7 and Fedora 26.

New link:
http://git.cendio.se/cgit/~derfian/thinlinc.git/commit/?h=bug4290&id=c88af92031852481b178c6b6404b5330054d6e4e
Comment 7 Karl Mikaelsson cendio 2018-10-17 10:24:07 CEST
Having systemd service files would make it much easier for a system administrator to add local configuration and/or setup to the services without having to modify the shipped init files.

systemd.unit(5) has more info about how this can be done, especially Example 2, Overriding vendor settings.

https://github.com/bjorn-fischer/fss-tools is an example use case for this. In this scenario, the sysadmin wants to add "Delegate=cpu,cpuacct" to a systemd service definition.
Comment 8 Pierre Ossman cendio 2019-08-22 10:25:02 CEST
SLE 15 ships with poor support for SysV services (see bug 7339).

They've also informed us that they will be completely dropping support for SysV services in a future service pack (probably SP3).
Comment 9 Rune Juhl Jacobsen 2020-01-15 23:08:09 CET
Is this ticket going anywhere?

I have a customer currently running ThinLinc 4.9.0 on CentOS 7.7, and the SysV init scripts are broken: when you call `service tlwebadm status` the init script detects that it's running under systemd and calls `systemctl status tlwebadm`, which just returns that everything is fine no matter the actual state of the service...

Since you already seem to have systemd unit files, can I ask why they haven't been added to the packages? As far as I can see none of the RPMs for ThinLinc 4.11.0 have systemd unit files either:

~~~
$ ls -l *.rpm; rpm -qlp *.rpm | grep \\.service || echo 'no service units :('
-rw------- 1 runejuhl runejuhl  256006 Dec 11 04:23 thinlinc-tladm-4.11.0-6323.i686.rpm
-rw------- 1 runejuhl runejuhl  256011 Dec 11 04:23 thinlinc-tladm-4.11.0-6323.x86_64.rpm
-rw------- 1 runejuhl runejuhl 8020475 Dec 11 04:29 thinlinc-tlmisc-4.11.0-6323.i686.rpm
-rw------- 1 runejuhl runejuhl 7789184 Dec 11 04:34 thinlinc-tlmisc-4.11.0-6323.x86_64.rpm
-rw------- 1 runejuhl runejuhl   34207 Dec 11 04:35 thinlinc-tlmisc-libs32-4.11.0-6323.i686.rpm
-rw------- 1 runejuhl runejuhl   33567 Dec 11 04:35 thinlinc-tlmisc-libs-4.11.0-6323.x86_64.rpm
-rw------- 1 runejuhl runejuhl   21717 Dec 11 04:49 thinlinc-tlprinter-4.11.0-6323.noarch.rpm
-rw------- 1 runejuhl runejuhl 8502248 Dec 11 04:42 thinlinc-vnc-server-4.11.0-6323.i686.rpm
-rw------- 1 runejuhl runejuhl 8137723 Dec 11 04:49 thinlinc-vnc-server-4.11.0-6323.x86_64.rpm
-rw------- 1 runejuhl runejuhl  230899 Dec 11 04:22 thinlinc-vsm-4.11.0-6323.i686.rpm
-rw------- 1 runejuhl runejuhl  230593 Dec 11 04:23 thinlinc-vsm-4.11.0-6323.x86_64.rpm
-rw------- 1 runejuhl runejuhl  612543 Dec 11 04:49 thinlinc-webaccess-4.11.0-6323.noarch.rpm
no service units :(
~~~

This loop causes the service to not be able to be started by systemd:

~~~
23:02:32 root@thinlinc01:~
# /etc/init.d/tlwebadm status
+ service=tlwebadm
+ description='ThinLinc Web Administration'
+ lockdir=/var/lock/subsys
+ lockfile=/var/lock/subsys/tlwebadm
+ pidfile=/var/run/tlwebadm.pid
+ SHELL=/bin/bash
+ export SHELL
+ . /opt/thinlinc/libexec/functions
+ systemctl_redirect tlwebadm 'ThinLinc Web Administration' status
+ local service=tlwebadm
+ local 'description=ThinLinc Web Administration'
+ local command=status
+ '[' 17486 -ne 1 ']'
+ '[' -z '' ']'
+ init_is_systemd
+ '[' '!' -x /bin/systemctl ']'
+ '[' '!' -d /cgroup/systemd/system -a '!' -d /sys/fs/cgroup/systemd ']'
+ /bin/systemctl --system
+ return 0
+ case "$command" in
+ exec /bin/systemctl status tlwebadm.service
\u25cf tlwebadm.service - LSB: Start or stop the ThinLinc Web Administration
   Loaded: loaded (/etc/rc.d/init.d/tlwebadm; bad; vendor preset: disabled)
   Active: active (exited) since Wed 2020-01-15 21:21:31 CET; 1h 41min ago
     Docs: man:systemd-sysv-generator(8)

Jan 15 21:21:31 thinlinc01.corp.domain.com systemd[1]: Starting LSB: Start or stop the ThinLinc Web Admi......
Jan 15 21:21:31 thinlinc01.corp.domain.com tlwebadm[3788]: Starting ThinLinc Web Administration
Jan 15 21:21:31 thinlinc01.corp.domain.com systemd[1]: Started LSB: Start or stop the ThinLinc Web Admin...on.
Hint: Some lines were ellipsized, use -l to show in full.
23:02:34 root@thinlinc01:~
# /etc/init.d/tlwebadm start
+ service=tlwebadm
+ description='ThinLinc Web Administration'
+ lockdir=/var/lock/subsys
+ lockfile=/var/lock/subsys/tlwebadm
+ pidfile=/var/run/tlwebadm.pid
+ SHELL=/bin/bash
+ export SHELL
+ . /opt/thinlinc/libexec/functions
+ systemctl_redirect tlwebadm 'ThinLinc Web Administration' start
+ local service=tlwebadm
+ local 'description=ThinLinc Web Administration'
+ local command=start
+ '[' 17486 -ne 1 ']'
+ '[' -z '' ']'
+ init_is_systemd
+ '[' '!' -x /bin/systemctl ']'
+ '[' '!' -d /cgroup/systemd/system -a '!' -d /sys/fs/cgroup/systemd ']'
+ /bin/systemctl --system
+ return 0
+ case "$command" in
+ echo 'Starting ThinLinc Web Administration (via systemctl)'
Starting ThinLinc Web Administration (via systemctl)
+ exec /bin/systemctl start tlwebadm.service
23:02:37 root@thinlinc01:~
# /etc/init.d/tlwebadm status
+ service=tlwebadm
+ description='ThinLinc Web Administration'
+ lockdir=/var/lock/subsys
+ lockfile=/var/lock/subsys/tlwebadm
+ pidfile=/var/run/tlwebadm.pid
+ SHELL=/bin/bash
+ export SHELL
+ . /opt/thinlinc/libexec/functions
+ systemctl_redirect tlwebadm 'ThinLinc Web Administration' status
+ local service=tlwebadm
+ local 'description=ThinLinc Web Administration'
+ local command=status
+ '[' 17486 -ne 1 ']'
+ '[' -z '' ']'
+ init_is_systemd
+ '[' '!' -x /bin/systemctl ']'
+ '[' '!' -d /cgroup/systemd/system -a '!' -d /sys/fs/cgroup/systemd ']'
+ /bin/systemctl --system
+ return 0
+ case "$command" in
+ exec /bin/systemctl status tlwebadm.service
\u25cf tlwebadm.service - LSB: Start or stop the ThinLinc Web Administration
   Loaded: loaded (/etc/rc.d/init.d/tlwebadm; bad; vendor preset: disabled)
   Active: active (exited) since Wed 2020-01-15 21:21:31 CET; 1h 41min ago
     Docs: man:systemd-sysv-generator(8)

Jan 15 21:21:31 thinlinc01.corp.domain.com systemd[1]: Starting LSB: Start or stop the ThinLinc Web Admi......
Jan 15 21:21:31 thinlinc01.corp.domain.com tlwebadm[3788]: Starting ThinLinc Web Administration
Jan 15 21:21:31 thinlinc01.corp.domain.com systemd[1]: Started LSB: Start or stop the ThinLinc Web Admin...on.
Hint: Some lines were ellipsized, use -l to show in full.
~~~

To avoid this bug you have to either stop and then start or restart the service.
Comment 10 Pierre Ossman cendio 2020-01-24 15:52:36 CET
The last non-systemd Ubuntu (14.04) is now EOL. The last non-system SLE (11) is also now EOL. The last non-systemd RHEL (6) is EOL in November.

As such it might now be possible to require systemd and drop SysV support.
Comment 11 Niko Lehto cendio 2020-02-27 16:49:37 CET
For RHEL 7 and Fedora 31 /usr/lib/systemd/system/ seems to be the de facto folder for systemd.unit (service) files. This however is not the case in Ubuntu.

Ubuntu instead wants these installed into /lib/systemd/system/. This behaviour seems to be stable since the code has been in place since 2014 and they explicitly ask for /usr/bin being separated from /bin as stated in: 
  Commit - 0638d09c4021dc56f81f5cb9b637ed8022c5adf8
  Repo   - https://git.launchpad.net/~ubuntu-core-dev/+git/systemd
Comment 12 Rune Juhl Jacobsen 2020-02-28 08:57:30 CET
(In reply to Niko Lehto from comment #11)
> For RHEL 7 and Fedora 31 /usr/lib/systemd/system/ seems to be the de facto
> folder for systemd.unit (service) files. This however is not the case in
> Ubuntu.
> 
> Ubuntu instead wants these installed into /lib/systemd/system/. This
> behaviour seems to be stable since the code has been in place since 2014 and
> they explicitly ask for /usr/bin being separated from /bin as stated in: 
>   Commit - 0638d09c4021dc56f81f5cb9b637ed8022c5adf8
>   Repo   - https://git.launchpad.net/~ubuntu-core-dev/+git/systemd

On RHEL /lib is a symlink to /usr/lib, so installing systemd unit files to /lib/systemd/system should work on any system.
Comment 13 Niko Lehto cendio 2020-03-02 16:36:31 CET
Tested my working copy on Fedora 31, Ubuntu 18 and RHEL 7.

Tests that seem to work:
 - Fresh installation
 - Upgrading from 4.11 to the new server
 - Upgrading new server to a newer server (Just increased build number)

At each step I tested with systemctl that the services was running and running from files in correct paths. These paths are "/lib/systemd/system" on Ubuntu and "/usr/lib/systemd/system" on RHEL/Fedora. Also verified that basic Webaccess, Webadmin and Native client worked fine in each step.

I modified our rpm2deb converter so files is placed in the correct folders on Debian systems. I checked that the modified rpm2deb converter does not modify non intended file paths by comparing new .deb files to old .deb files.

Further testing needed:
 - 32 bit system
Comment 16 Niko Lehto cendio 2020-03-10 14:29:38 CET
(In reply to Niko Lehto from comment #13)

Also tested my working copy on Ubuntu 16.04 (32-bit). Followed the same protocol as earlier:
> 
> Tests that seem to work:
>  - Fresh installation
>  - Upgrading from 4.11 to the new server
>  - Upgrading new server to a newer server (Just increased build number)
> 
> At each step I tested with systemctl that the services was running and
> running from files in correct paths. These paths are "/lib/systemd/system"
> on Ubuntu and "/usr/lib/systemd/system" on RHEL/Fedora. Also verified that
> basic Webaccess, Webadmin and Native client worked fine in each step.
Works fine.
Comment 18 Niko Lehto cendio 2020-03-12 11:24:16 CET
Also checked that logrotate works properly on RHEL7 server. RHEL6 (non-systemd) had crashes during setup, added error handling so it should now write clearer error messages in tlsetup.log too.

For anyone wanting to test this solution this is stuff that can be tested:

Systems:
 - Fedora 31
 - Ubuntu 18.04 (64-bit)
 - Ubuntu 16.04 (32-bit)
 - RHEL 7
 - RHEL 6 (Error handling, installation will not work)

Installation:
 - Fresh install

Upgrade:
 - Old (e.g. 4.11) -> New
 - New -> New(Just increased build number)

For each of these installations/upgrades you can check:
 - Printer setup
   - Do 'nearest' and 'local' printer configuration during setup and see that no errors occur.

 - Services
   - Run command 'systemctl status vsmagent vsmserver tlwebadm tlwebaccess'
   - Check that the services are running and that they are run from the correct file location.
     ("/lib/systemd/system" on Ubuntu and "/usr/lib/systemd/system" on RHEL/Fedora.)

 - Webadmin
   - Log in.
   - Check that vsmserver and vsmagent has correct status.
   - Turn services off/on and see that this actually affects services.

  - Uninstallation
    - Check that removing packages works fine for rpm/deb

Extra tests:
 - Check that logrotate works. You can run 'logrotate -vf CONFIG_FILE' to force rotation. 
   For more info see: https://www.shellhacks.com/logrotate-force-log-rotation/
Comment 20 Alex Tanskanen cendio 2020-03-17 13:51:41 CET
Tested everything that was mentioned in comment #18 on server build 6416 running on Fedora 31, RHEL8 and Ubuntu 18.04. Besides testing Services, Webadmin etc., I I tested basic functionalities for the native client and webaccess in each fresh installation of the different servers. Everything seems to work fine from what I could tell.

On systems which do not support systemd like RHEL6 I verified (on build 6420) that we get a error message in tlsetup.log saying that systemd is needed to run ThinLinc.

The release notes also looks good.
Comment 24 Samuel Mannehed cendio 2020-06-15 13:06:24 CEST
Upgrading from TL 4.11.0 to nightly doesn't work on Fedora 32. The last step of tl-setup that is supposed to start the services fail, this is what the log says:

2020-06-15 13:02:17,052: Starting service 'tlwebadm'...
2020-06-15 13:02:17,256: Output (stderr):
2020-06-15 13:02:17,256:     Job for tlwebadm.service failed because the service did not take the steps required by its unit configuration.
2020-06-15 13:02:17,256:     See "systemctl status tlwebadm.service" and "journalctl -xe" for details.
2020-06-15 13:02:17,256: Failed to start service 'tlwebadm'

And systemctl says:

$ sudo systemctl status tlwebadm
● tlwebadm.service - ThinLinc Web Administration
     Loaded: loaded (/usr/lib/systemd/system/tlwebadm.service; enabled; vendor preset: disabled)
     Active: failed (Result: protocol) since Mon 2020-06-15 13:06:02 CEST; 4s ago
    Process: 852826 ExecStart=/bin/bash --login -c /opt/thinlinc/sbin/tlwebadm (code=exited, status=0/SUCCESS)
        CPU: 135ms

jun 15 13:06:02 samuel.lkpg.cendio.se systemd[1]: Starting ThinLinc Web Administration...
jun 15 13:06:02 samuel.lkpg.cendio.se systemd[1]: tlwebadm.service: Can't open PID file /run/tlwebadm.pid (yet?) after start: Operation not permitted
jun 15 13:06:02 samuel.lkpg.cendio.se systemd[1]: tlwebadm.service: Failed with result 'protocol'.
jun 15 13:06:02 samuel.lkpg.cendio.se systemd[1]: Failed to start ThinLinc Web Administration.
Comment 25 Samuel Mannehed cendio 2020-06-15 14:36:33 CEST
Disregard the above comment. The 4.11.0 installation I upgraded from was broken due to problems with the old approach.

If I have a working 4.11.0 installation (I had to reboot after install) the upgrade to nightly works perfectly fine.
Comment 26 Niko Lehto cendio 2020-06-16 09:17:50 CEST
Found an unupdated line in documentation about services "...by running /opt/thinlinc/libexec/service vsmserver restart or..." in chapter 4.3.

We should  update this and take another look to see if there are more places that we missed to update.
Comment 29 Niko Lehto cendio 2020-06-16 12:06:15 CEST
(In reply to Niko Lehto from comment #26)
> Found an unupdated line in documentation about services "...by running
> /opt/thinlinc/libexec/service vsmserver restart or..." in chapter 4.3.
> 
> We should  update this and take another look to see if there are more places
> that we missed to update.

Nvm, I looked in the wrong version of the documentation. This was already fixed, so the earlier change was unnecessary.
Comment 30 Pierre Ossman cendio 2020-06-16 13:12:12 CEST
Verified a proper revert. Closing again.

Note You need to log in before you can comment on or make changes to this bug.