Bug 4586 - Handle distributions without Python 2
Summary: Handle distributions without Python 2
Status: CLOSED FIXED
Alias: None
Product: ThinLinc
Classification: Unclassified
Component: Server OS (show other bugs)
Target Milestone: 4.13.0
Assignee: Pierre Ossman
URL:
Keywords: relnotes
: 4867 (view as bug list)
Depends on: 4867 5742 6110 7546 7552 7553 7554 7558 7566 7572 7587 7649 7688 7691 7692 7702 7703 7727 7729 7730 7732 7733 7735 7736
Blocks: 5102 4245 7330 7597 7613
  Show dependency treegraph
 
Reported: 2013-04-08 09:49 CEST by Aaron Sowry
Modified: 2021-07-14 16:04 CEST (History)
7 users (show)

See Also:
Acceptance Criteria:


Attachments

Description Aaron Sowry cendio 2013-04-08 09:49:36 CEST
According to https://wiki.ubuntu.com/RaringRingtail/TechnicalOverview:

"We eventually intend to ship only Python 3 with the Ubuntu desktop image, not Python 2. This Ubuntu 13.04 Beta continues this process, although we will not be able to convert everything to Python 3 for Ubuntu 13.04."

Also, according to https://wiki.ubuntu.com/Python/3:

"It is a release goal for Ubuntu 14.04 LTS to have only Python 3 on the desktop CD images."

Since LTS releases are potentially of interest for us in terms of recommended/tested distributions, perhaps we should discuss how this might affect ThinLinc on this platform.
Comment 1 Pierre Ossman cendio 2013-10-24 09:33:49 CEST
Fedora are also aiming for only Python 3 in the default installation:

https://fedoraproject.org/wiki/Changes/Python_3_as_Default
Comment 2 Pierre Ossman cendio 2013-10-24 09:43:34 CEST
Also see bug 4867.
Comment 3 Pierre Ossman cendio 2016-04-01 14:17:46 CEST
The Ubuntu 16.04 Beta default server installation does not include Python 2. It therefore also does not include python-apt by default, preventing automatic package installation.

There are also no desktop alternatives in the server installer, so it might be that no option (except manually selecting the python package) gives you Python 2 during installation.
Comment 4 Pierre Ossman cendio 2016-04-01 14:35:29 CEST
Ubuntu 16.04 Desktop version includes Python 2 by default at least, but not python-apt.
Comment 6 Pierre Ossman cendio 2016-07-07 13:26:01 CEST
Bug 5742 has implemented checks for Python and PyGTK and tries to help the user install these.

Platform Specific Notes have also been written for both Red Hat/Fedora and Ubuntu. SLE/openSUSE still ship Python 2 even in the minimal install so they do not need any action yet.
Comment 7 Peter Åstrand cendio 2016-08-09 12:23:48 CEST
(In reply to comment #6)

> Platform Specific Notes have also been written for both Red Hat/Fedora and
> Ubuntu. 

Looks good.
Comment 8 Pierre Ossman cendio 2017-12-11 12:02:02 CET
More Ubuntu updates:

https://lists.ubuntu.com/archives/ubuntu-devel-announce/2017-December/001234.html
Comment 9 Peter Åstrand cendio 2018-06-26 07:45:52 CEST
https://lwn.net/Articles/756628/
Comment 10 Pierre Ossman cendio 2018-12-04 14:39:20 CET
RHEL 8 beta is now out, and this is their position on Python:

> Python 3.6 is the default Python implementation in RHEL 8; limited support for Python 2.7 is provided. No version of Python is installed by default.

A bit problematic that we can't even assume Python 3 is installed going forward...

More details here:

https://developers.redhat.com/blog/2018/11/27/what-no-python-in-rhel-8-beta/
Comment 11 Pierre Ossman cendio 2018-12-04 15:12:59 CET
Even more details about Python in RHEL 8:

https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/

One thing that is missing is a clear indication when Red Hat will stop supporting Python 2.
Comment 12 Samuel Mannehed cendio 2019-11-04 13:06:17 CET
"/usr/bin/python" now points to python 3 on Fedora 31.

Python 2 will be removed in Fedora version 32:

https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3
Comment 13 Pierre Ossman cendio 2020-02-04 12:59:52 CET
*** Bug 4867 has been marked as a duplicate of this bug. ***
Comment 14 Philipp Rehs 2020-02-18 09:48:03 CET
Is there any new progress on this issue? Archlinux is also currently removing python2 from its repos
Comment 15 Pierre Ossman cendio 2020-02-18 10:04:35 CET
Yes, it is something we intend to address in the coming releases.
Comment 16 Pierre Ossman cendio 2020-09-08 16:09:28 CEST
I've checked the major tools for helping out with conversion, and they are:

2to3: The original tool from the Python project. Meant for a conversion from
      Python 2 only code to Python 3 only code.

modernize: Variant of 2to3 that focuses on making Python 2 code compatible with
           Python 3.

futurize: Another variant that builds on modernize, but reverses things and tries
          to make Python 3 code compatible with Python 2.

As we have no interest in keeping Python 2 compatibility it seems like 2to3 is the main tool for us.

One major gotcha about it is that it stays clear of the mess around divisions. So those will need to be manually inspected when we convert things.
Comment 17 Linn cendio 2020-09-11 09:10:30 CEST
Cheetah3 only supports Python 2.7 and Python 3.4 and newer. This means that in order to use Cheetah3 we need to take this into consideration.

Documentation for Cheetah3 updates:
https://cheetahtemplate.org/news.html

See bug 7553.
Comment 18 Pierre Ossman cendio 2020-09-21 15:47:24 CEST
Apparently 2to3 doesn't enable all its fixers by default (despite what --help says). We probably want to use this line:

> 2to3 -f all -f buffer -f idioms -f set_literal
Comment 20 Pierre Ossman cendio 2020-10-06 16:44:34 CEST
One question is what version of Python 3 we should require. Red Hat are using 3.6 for RHEL 7 and 3.8 for RHEL 8. Ubuntu has 3.5 for 16.04, 3.6 for 18.04 and 3.8 for 20.04. SUSE has 3.4 for SLE 12 and 3.6 for SLE 15.

So it looks like 3.4 is the newest version we can require if we want to support all currently supported versions of the enterprise distributions.
Comment 22 Pierre Ossman cendio 2020-10-27 09:34:35 CET
Python 3 now requires unicode strings for many system APIs (e.g. getpwnam()), so we need to start removing calls to locale_encode()/locale_decode() as part of the conversion. Fortunately Python 3 behaves like we want, so users will get the same functionality.
Comment 26 Pierre Ossman cendio 2020-12-02 15:24:09 CET
When using development mode to catch issues, we also get this junk on stop:

> Debug memory block at address p=0x275cad0: API '0'
>     17293822569102704640 bytes originally requested
>     The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
>         at p-7: 0x00 *** OUCH
>         at p-6: 0x00 *** OUCH
>         at p-5: 0x00 *** OUCH
>         at p-4: 0x00 *** OUCH
>         at p-3: 0x00 *** OUCH
>         at p-2: 0x00 *** OUCH
>         at p-1: 0x00 *** OUCH
>     Because memory is corrupted at the start, the count of bytes requested
>        may be bogus, and checking the trailing pad bytes may segfault.
>     The 8 pad bytes at tail=0xf00000000275cad0 are 

followed by a segfault.

This is some kind of bug in Python 3.8+. I've reported it upstream, but for now we'll just have to ignore it:

https://bugs.python.org/issue42540
Comment 27 Frida cendio 2021-04-16 08:26:43 CEST
Right now we have a deprecation warning when loading licenses. This should be fixed when we remove Python 2 support from LicenseHandler. We could not do this when solving bug 7630 since tl-collect-licensestats needs Python 2 support.

> 2021-04-15 09:05:20 INFO vsmserver.license: Updating license data from disk to memory
> /opt/thinlinc/modules/thinlinc/vsm/licensehandler.py:387: DeprecationWarning: This method will be removed in future versions.  Use 'parser.read_file()' instead.
>   I1iiIII1i . readfp ( iII11iI1i )
> 2021-04-15 09:05:20 INFO vsmserver.license: License summary: 50 concurrent users. Hard limit of 55 concurrent users.
Comment 28 Pierre Ossman cendio 2021-05-20 16:20:29 CEST
We have some hacks to clean out old .pyc files in tlsetup.py and thinlinc-tlmisc.spec.in. As we no longer use Python 2 we can remove those.
Comment 41 Niko Lehto cendio 2021-06-29 11:20:07 CEST
Tested tl-session-param on a Fedora 33 server after conversion to Python 3.
Works good but I get the following resource warning from prefix.py:

>[cendio@lab-143 bin]$ python3 -Xdev -Xtracemalloc=5 tl-session-param -a >/client_params/
>/opt/thinlinc/modules/thinlinc/prefix.py:75: ResourceWarning: unclosed file <_io.TextIOWrapper name='/var/opt/thinlinc/sessions/cendio/11/sessnum' mode='r' >encoding='UTF-8'>
>  I1I1 = open ( Ooo0OO ) . read ( )
>Object allocated at (most recent call last):
>  File "/opt/thinlinc/bin/tl-session-param", lineno 143
>    IIiIiiIiI ( )
>  File "/opt/thinlinc/bin/tl-session-param", lineno 104
>    i1I11ii = get_public_sessioninfo ( oOo0O00O0ooo )
>  File "/opt/thinlinc/modules/thinlinc/sessioninfo.py", lineno 190
>    display = prefix . get_tl_sessnum ( oO00OOooOoO0o )
>  File "/opt/thinlinc/modules/thinlinc/prefix.py", lineno 75
>    I1I1 = open ( Ooo0OO ) . read ( )
Comment 51 Frida cendio 2021-07-01 13:20:15 CEST
I have done some cleanup in vsm.extproc, vsm.subcluster and vsm.uid_gid_util. I also deleted the thinlinc.locale module since it's not needed anymore. I have now tested the effected areas.

Tested extproc on RHEL 8:
✓ Session startup scripts are run.
✓ Unbind ports works.
✓ License warning mail sent to admin.


To make sure that the last parts of thinlinc.locale was successfully removed I tested some non-ASCII handling on Ubuntu 20.10:

✓ User 't€stör' can create a session.
✓ If 't€stör' tries to shadow a session when not allowed the name is printed correct in the log.
✓ If 't€stör' is associated with a subcluster by name, the correct subcluster is chosen when starting a new session.

I also grepped for 'locale_encode' and 'locale_decode' in the installed ThinLinc tree. Found nothing.
Comment 70 Frida cendio 2021-07-06 13:27:36 CEST
I can verify that tl-sso-fromfile and tl-sso-sendkeys has been successfully removed from the product and no traces are left of the scripts.
Comment 71 Pierre Ossman cendio 2021-07-06 13:29:11 CEST
Tested tl-wait-smartcard on Ubuntu 20.04:

 ✓ No client smart card support (webaccess)
 ✓ Waiting for reader (manually delayed client connection)
 ✓ Timeout
Comment 72 Pierre Ossman cendio 2021-07-06 13:37:41 CEST
Tested tl-memberof-group on Ubuntu 20.04:

 ✓ Invalid group
 ✓ Group not member of
 ✓ Primary group
 ✓ Secondary group
 ✓ Multiple groups, all wrong
 ✓ Multiple groups, one matching
Comment 73 Frida cendio 2021-07-06 14:20:59 CEST
Tested tl-collect-licensestats on Ubuntu 20.10.

 ✓ License log is correct
 ✓ Correct license info is shown in tlwebadm/status

Also tested some general license handling. 

 ✓ License warning mail sent to admin
 ✓ Blocking new users when hitting hard limit
 ✓ Loading license files
Comment 75 Pierre Ossman cendio 2021-07-06 14:38:30 CEST
Tested tl-env on Ubuntu 20.10:

 ✓ Missing xstartup.env
 ✓ Invalid display
 ✓ Default options
 ✓ Explicit display
 ✓ Exclude $DISPLAY
Comment 76 Frida cendio 2021-07-06 15:37:04 CEST
Tested tl-session-param on Fedora 34. Works fine

 ✓ Listing all parameters
 ✓ Listing values and directories 
 ✓ Printing for environment variable
 ✓ Not a folder
 ✓ No such parameter
 ✓ Using option -e on a folder


Also made sure there are no resource warnings as mentioned by Niko in comment #41.
Comment 77 Frida cendio 2021-07-07 12:54:03 CEST
The Python 2 support has now been removed from thinlinc.authutil, thinlinc.kdf and thinlinc.crypt. I have tested that the modules still works as intended on Fedora 34.

 ✓ Logging in with tlwebaccess
 ✓ auth_cookie looks random and works
 ✓ tlwebadm password look random and works
 ✓ tl-gen-auth
 ✓ tl-sso-update-password
 ✓ tl-sso-password
Comment 78 Frida cendio 2021-07-07 13:04:21 CEST
Tested this on Fedora 34: 

 ✓ tl-sso-update-password (non-ASCII)
 ✓ tl-sso-password (non-ASCII)

This means that the sso module is tested after Python 2 support was removed.
Comment 79 Frida cendio 2021-07-07 15:37:45 CEST
Asyncbase's Python 2 support has been removed and I have tested this to make sure everything works as before: 

 ✓ General session handling (Create, Reconnect, terminate)
 ✓ Stopping/starting vsmagent/vsmserver
 ✓ Delayed calls, periodic update of load info
 ✓ Terminated session cleanup

Tested build 2199 on Fedora 34.
Comment 88 Pierre Ossman cendio 2021-07-08 14:35:25 CEST
Tested thinlinc.prefix after cleanup on SLES 12:

 ✓ tl-mount-localdrives works (uses get_tl_sessnum())
Comment 89 Pierre Ossman cendio 2021-07-08 14:37:59 CEST
Tested thinlinc.sessinfo after cleanup on SLES 12:

 ✓ Agent hostname (IP) correctly detected
 ✓ tl-mount-localdrives (fetches session info)
Comment 90 Pierre Ossman cendio 2021-07-08 15:42:53 CEST
Tested thinlinc.crypto after cleanup on SLES 12:

 ✓ Valid licenses work
 ✓ Manipulated licenses are rejected
Comment 91 Pierre Ossman cendio 2021-07-08 16:45:16 CEST
Tested thinlinc.selinux after cleanup on RHEL 8:

 ✓ Everything is correctly configured by tlsetup (users, module, file contexts)
Comment 117 William Sjöblom cendio 2021-07-14 15:58:36 CEST
Running grep with the patterns attached in comment 116 on the non-vendor parts of the tree now only results in a bunch of false positives. We can therefore conclude this long journey towards a fully Python 3 converted ThinLinc.

Marking as closed.

Note You need to log in before you can comment on or make changes to this bug.