Bug 8328 - vsmserver crashes on upcoming Python 3.13
Summary: vsmserver crashes on upcoming Python 3.13
Status: CLOSED FIXED
Alias: None
Product: ThinLinc
Classification: Unclassified
Component: VSM Server (show other bugs)
Version: trunk
Hardware: PC Unknown
: P2 Normal
Target Milestone: 4.17.0
Assignee: Tobias
URL:
Keywords: prosaic
Depends on:
Blocks:
 
Reported: 2024-03-21 15:39 CET by Pierre Ossman
Modified: 2024-04-09 16:03 CEST (History)
2 users (show)

See Also:
Acceptance Criteria:
MUST: - Our patching of asyncio logging must not cause a crash with the upcoming Python 3.13 updates. SHOULD: - Future developers should be signaled when this patching is redundant.


Attachments

Description Pierre Ossman cendio 2024-03-21 15:39:52 CET
When running ThinLinc on the upcoming Python 3.13¹, it will crash² like this:

> 2024-03-21 11:54:00 WARNING vsmserver.loadinfo: Error getting load from agent 127.0.0.1: ii1Ii() takes 2 positional arguments but 3 were given
> 2024-03-21 11:54:00 WARNING vsmserver.loadinfo: Marking agent 127.0.0.1 as down
> Traceback (most recent call last):
>   File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
>     self._context.run(self._callback, *self._args)
>   File "/usr/lib/python3.10/asyncio/selector_events.py", line 531, in _sock_write_done
>     self.remove_writer(fd)
>   File "/usr/lib/python3.10/asyncio/selector_events.py", line 346, in remove_writer
>     return self._remove_writer(fd)
>   File "/usr/lib/python3.10/asyncio/selector_events.py", line 323, in _remove_writer
>     writer.cancel()
>   File "/usr/lib/python3.10/asyncio/events.py", line 71, in cancel
>     self._repr = repr(self)
>   File "/usr/lib/python3.10/asyncio/events.py", line 61, in __repr__
>     info = self._repr_info()
>   File "/usr/lib/python3.10/asyncio/events.py", line 51, in _repr_info
>     info.append(format_helpers._format_callback_source(
>   File "/usr/lib/python3.10/asyncio/format_helpers.py", line 23, in _format_callback_source
>     func_repr = _format_callback(func, args, None)
>   File "/usr/lib/python3.10/asyncio/format_helpers.py", line 56, in _format_callback
>     func_repr += _format_args_and_kwargs(args, kwargs, False)
> TypeError: ii1Ii() takes 2 positional arguments but 3 were given
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "/opt/thinlinc/sbin/vsmserver", line 119, in OO0o0O0o0
>     iIIiii1iI . run_forever ( )
>   File "/usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
>     self._run_once()
>   File "/usr/lib/python3.10/asyncio/base_events.py", line 1901, in _run_once
>     handle._run()
>   File "/usr/lib/python3.10/asyncio/events.py", line 84, in _run
>     cb = format_helpers._format_callback_source(
>   File "/usr/lib/python3.10/asyncio/format_helpers.py", line 23, in _format_callback_source
>     func_repr = _format_callback(func, args, None)
>   File "/usr/lib/python3.10/asyncio/format_helpers.py", line 46, in _format_callback
>     suffix = _format_args_and_kwargs(args, kwargs, False) + suffix
> TypeError: ii1Ii() takes 2 positional arguments but 3 were given
> > During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "/opt/thinlinc/sbin/vsmserver", line 107, in o0OoOOo00OOO
>     OO0o0O0o0 ( OOO00oO0oOo0O )
>   File "/opt/thinlinc/sbin/vsmserver", line 128, in OO0o0O0o0
>     iIIiii1iI . close ( )
>   File "/usr/lib/python3.10/asyncio/unix_events.py", line 68, in close
>     super().close()
>   File "/usr/lib/python3.10/asyncio/selector_events.py", line 87, in close
>     self._close_self_pipe()
>   File "/usr/lib/python3.10/asyncio/selector_events.py", line 94, in _close_self_pipe
>     self._remove_reader(self._ssock.fileno())
>   File "/usr/lib/python3.10/asyncio/selector_events.py", line 284, in _remove_reader
>     reader.cancel()
>   File "/usr/lib/python3.10/asyncio/events.py", line 71, in cancel
>     self._repr = repr(self)
>   File "/usr/lib/python3.10/asyncio/events.py", line 61, in __repr__
>     info = self._repr_info()
>   File "/usr/lib/python3.10/asyncio/events.py", line 51, in _repr_info
>     info.append(format_helpers._format_callback_source(
>   File "/usr/lib/python3.10/asyncio/format_helpers.py", line 23, in _format_callback_source
>     func_repr = _format_callback(func, args, None)
>   File "/usr/lib/python3.10/asyncio/format_helpers.py", line 56, in _format_callback
>     func_repr += _format_args_and_kwargs(args, kwargs, False)
> TypeError: ii1Ii() takes 2 positional arguments but 3 were given
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "/opt/thinlinc/sbin/vsmserver", line 152, in <module>
>     o0OoOOo00OOO ( )
>   File "/opt/thinlinc/sbin/vsmserver", line 109, in o0OoOOo00OOO
>     oOoO00 . close ( )
>   File "/opt/thinlinc/modules/thinlinc/vsm/vsmserver.py", line 158, in close
>     self . _server . close ( )
>   File "/usr/lib/python3.10/asyncio/base_events.py", line 342, in close
>     self._loop._stop_serving(sock)
>   File "/usr/lib/python3.10/asyncio/selector_events.py", line 615, in _stop_serving
>     self._remove_reader(sock.fileno())
>   File "/usr/lib/python3.10/asyncio/selector_events.py", line 284, in _remove_reader
>     reader.cancel()
>   File "/usr/lib/python3.10/asyncio/events.py", line 71, in cancel
>     self._repr = repr(self)
>   File "/usr/lib/python3.10/asyncio/events.py", line 61, in __repr__
>     info = self._repr_info()
>   File "/usr/lib/python3.10/asyncio/events.py", line 51, in _repr_info
>     info.append(format_helpers._format_callback_source(
>   File "/usr/lib/python3.10/asyncio/format_helpers.py", line 23, in _format_callback_source
>     func_repr = _format_callback(func, args, None)
>   File "/usr/lib/python3.10/asyncio/format_helpers.py", line 56, in _format_callback
>     func_repr += _format_args_and_kwargs(args, kwargs, False)
> TypeError: ii1Ii() takes 2 positional arguments but 3 were given
> sys:1: ResourceWarning: unclosed <socket.socket fd=7, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('::', 9000, 0, 0)>

The cause is the workaround we did on bug 8260. We got upstream to resolve the issue properly, but unfortunately in a way that is incompatible with our workaround.

¹ I didn't really test 3.13, but rather simulated the change on Python 3.10
² Technically, the service is still running, but fails to do something useful
Comment 1 Tobias cendio 2024-04-04 09:27:08 CEST
The upstream update mentioned in comment #0:
https://github.com/python/cpython/pull/115667
Comment 3 Tobias cendio 2024-04-05 13:12:37 CEST
This issue has been resolved by simply not patching when we detect that the relevant upstream updates are in place.

Tested using server build #3545 on Fedora39 (python 3.12.1) and RHEL9 (python 3.9.18). Python "3.13.0" was simulated by applying the commit mentioned in comment #1 on the corresponding python libraries.

         | python 3.9 | python 3.12 | python "3.13"
---------------------------------------------------
pre-fix  | OK         | OK          | CRASH
---------------------------------------------------
post-fix | OK         | OK          | OK

The crash could be observed with pre-fix code inappropriately patching 3.13, and conversely, no crash observed post-fix. Furthermore, things were working as before with post-fix code in place when running contemporary python versions.
Comment 4 Tobias cendio 2024-04-05 13:14:53 CEST
MUST:
> Our patching of asyncio logging must not cause a crash with the upcoming Python 3.13 updates.
✅ It doesn't.

SHOULD:
> Future developers should be signaled when this patching is redundant.
✅ They are: code comments have been left for future developers.


--------
Setting this bug as resolved.
Comment 7 Alexander Zeijlon cendio 2024-04-09 16:03:20 CEST
I tested the behavior of the patch function with and without the 3.13-changes applied, and could see that the new patch behavior prevents the crash that we could see in comment 0.

The commits look good, closing.

Note You need to log in before you can comment on or make changes to this bug.