Comment 1 Pierre Ossman 2021-11-17 13:05:27 CET

Note that modern systems have better support for this, e.g. Wayland and more recent Windows APIs.

What is extra interesting about Wayland is that it has to try to support older X11 applications that used the hacky system. A heuristic is implemented in Xwayland that tries to detect this behaviour from applications and map it to Wayland's cleaner system.

https://lists.x.org/archives/xorg-devel/2016-April/049351.html
https://cgit.freedesktop.org/xorg/xserver/tree/hw/xwayland/xwayland-input.c

Comment 2 Pierre Ossman 2021-11-17 13:08:08 CET

Upstream feature requests for this here:

https://github.com/TigerVNC/tigervnc/issues/619
https://github.com/novnc/noVNC/issues/1493

And PRs that implements VMware's extension for this problem:

https://github.com/TigerVNC/tigervnc/pull/1198
https://github.com/novnc/noVNC/pull/1520

Comment 3 William Sjöblom 2021-11-26 10:56:56 CET

It seems like this did not work straight out of the box. 

I've tried playing both Xonotic and Counter-Strike: Global Offensive along with the pointer lock API in Google Chrome [1] with the recently vendor dropped TigerVNC which incorporates the PRs mentioned in comment 2. All of these show similar symptom:

Xonotic and CS:GO:
- Camera fixed straight down towards the ground after the first slight mouse movement.
- Sideways motion of the mouse results in yaw being applied to the camera, but the remaining degrees of freedom are seemingly fixed.
- Fast mouse movements cause the cursor to leave the screen if the ThinLinc client is set to single monitor full screen on a multi-monitor setup.

Chrome pointer lock:
- Works seemingly fine for slow mouse movements but suffers the same issue as Xonotic and CS:GO when moving the mouse too fast.

[1] https://mdn.github.io/dom-examples/pointer-lock/

Comment 4 William Sjöblom 2021-11-26 11:03:06 CET

In addition to comment 3:

These behaviors were observed using Fedora 34 on both the client and server (i.e. the display server being Wayland on both ends).

Comment 5 William Sjöblom 2021-11-26 11:22:27 CET

Addition to comment 4: the server still use X when used through ThinLinc.

Comment 6 William Sjöblom 2021-11-26 14:13:55 CET

I've now tested the (native) client with both X and Wayland running on Fedora 34 and observed very similar behavior. While performing these tests I've kept `xeyes' running on the client machine to observe any pointer movements.

The Chrome pointer lock API seems to be working satisfactorily assuming that the ThinLinc session is set to full-screen mode over all monitors. When set to single monitor full screen, it also works as intended assuming that the pointer is moved slowly. If the pointer is moved too fast, it escapes the ThinLinc display. When the client is running in windowed mode, the cursor escapes the window regardless of movement speed. 

Xonotic and CS:GO suffers the same problems as regarding keeping the pointer fixed, but with very erratic camera movement as described in comment 3.

This is universal for both X and Wayland.

Comment 7 William Sjöblom 2021-11-26 14:17:34 CET

Regarding `xeyes' running on the client:

For all the above cases the client display server does not detect any pointer movement when the pointer is locked by the application running inside the ThinLinc client.

Comment 8 William Sjöblom 2021-11-29 14:57:58 CET

I have now tested the client on all platforms.

The behavior between all platforms is seemingly identical, but relative mouse movement still has some rough edges.

- The game Tremulous works surprisingly well with the one issue that attempting to look down further than 90 degrees results in the camera starting to spin (this does not happen when playing locally).
- The games CS:GO and Xonotic are completely unplayable since the camera spins very fast on the slightest mouse movement. This behavior is similar to what is seen in Tremulous but is significantly worse.
- Pointer lock API works perfectly fine.

The above points assume that we have full-screen mode enabled over all screens. If  we only enable full-screen on a single screen, fast mouse movements can result in the pointer leaving the screen with the ThinLinc session, which, in combination with a click, results in loss of focus.

Without full-screen mode, the pointer lock functionality is disabled, similar to system key capture (which is expected).

The same problems apply to upstream TigerVNC.

Comment 9 William Sjöblom 2021-11-29 16:48:24 CET

Let's also clarify the previous behavior:

Before vendor dropping the new TigerVNC with these changes, it was simply not possible to constrain the cursor within the ThinLinc session. Thus, using software that relied on relative input from the mouse would simply result in the mouse falling over to the other screen (where events would stop registering), and/or getting further relative motion blocked by the edges of the screen.

Comment 10 William Sjöblom 2021-12-01 17:17:20 CET

Ok, so after a fair bit of hair-pulling I managed to track down the issue. The problem seems to be that Fedora ships SDL compiled only with Wayland support (and thus does not do any XWarpPointer calls for us to intercept). It will require some thought on how to approach issues like these.

Regarding my earlier comments on the cursor escaping the pointer lock when moving the mouse too fast, we check for the pointer being contained within the session window, and if not, we release the pointer lock. My initial thought is that this may cause some trouble if dealing with network congestion and/or high RTTs. The suitability of this approach to pointer locking likely needs more consideration.

Comment 11 William Sjöblom 2021-12-02 09:19:06 CET

Just as expected, synthetically increasing the latency between client and server significantly worsens the issue with the pointer escaping session on fast mouse movements.

The issue stems from the fact that the client stops respecting `setCursorPos' calls if the mouse is not inside the client window. So in case the client's local cursor manages to move outside the client window before the server manages to before the server's call to set the cursor position is delivered we are out of luck. This, of course, requires us to have multiple monitors with the client being in full-screen mode on a subset of these, as the mouse otherwise would have nowhere to escape.

This becomes increasingly problematic for applications that have their pointer locking component close to the screen seam (i.e. the cursor's "fixed" position is close to the screen seam). This scenario is not especially realistic for games (where the pointer grabbing component tends to cover the whole screen), but more so for productivity software that may have multiple panes with relative mouse input, where one or more of these are relatively far to the side.

Comment 12 William Sjöblom 2021-12-02 14:33:45 CET

Worth noting is that the problems with the cursor escaping the session only apply when we capture the mouse without holding any mouse buttons. If a mouse button is held, the pointer stays captured even during fast mouse movements. Holding down a mouse button during pointer capture (e.g. dragging to rotate a 3D model) is likely the most common scenario in the context of productivity applications, so this may not be that big of an issue.

Gaming on the other hand becomes increasingly more fragile as especially FPS games require quick mouse movements without any mouse button held. Solving this issue is certainly possible, but should possibly be seen as a future enhancement since the use case is pretty niche based on our current customer base. I've done some looking around and it seems that the most common approach to this issue in other applications in the remote desktop sphere is to have a setting for absolute or relative mouse mode.

Comment 13 William Sjöblom 2021-12-06 16:26:55 CET

So after some more investigation, the problem was not related to the SDL version as suspected in comment 10, but rather if SDL was compiled with raw mouse input support (which we do not currently support). See https://github.com/TigerVNC/tigervnc/issues/619#issuecomment-986871878 for more details.

Since implementing raw mouse support is a rather large feat compared to the time left until our planned release, we'll postpone that work.

Nonetheless, we still need to verify that the partially complete upstream changes did not introduce any bugs. So what remains, for the time being, is testing the relative cursor changes on all platforms.

Comment 14 William Sjöblom 2021-12-07 14:49:22 CET

Testing was done against two servers running Fedora server 35 and SLES12
respectively. This was done on all native client platforms with server
bundle #2371 and client bundle #2287.

Tested client platforms:
• Fedora 35
  • Wayland
  • X11
• Windows 10
• macOS


• ☑ Relative input that does not use raw mouse input works as intended
  • ☑ Google Chrome 96 is able to use the mouse as a relative input
    input device when using the pointer lock API [1].
  • ☑ Mozilla Firefox 94 is able to use the mouse as a relative input
    input device when using the pointer lock API [1].
  • ☑ F8-menu usable when the pointer is "locked" by one of the above
    applications
  • ☑ It is possible to disconnect when the pointer is "locked" by one
    of the above applications
  • ☑ The pointer is still "locked" when reconnecting if pointer was
    "locked" by one of the above applications when disconnecting.
  • ☑ Emacs mouse avoidance `(mouse-avoidance-mode 'animate)' moves
    the pointer away when approached by the point.
• ☑ Pointer is still grabbed when drag-tiling a window to left- or
  right-hand side of the screen.


On top of this I performed a couple of tests I deem to be 
platform-independent (these were all tested on Fedora 34 running Wayland):

• Running a nested Thinlinc session. In the inner ThinLinc session I
  tested the Pointer Lock API in Firefox without any issues. This was
  done using the following sequence of nested sessions:

  Fedora 34 (Wayland) → SLES12 → Fedora 35

• Using a 4.13.0 client (without the relative mouse changes) to connect
  to a server with the relative mouse changes, testing the pointer lock
  API once again [1]. This worked without any issues (but with relative
  mouse input being all over the place as expected).


[1] Using the following demo https://mdn.github.io/dom-examples/pointer-lock/

Comment 15 William Sjöblom 2021-12-07 14:52:16 CET

So, this concludes this bug for now. I will leave it marked as NEW until we can get raw mouse input as well.