Bug 5241 - upgrade Xorg
Summary: upgrade Xorg
Status: CLOSED FIXED
Alias: None
Product: ThinLinc
Classification: Unclassified
Component: VNC (show other bugs)
Version: trunk
Hardware: PC Unknown
: P2 Normal
Target Milestone: 4.10.0
Assignee: Pierre Ossman
URL:
Keywords: relnotes, samuel_tester
: 7252 (view as bug list)
Depends on: 7158 7224 7245
Blocks: 400 4834 6177 6234 7139
  Show dependency treegraph
 
Reported: 2014-09-04 16:00 CEST by Pierre Ossman
Modified: 2023-08-30 12:48 CEST (History)
1 user (show)

See Also:
Acceptance Criteria:
* Our Xvnc should be based on the latest stable Xorg release * Xorg dependencies should be upgraded (e.g. all the X11 proto libs), see the list in the serverdeps spec file * The bundled swrast_dri.so should be upgraded to the latest Mesa * The xkb tools and data should be updated to the latest version


Attachments

Description Pierre Ossman cendio 2014-09-04 16:00:06 CEST
We are currently using version 1.14 of the Xorg server for our Xvnc. There has been at least two releases since then, so we should look at upgrading to make sure we get new bug fixes and functionality.
Comment 1 Pierre Ossman cendio 2014-09-05 10:24:15 CEST
Remember to check for new extensions which might require more dependencies in our build system.
Comment 2 Pierre Ossman cendio 2014-12-08 15:00:35 CET
They changed the integration with Mesa in 1.15 that might cause problems for us. See upstream commit be6680967a479eedbcab2fe1718c5f981e1029c7.
Comment 3 Pierre Ossman cendio 2016-06-21 08:01:13 CEST
We are now quite a bit behind everyone else. Even RHEL 6 has been upgraded to 1.17 (from 1.15).
Comment 7 Pierre Ossman cendio 2018-08-29 13:26:14 CEST
We require a newer libdrm, which in turn needs a newer glibc. So we need to do bug 7224 first.
Comment 10 Pierre Ossman cendio 2018-08-30 13:32:08 CEST
A newer mesa is also required, which also needs a newer glibc.
Comment 14 Pierre Ossman cendio 2018-08-31 10:15:44 CEST
All dependencies are now updated and I'm able to do a prototype build with an upgraded Mesa. However I'm not able to get a properly linked Xvnc because of changes to the Xorg/Mesa interaction:

* Xorg no longer includes some Frankenstein copy of parts of Mesa. Instead it expects to link to libGL and get what it needs.

* Xorg still manually loads swrast_dri.so though. (because it cannot use the standard context setup in libGL?)

* Mesa refuses to do a static build with DRI enabled. Not sure how that will interact with loading swrast_dri.so, or the DRI the X11 applications need to use.

So right now I have a runtime dependency on libGL. It seems to work on my Fedora though, but it's unknown what problems that dependency might cause.

More investigation is needed.
Comment 16 Pierre Ossman cendio 2018-08-31 15:32:41 CEST
I tried a static build of Mesa, even though I had to disable a whole bunch of things to get it through. It actually seems to work properly despite the lack of DRI code.

I tested the following:

 * Direct rendering:

   Works well. I can see llvmpipe being reported as the renderer and I see a bunch of llvmpipe threads being active

 * Indirect rendering:

   Also works well. I don't see any traces of llvmpipe though. But then again, I can't see any traces of it in 4.9.0 either.

Since this seems to work I'm not going to try to dig deeper in to the DRI magic.
Comment 17 Pierre Ossman cendio 2018-08-31 15:41:34 CEST
(In reply to comment #16)
> 
>    Also works well. I don't see any traces of llvmpipe though. But then again,
> I can't see any traces of it in 4.9.0 either.
> 

Seems like there are two "swrast", llvmpipe and an simpler older one. We've always used the older one in ThinLinc. So there might be room for improvement. But it would only affect indirect rendering, so it's probably not worth the effort.
Comment 24 Pierre Ossman cendio 2018-09-04 09:28:11 CEST
The linking of swrast_dri.so required some adjustments. Normally it would link to libglapi.so. However we linked that statically in to Xvnc, and we don't want a second copy in swrast_dri.so as that library contains dispatch logic for the OpenGL calls.

So we do a manual link where all the glapi symbols are left undefined and will get resolved as swrast_dri.so is loaded in to Xvnc at runtime. The downside of this is that we can't have the linker check for unresolved symbols and get an early error of any link issues.
Comment 25 Pierre Ossman cendio 2018-09-04 09:29:08 CEST
I had to revert the fix for bug 6044 as that code had gotten a large rewrite upstream. The principle seems to have been altered though, so hopefully the issue is still gone. Will have to re-test.
Comment 27 Pierre Ossman cendio 2018-09-04 13:25:23 CEST
This upgrade also disables indirect rendering by default as upstream consider that code untrustworthy:

    Almost every situation of someone running indirect GLX is a mistake
    that results in X Server crashes.  Indirect GLX is the cause of
    regular security vulnerabilities, and rarely provides any capability
    to the user.  Just disable it unless someone wants to enable it for
    their special use case (using +iglx on the command line).

We should probably mention this in the release notes, but I don't think we need to be proactive with documenting the flag in the TAG (just like most arguments to Xvnc aren't documented).
Comment 31 Pierre Ossman cendio 2018-09-06 15:13:49 CEST
(In reply to comment #25)
> I had to revert the fix for bug 6044 as that code had gotten a large rewrite
> upstream. The principle seems to have been altered though, so hopefully the
> issue is still gone. Will have to re-test.

I retested this on RHEL 6 with Chromium 68. I could reproduce the issue with ThinLinc 4.6.0, but after an upgrade to the nightly build the problem went away.
Comment 32 Pierre Ossman cendio 2018-09-10 09:34:34 CEST
Everything should be done here. Ready for regression testing.
Comment 33 Pierre Ossman cendio 2018-09-12 09:58:30 CEST
I can not find any issues in general usage with normal applications. I've also run some conformance tests and compared results to 4.9.0:


 - rendercheck: no change

42336 tests passed of 42948 total
Successful Groups:
	fill, dcoords, scoords, mcoords, tscoords
	tmcoords, blend, composite, cacomposite, repeat
	bug7366, gtk_argb_xbgr, libreoffice_xrgb

 - xts: Slight improvement

========================
109 of 1078 tests failed
(339 tests were not run)
========================

vs

========================
112 of 1078 tests failed
(337 tests were not run)
========================

 - piglit: Much better

[39411/39411] skip: 15897, pass: 23345, warn: 2, fail: 166, crash: 1         

vs

[39411/39411] skip: 31205, pass: 8112, warn: 2, fail: 88, crash: 4
Comment 34 Pierre Ossman cendio 2018-09-12 09:59:26 CEST
Piglit crashes Xvnc in indirect mode (see bug 3625), so I couldn't test that the same way. I did however run mesa's demos in indirect mode and didn't see any issues.
Comment 35 Pierre Ossman cendio 2018-09-12 12:09:00 CEST
Similar results when running tests on RHEL 6:

 - rendercheck:

42336 tests passed of 42948 total
Successful Groups:
	fill, dcoords, scoords, mcoords, tscoords
	tmcoords, blend, composite, cacomposite, repeat
	bug7366, gtk_argb_xbgr, libreoffice_xrgb

 - xts:

========================
110 of 1078 tests failed
(338 tests were not run)
========================

vs

========================
112 of 1078 tests failed
(337 tests were not run)
========================

 - piglit:

[26100/26100] crash: 9, fail: 382, pass: 11139, skip: 14568, warn: 2 |       

vs

[26100/26100] crash: 8, fail: 98, pass: 6266, skip: 19726, warn: 2 |
Comment 36 Pierre Ossman cendio 2018-09-13 09:28:35 CEST
Performance tests:

 - xrenderbenchmark:

   Mostly the same. Perhaps a few percentages better in some places.

   HOWEVER! We see a massive performance reduction in a bunch of operations. Apparently this is per design and caused by upstream commit 15aa37ad. The old code had rounding errors so they had to resort to a slower code path for these things. Hopefully these are not common operations.

 - gtkperf

   Mostly the same. Text rendering is slower for some reason though. Seems to have something to do with change tracking as I'm seeing a lot of CPU time in the region code. Will investigate further.
Comment 37 Pierre Ossman cendio 2018-09-13 10:57:57 CEST
(In reply to comment #36)
>  - gtkperf
> 
>    Mostly the same. Text rendering is slower for some reason though. Seems to
> have something to do with change tracking as I'm seeing a lot of CPU time in
> the region code. Will investigate further.

I'm having a hard time pinpointing the cause of this. The cause seems to be two-fold:

 - 25% more overhead going from 4.9.0 to new TigerVNC / old Xorg. However the old version gets bogged down by the text rendering and fails to send updates to the client. So the user experience is a lot better now. And more updates means more CPU, so this is natural.

 - 25% more overhead going from old Xorg to new Xorg (both with new TigerVNC). This is where it spends more CPU tracking changes. It is also spending more time in select(). Perhaps the select() loop is rewritten, causing the pattern of updates to change, in turn causing the complexity of the changed region between updates to change?

It is still very fast, so I don't think it's worth digging further in to.
Comment 38 Pierre Ossman cendio 2018-09-13 13:24:34 CEST
Everything seems to work fine and I can't think of any more things to test.

Someone else should have a look that the release notes look fine, and then we should be done here.
Comment 39 Karl Mikaelsson cendio 2018-09-14 15:37:03 CEST
(In reply to comment #38)
> Everything seems to work fine and I can't think of any more things to test.
> 
> Someone else should have a look that the release notes look fine, and then we
> should be done here.

Release notes look fine.
Comment 40 Karl Mikaelsson cendio 2018-09-17 15:42:26 CEST
> * Our Xvnc should be based on the latest stable Xorg release
> * Xorg dependencies should be upgraded (e.g. all the X11 proto libs)

Mesa, makedepend, libSM, libICE and libXxf86misc are not up to date.

| component    | cenbuild | latest | up to date? |
|--------------+----------+--------+-------------|
| libICE       |    1.0.4 |  1.0.9 | n           |
| libSM        |    1.1.0 |  1.2.2 | n           |
| libXxf86misc |    1.0.1 |  1.0.4 | n           |
| makedepend   |    1.0.4 |  1.0.5 | n           |
| mesa         |   18.1.7 | 18.1.8 | n           |
|--------------+----------+--------+-------------|
| libX11       |    1.6.6 |  1.6.6 | y           |
| libXau       |    1.0.8 |  1.0.8 | y           |
| libXdamage   |    1.1.4 |  1.1.4 | y           |
| libXdmcp     |    1.1.2 |  1.1.2 | y           |
| libXext      |    1.3.3 |  1.3.3 | y           |
| libXfixes    |    5.0.3 |  5.0.3 | y           |
| libXfont     |    2.0.3 |  2.0.3 | y           |
| libXrandr    |    1.5.1 |  1.5.1 | y           |
| libXxf86vm   |    1.1.4 |  1.1.4 | y           |
| libdrm       |   2.4.94 | 2.4.94 | y           |
| libxkbfile   |    1.0.9 |  1.0.9 | y           |
| libxshmfence |      1.3 |    1.3 | y           |
| pixman       |   0.34.0 | 0.34.0 | y           |
| xcb-proto    |     1.13 |   1.13 | y           |
| font-util    |    1.3.1 |  1.3.1 | y           |
| util-macros  |   1.19.2 | 1.19.2 | y           |
| xorgproto    |   2018.4 | 2018.4 | y           |
| xtrans       |    1.3.5 |  1.3.5 | y           |
|--------------+----------+--------+-------------|


> * The bundled swrast_dri.so should be upgraded to the latest Mesa

Mesa is not up to date.

| component |    ctc | latest | up to date? |
|-----------+--------+--------+-------------|
| vnc/mesa  | 18.1.7 | 18.1.8 | n           |
|-----------+--------+--------+-------------|


> * The xkb tools and data should be updated to the latest version

| component            |   ctc | latest | up to date? |
|----------------------+-------+--------+-------------|
| vnc/setxkbmap        | 1.3.1 |  1.3.1 | y           |
| vnc/xkeyboard-config |  2.24 |   2.24 | y           |
| vnc/xkbcomp          | 1.4.2 |  1.4.2 | y           |
|----------------------+-------+--------+-------------|
Comment 44 Pierre Ossman cendio 2018-09-18 15:01:46 CEST
(In reply to comment #40)
> > * Our Xvnc should be based on the latest stable Xorg release
> > * Xorg dependencies should be upgraded (e.g. all the X11 proto libs)
> 
> Mesa, makedepend, libSM, libICE and libXxf86misc are not up to date.
> 

Mesa has been updated. makedepend isn't actually needed anymore (I forgot to update the serverdeps list when doing the initial upgrade).

The last three libraries are not used by Xvnc so they have not been changed.
Comment 45 Pierre Ossman cendio 2018-09-27 14:28:48 CEST
*** Bug 7252 has been marked as a duplicate of this bug. ***
Comment 47 Samuel Mannehed cendio 2018-09-27 14:54:23 CEST
The required dependencies have now been upgraded and the release note looks good.

Note You need to log in before you can comment on or make changes to this bug.