Bug 5492 - crash in Xvnc analyseRect()
Summary: crash in Xvnc analyseRect()
Status: CLOSED FIXED
Alias: None
Product: ThinLinc
Classification: Unclassified
Component: VNC (show other bugs)
Version: pre-1.0
Hardware: PC Unknown
: P2 Normal
Target Milestone: 4.5.0
Assignee: Pierre Ossman
URL:
Keywords: relnotes, samuel_tester
Depends on:
Blocks:
 
Reported: 2015-04-13 09:48 CEST by Pierre Ossman
Modified: 2017-01-04 16:24 CET (History)
2 users (show)

See Also:
Acceptance Criteria:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Description Pierre Ossman cendio 2015-04-13 09:48:40 CEST 
We've had several reports of Xvnc crashing with this backtrace:

> (EE) 
> (EE) Backtrace:
> (EE) 0: /opt/thinlinc/libexec/Xvnc (xorg_backtrace+0x36) [0x5dac96]
> (EE) 1: /opt/thinlinc/libexec/Xvnc (0x400000+0x1deb69) [0x5deb69]
> (EE) 2: /lib64/libpthread.so.0 (0x7f0195916000+0xf710) [0x7f0195925710]
> (EE) 3: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager11analyseRectEPKNS_11PixelBufferEPNS_8RectInfoEi+0x387) [0x61aed7]
> (EE) 4: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager12writeSubRectERKNS_4RectEPKNS_11PixelBufferE+0xef) [0x61bdcf]
> (EE) 5: /opt/thinlinc/libexec/Xvnc (_ZN3rfb13EncodeManager11writeUpdateERKNS_10UpdateInfoEPKNS_11PixelBufferEPKNS_14RenderedCursorE+0xfc) [0x61c96c]
> (EE) 6: /opt/thinlinc/libexec/Xvnc (_ZN3rfb16VNCSConnectionST22writeFramebufferUpdateEv+0x4eb) [0x61600b]
> (EE) 7: /opt/thinlinc/libexec/Xvnc (_ZN3rfb16VNCSConnectionST15processMessagesEv+0xdc) [0x61676c]
> (EE) 8: /opt/thinlinc/libexec/Xvnc (_ZN14XserverDesktop13wakeupHandlerEP6fd_seti+0xe8) [0x5f7858]
> (EE) 9: /opt/thinlinc/libexec/Xvnc (0x400000+0x1ee1b4) [0x5ee1b4]
> (EE) 10: /opt/thinlinc/libexec/Xvnc (WakeupHandler+0x5b) [0x57f9fb]
> (EE) 11: /opt/thinlinc/libexec/Xvnc (WaitForSomething+0x4b6) [0x5d7976]
> (EE) 12: /opt/thinlinc/libexec/Xvnc (Dispatch+0xb2) [0x57ae92]
> (EE) 13: /opt/thinlinc/libexec/Xvnc (main+0x3da) [0x56888a]
> (EE) 14: /lib64/libc.so.6 (__libc_start_main+0xfd) [0x7f01955a0d5d]
> (EE) 15: /opt/thinlinc/libexec/Xvnc (0x400000+0x57529) [0x457529]
> (EE) 
> (EE) Segmentation fault at address 0x0
> 
> Fatal server error:
> Caught signal 11 (Segmentation fault). Server aborting

We can't see any obvious cause for this in the code, and we have yet to find a way to reproduce it. For now we'll just have to keep an eye on it and see if more information surfaces.
Comment 2 Pierre Ossman cendio 2015-06-08 12:18:55 CEST 
We've gotten a decent core dump with proper debug info now.

The problem is that we're feeding an empty rect into the encoder. And it is caused by an empty server side cursor rect. This shouldn't really happen, but the logic for this is very messy so there is probably a bug in the handling. No idea how to provoke it though.
Comment 3 Pierre Ossman cendio 2015-06-09 16:16:00 CEST 
Hopefully fixed in r30365 by cleaning up the logic.

I couldn't figure out a way to provoke this, so the tester will have to make due with merely testing server side cursor rendering. This is most easily done by exploiting bug 2251.
Comment 5 Pierre Ossman cendio 2015-06-23 15:29:01 CEST 
This vendor drop causes Xvnc to consume 100% CPU whilst a client is connected.
Comment 6 Pierre Ossman cendio 2015-06-23 15:36:27 CEST 
Refixed in r30484.
Note You need to log in before you can comment on or make changes to this bug.