Bug 4416 - Xvnc crashes on Gnome login (composite)
Summary: Xvnc crashes on Gnome login (composite)
Status: CLOSED FIXED
Alias: None
Product: ThinLinc
Classification: Unclassified
Component: VNC (show other bugs)
Version: 3.4.0
Hardware: PC Unknown
: P2 Normal
Target Milestone: 4.1.0
Assignee: Pierre Ossman
URL:
Keywords: hean01_tester
Depends on: 4417
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-05 14:11 CEST by Pierre Ossman
Modified: 2013-06-24 10:03 CEST (History)
1 user (show)

See Also:
Acceptance Criteria:


Attachments

Description Pierre Ossman cendio 2012-10-05 14:11:24 CEST
Reported in Issue 13451:

Xvnc crashes when Gnome starts up, but only for some users. Managed to get a core file finally, and it gives us this backtrace:

(gdb) bt
#0  0x00000000006ae784 in sse2_composite_src_x888_8888 ()
#1  0x00000000006690b5 in pixman_image_composite32 ()
#2  0x00000000004578fd in fbComposite (op=1 '\001', pSrc=0x1294100, pMask=0x0, 
    pDst=0x12944e0, xSrc=<optimized out>, ySrc=<optimized out>, xMask=0, 
    yMask=0, xDst=0, yDst=0, width=30, height=26) at fbpict.c:185
#3  0x000000000054119c in vncHooksComposite (op=1 '\001', pSrc=0x1294100, 
    pMask=0x0, pDst=0x12944e0, xSrc=<optimized out>, ySrc=<optimized out>, 
    xMask=0, yMask=0, xDst=0, yDst=0, width=30, height=26) at vncHooks.cc:635
#4  0x00000000004effe6 in damageComposite (op=1 '\001', pSrc=0x1294100, 
    pMask=0x0, pDst=0x12944e0, xSrc=0, ySrc=0, xMask=0, yMask=0, xDst=0, 
    yDst=0, width=30, height=26) at damage.c:576
#5  0x000000000053c89d in compNewPixmap (pWin=<optimized out>, 
    x=<optimized out>, y=<optimized out>, w=30, h=26) at compalloc.c:522
#6  0x000000000053d56c in compReallocPixmap (pWin=0x1244320, 
    draw_x=<optimized out>, draw_y=<optimized out>, w=30, h=<optimized out>, 
    bw=<optimized out>) at compalloc.c:620
#7  0x000000000053b533 in compResizeWindow (pWin=<optimized out>, x=0, y=0, 
    w=30, h=26, pSib=0x1226a30) at compwindow.c:401
#8  0x00000000005e708a in ConfigureWindow (pWin=<optimized out>, 
    mask=<optimized out>, vlist=<optimized out>, client=<optimized out>)
    at window.c:2483
#9  0x00000000005b901e in ProcConfigureWindow (client=0x1049640)
    at dispatch.c:764
#10 0x00000000005beeac in Dispatch () at dispatch.c:454
#11 0x00000000005d4d5a in main (argc=22, argv=0x7fff8ac54b88, 
    envp=<optimized out>) at main.c:441
Comment 1 Pierre Ossman cendio 2012-10-05 16:05:24 CEST
Initial analysis:

The triggering condition is a window being resized, which in composite mode results in a new backing pixmap being created. In order for this pixmap to have sensible initial data, the code tries to copy the data from the old window over.

The crash happens because of a read from a bad address (0x7f515433feb0).

Digging upwards in the stack, this seems to come from the frame buffer, which is at address 0x7f5154039010. It is 1024x768 and has a stride of 4096 bytes. Calculating the offset gives us a line number of 774, which is outside the valid memory area.

The reason this happens seems to be that the source window is not the root window, but one that is partially off screen. Hence it gets a height that isn't cropped by the backing frame buffer.



Unable to reproduce it locally though, so there is some finer point to this that I'm yet to discover.
Comment 2 Pierre Ossman cendio 2012-10-17 10:37:47 CEST
There was a lot of work done upstream to handle out-of-bounds access in this specific scenario, so I'm just going to assume it is fixed there and an upgrade of xorg will fix it for us.

I tried cherry-picking the most relevant commit, but it results in massive rendering bugs.

Moving this forward until we can upgrade xorg.
Comment 3 Peter Åstrand cendio 2012-11-12 14:07:32 CET
This bug happens on SLED11, about every 2 logins or so.
Comment 4 Peter Åstrand cendio 2012-11-12 14:17:40 CET
Traceback on SLED11:

#0  sse2_composite_src_x888_8888 (imp=<optimized out>, op=<optimized out>, src_image=<optimized out>, mask_image=<optimized out>, 
    dst_image=<optimized out>, src_x=<optimized out>, src_y=0, mask_x=0, mask_y=0, dest_x=0, dest_y=0, width=1, 
    height=<optimized out>) at pixman-sse2.c:2923
#1  0x00000000006690b5 in pixman_image_composite32 (op=<optimized out>, src=0x167bf80, mask=0x0, dest=0x167c0a0, src_x=2, 
    src_y=0, mask_x=0, mask_y=0, dest_x=0, dest_y=0, width=1, height=24) at pixman.c:848
#2  0x00000000004578fd in fbComposite (op=1 '\001', pSrc=0x1669fa0, pMask=0x0, pDst=0x166a040, xSrc=<optimized out>, 
    ySrc=<optimized out>, xMask=0, yMask=0, xDst=0, yDst=0, width=1, height=24) at fbpict.c:185
#3  0x000000000054119c in vncHooksComposite (op=1 '\001', pSrc=0x1669fa0, pMask=0x0, pDst=0x166a040, xSrc=<optimized out>, 
    ySrc=<optimized out>, xMask=0, yMask=0, xDst=0, yDst=0, width=1, height=24) at vncHooks.cc:635
#4  0x00000000004effe6 in damageComposite (op=1 '\001', pSrc=0x1669fa0, pMask=0x0, pDst=0x166a040, xSrc=2, ySrc=0, xMask=0, 
    yMask=0, xDst=0, yDst=0, width=1, height=24) at damage.c:576
#5  0x000000000053c89d in compNewPixmap (pWin=<optimized out>, x=<optimized out>, y=<optimized out>, w=1, h=24) at compalloc.c:522
#6  0x000000000053d56c in compReallocPixmap (pWin=0x16693a0, draw_x=<optimized out>, draw_y=<optimized out>, w=1, 
    h=<optimized out>, bw=<optimized out>) at compalloc.c:620
#7  0x000000000053b533 in compResizeWindow (pWin=<optimized out>, x=2, y=0, w=1, h=24, pSib=0x1615110) at compwindow.c:401
#8  0x00000000005e708a in ConfigureWindow (pWin=<optimized out>, mask=<optimized out>, vlist=<optimized out>, 
    client=<optimized out>) at window.c:2483
#9  0x00000000005b901e in ProcConfigureWindow (client=0x1556580) at dispatch.c:764
#10 0x00000000005beeac in Dispatch () at dispatch.c:454
#11 0x00000000005d4d5a in main (argc=22, argv=0x7fff6f8d6048, envp=<optimized out>) at main.c:441
Comment 5 Peter Åstrand cendio 2012-11-12 14:24:31 CET
Unfortunately it also happens with the 32-bit version of Xvnc:

#0  sse2_composite_src_x888_8888 (imp=0x83ccc38, op=PIXMAN_OP_SRC, src_image=0x8846248, mask_image=0x0, dst_image=0x882cb68, 
    src_x=0, src_y=0, mask_x=0, mask_y=0, dest_x=0, dest_y=0, width=48, height=<optimized out>) at pixman-sse2.c:2923
#1  0x082ad6e2 in pixman_image_composite32 (op=<optimized out>, src=0x8846248, mask=0x0, dest=0x882cb68, src_x=0, src_y=0, 
    mask_x=0, mask_y=0, dest_x=0, dest_y=0, width=48, height=24) at pixman.c:848
#2  0x0808cc19 in fbComposite (op=1 '\001', pSrc=0x87ff0a0, pMask=0x0, pDst=0x882bc18, xSrc=1263, ySrc=1155, xMask=0, yMask=0, 
    xDst=0, yDst=0, width=48, height=24) at fbpict.c:185
#3  0x0817daaa in vncHooksComposite (op=1 '\001', pSrc=0x87ff0a0, pMask=0x0, pDst=0x882bc18, xSrc=0, ySrc=0, xMask=0, yMask=0, 
    xDst=0, yDst=0, width=48, height=24) at vncHooks.cc:635
#4  0x08124c1c in damageComposite (op=1 '\001', pSrc=0x87ff0a0, pMask=0x0, pDst=0x882bc18, xSrc=0, ySrc=0, xMask=0, yMask=0, 
    xDst=0, yDst=0, width=48, height=24) at damage.c:576
#5  0x08179559 in compNewPixmap (pWin=<optimized out>, x=1263, y=1155, w=48, h=24) at compalloc.c:522
#6  0x0817a407 in compReallocPixmap (pWin=0x882e350, draw_x=1263, draw_y=1155, w=48, h=24, bw=0) at compalloc.c:620
#7  0x08177d08 in compResizeWindow (pWin=0x882e350, x=0, y=0, w=48, h=24, pSib=0x8791538) at compwindow.c:401
#8  0x08225b8f in ConfigureWindow (pWin=0x882e350, mask=15, vlist=0x87aca00, client=0x87682e8) at window.c:2483
#9  0x081f9ede in ProcConfigureWindow (client=0x87682e8) at dispatch.c:764
#10 0x081ffa7f in Dispatch () at dispatch.c:454
#11 0x0821751a in main (argc=22, argv=0xffac9494, envp=0xffac94f0) at main.c:441
Comment 6 Peter Åstrand cendio 2012-11-12 16:23:22 CET
Workaround that works on SLED11: disable Composite with:

-extension Composite
Comment 8 Pierre Ossman cendio 2013-04-23 12:46:50 CEST
This needs to be retested with our upgraded Xorg.
Comment 9 Henrik Andersson cendio 2013-06-24 10:03:25 CEST
I have been doing a lot of testing against SLED 11 Sp2 without stumble upon this issue. I have also tried to reproduce it without success so we could probably consider this issue being solved with the big Xorg update in 4.1.0

Note You need to log in before you can comment on or make changes to this bug.