issue7231 is one example of when it can be useful to generate core files on a customer system. In this case, the debug information is needed. There seems to be various ways of accomplishing this, the easiest one being to simply ship non-stripped binaries. These are huge, however. For example, a non-stripped Xvnc is 14 MiB, while a stripped one is 3.3 MiB. Another approach, adopted by Fedora, is to build special debuginfo packages. This is described on https://fedoraproject.org/wiki/Packaging:Debuginfo. I think this looks like a nice approach, and following the Fedora approach is also good. The problem, however, is that our RPM version on Gorgonzola is too old to generate debuginfo RPMs. We also need to verify that debuginfo packages will work on other distributions.
As a side note, for unknown reasons, most of our binaries shipped for Solaris are not stripped: bash-2.03# file * | grep stripped | grep not Xvnc: ELF 32-bit MSB executable SPARC32PLUS Version 1, V8+ Required, dynamically linked, not stripped dialog: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, not stripped openssl: ELF 32-bit MSB executable SPARC32PLUS Version 1, V8+ Required, dynamically linked, not stripped rrdtool: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, not stripped ssh-keyscan: ELF 32-bit MSB executable SPARC32PLUS Version 1, V8+ Required, dynamically linked, not stripped tl-cert-dn: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, not stripped tl-initgroups: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, not stripped xprop: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, not stripped bash-2.03# bash-2.03# file * | grep stripped | grep -v not Xdialog: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, stripped cyclades-ser-cli: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, stripped noctty: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, stripped rsync: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, stripped tl-pam-passwd: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, stripped
It turns out that many binaries are unstripped even on Linux: # file /opt/thinlinc/bin/* /opt/thinlinc/sbin/* /opt/thinlinc/libexec/* | grep -v python | grep -v script | grep "not stripped" /opt/thinlinc/bin/vncconfig: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped /opt/thinlinc/sbin/encrypt-vnc-pw: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped /opt/thinlinc/sbin/lsh-pam-checkpw: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped /opt/thinlinc/libexec/dialog: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped /opt/thinlinc/libexec/openssl: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped /opt/thinlinc/libexec/rrdtool: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped /opt/thinlinc/libexec/ssh-keyscan: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped /opt/thinlinc/libexec/tl-cert-dn: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped /opt/thinlinc/libexec/tl-initgroups: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped /opt/thinlinc/libexec/xprop: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped
rpm needs elfutils to be able to produce debuginfo packages. spent a little time here and there and have now added elfutils to cenbuild. What remains to be able to have debuginfo packages: - Stop stripping stuff - Make a zip of the debuginfo packages produced in a build
More things are needed to get it working. Complete list should be: - Stop stripping. Doing "make install STRIP=/bin/true" should be sufficient in most cases. - Turn on debug symbols. I.e. make sure "-g" is included everywhere. - Turn on debuginfo creation by getting "%{debug_package}" into our spec files. We can put it directly in the files, or modify rpm in cenbuild to always include it. - Make sure rpm uses the correct strip. It uses /usr/bin/strip by default. For Solaris we need to change it. This code generally works: %if %{solaris} %define __strip /usr/bin/${CB_host}-strip %endif - Make a zip of the resulting debug packages Possible extra steps required: - Stop using ".." in paths given to rpm. It seems to confuse the debuginfo generation. - Make sure the binaries have build ids. Our binutils is new enough, but our gcc needs to be patched to give --build-id to ld by default. We might also need to fix this bug: https://bugzilla.redhat.com/show_bug.cgi?id=472152 - Make sure all cenbuild packages also have debuginfo. We link a lot of stuff statically, so we might not get full debug support unless we also have cenbuild packages.
(In reply to comment #4) > More things are needed to get it working. Complete list should be: > > - Stop stripping. > > - Turn on debuginfo creation > > - Make sure rpm uses the correct strip. r25933. > > - Stop using ".." in paths given to rpm. It seems to confuse the debuginfo > generation. > r25931.
(In reply to comment #4) > > - Turn on debug symbols. I.e. make sure "-g" is included everywhere. > r25937.
(In reply to comment #4) > > - Make sure the binaries have build ids. > Bug 4410.
(In reply to comment #4) > - Make a zip of the resulting debug packages > r25942.
(In reply to comment #4) > > - Make sure all cenbuild packages also have debuginfo. Bug 4414.
All done. Tester should make sure that relevant binaries seem to include the proper debug information.
Using latest server build 3680: find /opt/thinlinc/ -executable -type f -exec file {} \; | grep -i elf | grep "not stripped" Does not match any file, so all our binaries are stripped as the should be.
for file in $(find /opt/thinlinc/ -executable -type f -exec file {} \; | grep -i elf | awk -F: '{print($1);}'); do eu-unstrip -n -e $file; done Shows that our binaries are built without buildid , which seems to be the truth due to bug #4410.
After installing all debug packages on my ubuntu test setup build 3680, and running: for file in $(find /opt/thinlinc/ -executable -type f -exec file {} \; | grep -i elf | awk -F: '{print($1);}'); do eu-unstrip -n -e $file; done The following list shows what binaries that have debug info symbols 0x400000+0x201278 - /opt/thinlinc/sbin/lsh-pam-checkpw /usr/lib/debug/opt/thinlinc/sbin/lsh-pam-checkpw.debug 0x400000+0x203a80 - /opt/thinlinc/sbin/encrypt-vnc-pw /usr/lib/debug/opt/thinlinc/sbin/encrypt-vnc-pw.debug 0+0x29c0 - /opt/thinlinc/lib/libhomecreatefilter.so - 0+0x8e18 - /opt/thinlinc/lib/libpcsctun.so - 0+0x3794 - /opt/thinlinc/lib/libcyclades-ser-cli.so - 0x400000+0x23afc8 - /opt/thinlinc/bin/vncconfig /usr/lib/debug/opt/thinlinc/bin/vncconfig.debug 0x400000+0x8d9c48 - /opt/thinlinc/bin/rdesktop /usr/lib/debug/opt/thinlinc/bin/rdesktop.debug 0x400000+0x200dc8 - /opt/thinlinc/bin/tl-vdi-tool /usr/lib/debug/opt/thinlinc/bin/tl-vdi-tool.debug 0x400000+0x2013f8 - /opt/thinlinc/bin/tl-while-x11 /usr/lib/debug/opt/thinlinc/bin/tl-while-x11.debug 0x400000+0x255e90 - /opt/thinlinc/bin/xkbcomp /usr/lib/debug/opt/thinlinc/bin/xkbcomp.debug 0+0x4bdbe0 - /opt/thinlinc/lib64/swrast_dri.so /usr/lib/debug/opt/thinlinc/lib64/swrast_dri.so.debug 0+0x201fd0 - /opt/thinlinc/lib64/libhomecreatefilter.so - 0+0x2096b0 - /opt/thinlinc/lib64/libpcsctun.so /usr/lib/debug/opt/thinlinc/lib64/libpcsctun.so.debug 0+0x202df8 - /opt/thinlinc/lib64/libcyclades-ser-cli.so /usr/lib/debug/opt/thinlinc/lib64/libcyclades-ser-cli.so.debug 0x400000+0x201288 - /opt/thinlinc/libexec/tl-pam-passwd /usr/lib/debug/opt/thinlinc/libexec/tl-pam-passwd.debug 0x400000+0x221e08 - /opt/thinlinc/libexec/tl-crltool /usr/lib/debug/opt/thinlinc/libexec/tl-crltool.debug 0x400000+0x24ef08 - /opt/thinlinc/libexec/rrdtool /usr/lib/debug/opt/thinlinc/libexec/rrdtool.debug 0x400000+0x205c18 - /opt/thinlinc/libexec/xset /usr/lib/debug/opt/thinlinc/libexec/xset.debug 0x400000+0x2034e8 - /opt/thinlinc/libexec/xdpyinfo /usr/lib/debug/opt/thinlinc/libexec/xdpyinfo.debug 0x400000+0x21dff8 - /opt/thinlinc/libexec/cyclades-ser-cli /usr/lib/debug/opt/thinlinc/libexec/cyclades-ser-cli.debug 0x400000+0x613878 - /opt/thinlinc/libexec/Xvnc /usr/lib/debug/opt/thinlinc/libexec/Xvnc.debug 0x400000+0x2023a8 - /opt/thinlinc/libexec/tl-umount-personal /usr/lib/debug/opt/thinlinc/libexec/tl-umount-personal.debug 0x400000+0x200b58 - /opt/thinlinc/libexec/noctty /usr/lib/debug/opt/thinlinc/libexec/noctty.debug 0x400000+0x20efc8 - /opt/thinlinc/libexec/xprop /usr/lib/debug/opt/thinlinc/libexec/xprop.debug 0x400000+0x24cc38 - /opt/thinlinc/libexec/rsync /usr/lib/debug/opt/thinlinc/libexec/rsync.debug 0x400000+0x221ee8 - /opt/thinlinc/libexec/tl-certtool /usr/lib/debug/opt/thinlinc/libexec/tl-certtool.debug 0x400000+0x206450 - /opt/thinlinc/libexec/tl-xinit /usr/lib/debug/opt/thinlinc/libexec/tl-xinit.debug 0x400000+0x203b20 - /opt/thinlinc/libexec/tl-session /usr/lib/debug/opt/thinlinc/libexec/tl-session.debug 0x400000+0x202398 - /opt/thinlinc/libexec/tl-mount-personal /usr/lib/debug/opt/thinlinc/libexec/tl-mount-personal.debug 0x400000+0x3407e8 - /opt/thinlinc/libexec/ssh-keyscan /usr/lib/debug/opt/thinlinc/libexec/ssh-keyscan.debug 0x400000+0x308cc8 - /opt/thinlinc/libexec/tlstunnel /usr/lib/debug/opt/thinlinc/libexec/tlstunnel.debug
When attaching to a running Xvnc i can see that debug symbols are loaded except with this particular issue: Reading symbols from /opt/thinlinc/lib64/swrast_dri.so...Reading symbols from /usr/lib/debug/opt/thinlinc/lib64/swrast_dri.so.debug...(no debugging symbols found)...done. Is this a problem ?
(In reply to comment #14) > When attaching to a running Xvnc i can see that debug symbols are loaded > except with this particular issue: > > Reading symbols from /opt/thinlinc/lib64/swrast_dri.so...Reading symbols from > /usr/lib/debug/opt/thinlinc/lib64/swrast_dri.so.debug...(no debugging symbols > found)...done. > > Is this a problem ? r26015.
Tested latest build and swrast_dri does now have debugs symbols, running the mentioned eu-unstrip mentioned about reveals that the following libraries not have debug symbols: /opt/thinlinc/lib/libhomecreatefilter.so /opt/thinlinc/lib/libpcsctun.so /opt/thinlinc/lib/libcyclades-ser-cli.so /opt/thinlinc/lib64/libhomecreatefilter.so Something we probably should add for completion.
(In reply to comment #16) > Tested latest build and swrast_dri does now have debugs symbols, > running the mentioned eu-unstrip mentioned about reveals that > the following libraries not have debug symbols: > > /opt/thinlinc/lib/libhomecreatefilter.so > /opt/thinlinc/lib/libpcsctun.so > /opt/thinlinc/lib/libcyclades-ser-cli.so > User error. The 32-bit stuff has its debug symbols in the 32-bit package. > /opt/thinlinc/lib64/libhomecreatefilter.so > r26037. > Something we probably should add for completion.
(In reply to comment #17) > (In reply to comment #16) > > Tested latest build and swrast_dri does now have debugs symbols, > > running the mentioned eu-unstrip mentioned about reveals that > > the following libraries not have debug symbols: > > > > /opt/thinlinc/lib/libhomecreatefilter.so > > /opt/thinlinc/lib/libpcsctun.so > > /opt/thinlinc/lib/libcyclades-ser-cli.so > > > > User error. The 32-bit stuff has its debug symbols in the 32-bit package. > Not entirely, if we have packages with mixed arch binaries, the debug packages should reflect that ? :) > > /opt/thinlinc/lib64/libhomecreatefilter.so > > > > r26037. > > > Something we probably should add for completion.
(In reply to comment #18) > (In reply to comment #17) > > (In reply to comment #16) > > > Tested latest build and swrast_dri does now have debugs symbols, > > > running the mentioned eu-unstrip mentioned about reveals that > > > the following libraries not have debug symbols: > > > > > > /opt/thinlinc/lib/libhomecreatefilter.so > > > /opt/thinlinc/lib/libpcsctun.so > > > /opt/thinlinc/lib/libcyclades-ser-cli.so > > > > > > > User error. The 32-bit stuff has its debug symbols in the 32-bit package. > > > > Not entirely, if we have packages with mixed arch binaries, the debug packages > should reflect that ? :) > ubuntu problem, deb packages are name as amd64 even if it contains 32bit binaries.. > > > /opt/thinlinc/lib64/libhomecreatefilter.so > > > > > > > r26037. > > > > > Something we probably should add for completion.
I'm all good with this, closing this bug.
