I suffered a similar buffer overflow error on Ubuntu 12.10 with x11vnc 0.9.13 on an x86 32 bit installation upgraded from 12.04 to 12.10.

This error did *not* occur on an AMD 64 bit 12.10 upgrade (also from 12.04).  

I get the error consistently on the x86 32 bit machine when trying to use the -reflect switch, i.e.


sudo x11vnc -forever -shared -noxdamage -reflect 192.168.30.115 -autoport 5915 -nopw
06/11/2012 17:19:31 x11vnc version: 0.9.13 lastmod: 2011-08-10  pid: 26765
06/11/2012 17:19:31 Not opening DISPLAY in -rawfb mode (force via -rawfb +str)
06/11/2012 17:19:31 Continuing without X display in -rawfb mode.
06/11/2012 17:19:31 rfbGetClient(bitsPerSample=8, samplesPerPixel=3, bytesPerPixel=4)
06/11/2012 17:19:31 rawfb: vnc:192.168.30.115
06/11/2012 17:19:31 VNC server supports protocol version 3.8 (viewer 3.8)
06/11/2012 17:19:31 We have 2 security types to read
06/11/2012 17:19:31 0) Received security type 2
06/11/2012 17:19:31 Selecting security type 2 (0/2 in the list)
06/11/2012 17:19:31 1) Received security type 16
06/11/2012 17:19:31 Selected Security Scheme 2
Password: 06/11/2012 17:19:36 VNC authentication succeeded
06/11/2012 17:19:37 Desktop name "d4z8jqg1"
06/11/2012 17:19:37 Connected to VNC server, using protocol version 3.8
06/11/2012 17:19:37 VNC server default format:
06/11/2012 17:19:37   16 bits per pixel.
06/11/2012 17:19:37   Least significant byte first in each pixel.
06/11/2012 17:19:37   TRUE colour: max red 31 green 63 blue 31, shift red 11 green 5 blue 0
06/11/2012 17:19:37 vnc_reflect_resize: 2960x1050x32 first=1
06/11/2012 17:19:37 
06/11/2012 17:19:37 vnc_reflector set rawfb str to: map:/dev/null@2960x1050x32:0xff/0xff00/0xff0000
06/11/2012 17:19:37 raw fb is non-regular file: /dev/null
06/11/2012 17:19:37 rawfb: vnc fb: /dev/null
06/11/2012 17:19:37    w: 2960 h: 1050 b: 32 addr: 0xb61fc008 sz: 12432000
06/11/2012 17:19:37 initialize_screen: fb_depth/fb_bpp/fb_Bpl 24/32/11840
06/11/2012 17:19:37 
06/11/2012 17:19:37 Raw fb at addr 0xb61fc008 is 32bpp depth=24 true color
06/11/2012 17:19:37 
06/11/2012 17:19:37 Listening for VNC connections on TCP port 5915
06/11/2012 17:19:37 rfbListenOnTCP6Port: error in bind IPv6 socket: Address already in use
06/11/2012 17:19:37 Listening also on IPv6 port 5915 (socket 6)
06/11/2012 17:19:37 fb read rate: 222 MB/sec
06/11/2012 17:19:37 fast read: reset -wait  ms to: 10
06/11/2012 17:19:37 fast read: reset -defer ms to: 10
06/11/2012 17:19:37 screen setup finished.
06/11/2012 17:19:37 

The VNC desktop is:      gateway:15
PORT=5915

[Connection from VNC client happens here, fails immediately]

*** buffer overflow detected ***: x11vnc terminated
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(__fortify_fail+0x65)[0xb71c5065]
/lib/i386-linux-gnu/libc.so.6(+0x102e1a)[0xb71c3e1a]
/lib/i386-linux-gnu/libc.so.6(+0x103fda)[0xb71c4fda]
/usr/lib/i386-linux-gnu/libvncserver.so.0(rfbProcessNewConnection+0x123)[0xb76760d3]
/usr/lib/i386-linux-gnu/libvncserver.so.0(rfbCheckFds+0x390)[0xb76765e0]
/usr/lib/i386-linux-gnu/libvncserver.so.0(rfbProcessEvents+0x2e)[0xb766cd6e]
x11vnc[0x80f2a39]
x11vnc[0x80b1cef]
x11vnc[0x8056453]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb70da4d3]
x11vnc[0x80632f5]
======= Memory map: ========
08048000-08194000 r-xp 00000000 08:01 9939       /usr/bin/x11vnc
08194000-08195000 r--p 0014b000 08:01 9939       /usr/bin/x11vnc
08195000-081da000 rw-p 0014c000 08:01 9939       /usr/bin/x11vnc
081da000-083b5000 rw-p 00000000 00:00 0 
093a4000-0961a000 rw-p 00000000 00:00 0          [heap]
b3f31000-b3f4d000 r-xp 00000000 08:01 133990     /lib/i386-linux-gnu/libgcc_s.so.1
b3f4d000-b3f4e000 r--p 0001b000 08:01 133990     /lib/i386-linux-gnu/libgcc_s.so.1
b3f4e000-b3f4f000 rw-p 0001c000 08:01 133990     /lib/i386-linux-gnu/libgcc_s.so.1
b3f6a000-b6e35000 rw-p 00000000 00:00 0 
b6e35000-b6e3c000 r-xp 00000000 08:01 168296     /lib/i386-linux-gnu/librt-2.15.so
b6e3c000-b6e3d000 r--p 00006000 08:01 168296     /lib/i386-linux-gnu/librt-2.15.so
b6e3d000-b6e3e000 rw-p 00007000 08:01 168296     /lib/i386-linux-gnu/librt-2.15.so
b6e3e000-b6e43000 r-xp 00000000 08:01 134596     /usr/lib/i386-linux-gnu/libXdmcp.so.6.0.0
b6e43000-b6e44000 r--p 00004000 08:01 134596     /usr/lib/i386-linux-gnu/libXdmcp.so.6.0.0
b6e44000-b6e45000 rw-p 00005000 08:01 134596     /usr/lib/i386-linux-gnu/libXdmcp.so.6.0.0
b6e45000-b6e47000 r-xp 00000000 08:01 134106     /usr/lib/i386-linux-gnu/libXau.so.6.0.0
b6e47000-b6e48000 r--p 00001000 08:01 134106     /usr/lib/i386-linux-gnu/libXau.so.6.0.0
b6e48000-b6e49000 rw-p 00002000 08:01 134106     /usr/lib/i386-linux-gnu/libXau.so.6.0.0
b6e49000-b6e5b000 r-xp 00000000 08:01 132680     /usr/lib/i386-linux-gnu/libp11-kit.so.0.0.0
b6e5b000-b6e5c000 r--p 00011000 08:01 132680     /usr/lib/i386-linux-gnu/libp11-kit.so.0.0.0
b6e5c000-b6e5d000 rw-p 00012000 08:01 132680     /usr/lib/i386-linux-gnu/libp11-kit.so.0.0.0
b6e5d000-b6e5e000 rw-p 00000000 00:00 0 
b6e5e000-b6e6e000 r-xp 00000000 08:01 132725     /usr/lib/i386-linux-gnu/libtasn1.so.3.1.16
b6e6e000-b6e6f000 r--p 0000f000 08:01 132725     /usr/lib/i386-linux-gnu/libtasn1.so.3.1.16
b6e6f000-b6e70000 rw-p 00010000 08:01 132725     /usr/lib/i386-linux-gnu/libtasn1.so.3.1.16
b6e70000-b6e73000 r-xp 00000000 08:01 138372     /lib/i386-linux-gnu/libgpg-error.so.0.8.0
b6e73000-b6e74000 r--p 00002000 08:01 138372     /lib/i386-linux-gnu/libgpg-error.so.0.8.0
b6e74000-b6e75000 rw-p 00003000 08:01 138372     /lib/i386-linux-gnu/libgpg-error.so.0.8.0
b6e75000-b6ebd000 r-xp 00000000 08:01 134491     /lib/i386-linux-gnu/libdbus-1.so.3.7.2
b6ebd000-b6ebe000 r--p 00047000 08:01 134491     /lib/i386-linux-gnu/libdbus-1.so.3.7.2
b6ebe000-b6ebf000 rw-p 00048000 08:01 134491     /lib/i386-linux-gnu/libdbus-1.so.3.7.2
b6ebf000-b6edf000 r-xp 00000000 08:01 131076     /usr/lib/i386-linux-gnu/libxcb.so.1.1.0
b6edf000-b6ee0000 r--p 0001f000 08:01 131076     /usr/lib/i386-linux-gnu/libxcb.so.1.1.0
b6ee0000-b6ee1000 rw-p 00020000 08:01 131076     /usr/lib/i386-linux-gnu/libxcb.so.1.1.0
b6ee1000-b6ee9000 r-xp 00000000 08:01 132507     /usr/lib/i386-linux-gnu/libXrender.so.1.3.0
b6ee9000-b6eea000 r--p 00007000 08:01 132507     /usr/lib/i386-linux-gnu/libXrender.so.1.3.0
b6eea000-b6eeb000 rw-p 00008000 08:01 132507     /usr/lib/i386-linux-gnu/libXrender.so.1.3.0
b6eeb000-b6eec000 rw-p 00000000 00:00 0 
b6eec000-b6eef000 r-xp 00000000 08:01 168341     /lib/i386-linux-gnu/libdl-2.15.so
b6eef000-b6ef0000 r--p 00002000 08:01 168341     /lib/i386-linux-gnu/libdl-2.15.so
b6ef0000-b6ef1000 rw-p 00003000 08:01 168341     /lib/i386-linux-gnu/libdl-2.15.so
b6ef1000-b6fb0000 r-xp 00000000 08:01 132906     /usr/lib/i386-linux-gnu/libgnutls.so.26.21.8
b6fb0000-b6fb4000 r--p 000be000 08:01 132906     /usr/lib/i386-linux-gnu/libgnutls.so.26.21.8
b6fb4000-b6fb5000 rw-p 000c2000 08:01 132906     /usr/lib/i386-linux-gnu/libgnutls.so.26.21.8
b6fb5000-b7036000 r-xp 00000000 08:01 134379     /lib/i386-linux-gnu/libgcrypt.so.11.7.0
b7036000-b7037000 r--p 00080000 08:01 134379     /lib/i386-linux-gnu/libgcrypt.so.11.7.0
b7037000-b7039000 rw-p 00081000 08:01 134379     /lib/i386-linux-gnu/libgcrypt.so.11.7.0
b7039000-b704c000 r-xp 00000000 08:01 168057     /lib/i386-linux-gnu/libresolv-2.15.so
b704c000-b704d000 r--p 00013000 08:01 168057     /lib/i386-linux-gnu/libresolv-2.15.so
b704d000-b704e000 rw-p 00014000 08:01 168057     /lib/i386-linux-gnu/libresolv-2.15.so
b704e000-b7050000 rw-p 00000000 00:00 0 
b7050000-b7095000 r-xp 00000000 08:01 133068     /usr/lib/i386-linux-gnu/libjpeg.so.8.0.2
b7095000-b7096000 r--p 00044000 08:01 133068     /usr/lib/i386-linux-gnu/libjpeg.so.8.0.2
b7096000-b7097000 rw-p 00045000 08:01 133068     /usr/lib/i386-linux-gnu/libjpeg.so.8.0.2
b7097000-b70a8000 rw-p 00000000 00:00 0 
b70a8000-b70bf000 r-xp 00000000 08:01 137918     /lib/i386-linux-gnu/libz.so.1.2.7
b70bf000-b70c0000 r--p 00016000 08:01 137918     /lib/i386-linux-gnu/libz.so.1.2.7
b70c0000-b70c1000 rw-p 00017000 08:01 137918     /lib/i386-linux-gnu/libz.so.1.2.7
b70c1000-b7264000 r-xp 00000000 08:01 168031     /lib/i386-linux-gnu/libc-2.15.so
b7264000-b7265000 ---p 001a3000 08:01 168031     /lib/i386-linux-gnu/libc-2.15.so
b7265000-b7267000 r--p 001a3000 08:01 168031     /lib/i386-linux-gnu/libc-2.15.so
b7267000-b7268000 rw-p 001a5000 08:01 168031     /lib/i386-linux-gnu/libc-2.15.so
b7268000-b726b000 rw-p 00000000 00:00 0 
b726b000-b727b000 r-xp 00000000 08:01 136916     /usr/lib/i386-linux-gnu/libavahi-client.so.3.2.9
b727b000-b727c000 r--p 0000f000 08:01 136916     /usr/lib/i386-linux-gnu/libavahi-client.so.3.2.9
b727c000-b727d000 rw-p 00010000 08:01 136916     /usr/lib/i386-linux-gnu/libavahi-client.so.3.2.9
b727d000-b7289000 r-xp 00000000 08:01 134002     /usr/lib/i386-linux-gnu/libavahi-common.so.3.5.3
b7289000-b728a000 r--p 0000b000 08:01 134002     /usr/lib/i386-linux-gnu/libavahi-common.so.3.5.3
b728a000-b728b000 rw-p 0000c000 08:01 134002     /usr/lib/i386-linux-gnu/libavahi-common.so.3.5.3
b728b000-b73bd000 r-xp 00000000 08:01 131616     /usr/lib/i386-linux-gnu/libX11.so.6.3.0
b73bd000-b73be000 r--p 00132000 08:01 131616     /usr/lib/i386-linux-gnu/libX11.so.6.3.0
b73be000-b73c1000 rw-p 00133000 08:01 131616     /usr/lib/i386-linux-gnu/libX11.so.6.3.0
b73c1000-b73c2000 rw-p 00000000 00:00 0 
b73c2000-b73c4000 r-xp 00000000 08:01 133409     /usr/lib/i386-linux-gnu/libXdamage.so.1.1.0
b73c4000-b73c5000 r--p 00001000 08:01 133409     /usr/lib/i386-linux-gnu/libXdamage.so.1.1.0
b73c5000-b73c6000 rw-p 00002000 08:01 133409     /usr/lib/i386-linux-gnu/libXdamage.so.1.1.0
b73c6000-b73cb000 r-xp 00000000 08:01 134600     /usr/lib/i386-linux-gnu/libXfixes.so.3.1.0
b73cb000-b73cc000 r--p 00004000 08:01 134600     /usr/lib/i386-linux-gnu/libXfixes.so.3.1.0
b73cc000-b73cd000 rw-p 00005000 08:01 134600     /usr/lib/i386-linux-gnu/libXfixes.so.3.1.0
b73cd000-b73d6000 r-xp 00000000 08:01 132170     /usr/lib/i386-linux-gnu/libXrandr.so.2.2.0
b73d6000-b73d7000 r--p 00008000 08:01 132170     /usr/lib/i386-linux-gnu/libXrandr.so.2.2.0
b73d7000-b73d8000 rw-p 00009000 08:01 132170     /usr/lib/i386-linux-gnu/libXrandr.so.2.2.0
b73d8000-b73da000 r-xp 00000000 08:01 133899     /usr/lib/i386-linux-gnu/libXinerama.so.1.0.0
b73da000-b73db000 r--p 00001000 08:01 133899     /usr/lib/i386-linux-gnu/libXinerama.so.1.0.0
b73db000-b73dc000 rw-p 00002000 08:01 133899     /usr/lib/i386-linux-gnu/libXinerama.so.1.0.0
b73dc000-b73ec000 r-xp 00000000 08:01 142044     /usr/lib/i386-linux-gnu/libXext.so.6.4.0
b73ec000-b73ed000 r--p 0000f000 08:01 142044     /usr/lib/i386-linux-gnu/libXext.so.6.4.0
b73ed000-b73ee000 rw-p 00010000 08:01 142044     /usr/lib/i386-linux-gnu/libXext.so.6.4.0
b73ee000-b73ef000 rw-p 00000000 00:00 0 
b73ef000-b73f4000 r-xp 00000000 08:01 134665     /usr/lib/i386-linux-gnu/libXtst.so.6.1.0
b73f4000-b73f5000 r--p 00004000 08:01 134665     /usr/lib/i386-linux-gnu/libXtst.so.6.1.0
b73f5000-b73f6000 rw-p 00005000 08:01 134665     /usr/lib/i386-linux-gnu/libXtst.so.6.1.0
b73f6000-b73fe000 r-xp 00000000 08:01 168034     /lib/i386-linux-gnu/libcrypt-2.15.so
b73fe000-b73ff000 r--p 00007000 08:01 168034     /lib/i386-linux-gnu/libcrypt-2.15.so
b73ff000-b7400000 rw-p 00008000 08:01 168034     /lib/i386-linux-gnu/libcrypt-2.15.so
b7400000-b7427000 rw-p 00000000 00:00 0 
b7427000-b75b9000 r-xp 00000000 08:01 136050     /lib/i386-linux-gnu/libcrypto.so.1.0.0
b75b9000-b75ba000 ---p 00192000 08:01 136050     /lib/i386-linux-gnu/libcrypto.so.1.0.0
b75ba000-b75c9000 r--p 00192000 08:01 136050     /lib/i386-linux-gnu/libcrypto.so.1.0.0
b75c9000-b75d0000 rw-p 001a1000 08:01 136050     /lib/i386-linux-gnu/libcrypto.so.1.0.0
b75d0000-b75d3000 rw-p 00000000 00:00 0 
b75d3000-b7622000 r-xp 00000000 08:01 137155     /lib/i386-linux-gnu/libssl.so.1.0.0
b7622000-b7623000 ---p 0004f000 08:01 137155     /lib/i386-linux-gnu/libssl.so.1.0.0
b7623000-b7625000 r--p 0004f000 08:01 137155     /lib/i386-linux-gnu/libssl.so.1.0.0
b7625000-b7629000 rw-p 00051000 08:01 137155     /lib/i386-linux-gnu/libssl.so.1.0.0
b7629000-b7640000 r-xp 00000000 08:01 168321     /lib/i386-linux-gnu/libpthread-2.15.so
b7640000-b7641000 r--p 00016000 08:01 168321     /lib/i386-linux-gnu/libpthread-2.15.so
b7641000-b7642000 rw-p 00017000 08:01 168321     /lib/i386-linux-gnu/libpthread-2.15.so
b7642000-b7645000 rw-p 00000000 00:00 0 
b7645000-b7662000 r-xp 00000000 08:01 136824     /usr/lib/i386-linux-gnu/libvncclient.so.0.0.0
b7662000-b7663000 r--p 0001d000 08:01 136824     /usr/lib/i386-linux-gnu/libvncclient.so.0.0.0
b7663000-b7664000 rw-p 0001e000 08:01 136824     /usr/lib/i386-linux-gnu/libvncclient.so.0.0.0
b7664000-b76ac000 r-xp 00000000 08:01 136820     /usr/lib/i386-linux-gnu/libvncserver.so.0.0.0
b76ac000-b76ad000 r--p 00048000 08:01 136820     /usr/lib/i386-linux-gnu/libvncserver.so.0.0.0
b76ad000-b76ae000 rw-p 00049000 08:01 136820     /usr/lib/i386-linux-gnu/libvncserver.so.0.0.0
b76ae000-b76c3000 rw-p 00000000 00:00 0 
b76dd000-b76e0000 rw-p 00000000 00:00 0 
b76e0000-b76e1000 r-xp 00000000 00:00 0          [vdso]
b76e1000-b7701000 r-xp 00000000 08:01 168332     /lib/i386-linux-gnu/ld-2.15.so
b7701000-b7702000 r--p 0001f000 08:01 168332     /lib/i386-linux-gnu/ld-2.15.so
b7702000-b7703000 rw-p 00020000 08:01 168332     /lib/i386-linux-gnu/ld-2.15.so
bfe34000-bfe55000 rw-p 00000000 00:00 0          [stack]
caught signal: 6
06/11/2012 17:19:58 deleted 93 tile_row polling images.