Segmentation fault

[Jack Radigan]

I'm running VMware Workstation 8.0.5 on Xubuntu 12.04 amd64, all patches current.

The guest is using xserver-xorg-video-vmware, 1:12.0.1-1ubuntu1.1

Steps to reproduce the problem:
  1. Create a guest with version 8 hardware compatibility, install Xubuntu 12.04.1 amd64, accept defaults for eveything
  2. Log into guest, run update manager to install all patches
  3. Set the display resolution, in my case my host has a 1920x1200 display and the guest was set to 1680x1050.
  4. Log out of the session.
  5. Switch to a console with <CTRL><ALT><SPACE>, <CTRL><ALT><F1>
  6. After logging in, do the following:
      a. sudo service lightdm stop
      b. rm ${HOME}/.config/xfce4/xfconf/xfce-perchannel-xml/displays.xml
      c. sudo rm /var/log/Xorg.0.*
      d. sudo service lightdm start
   7. Log into guest, chage display resolution again.

At this point the display size expands to the new resolution, but it freezes before finishing the resize, segfaults, and returns to the LightDM login.

Here's a clip from the Xorg.0.log with the last good log entry before the backtrace and segfault are logged:

[ 1720.792] (II) vmware(0): Modeline "800x600"x60.0 42.75 800 850 900 950 600 650 700 750 -hsync +vsync (45.0 kHz)
[ 1720.811]
Backtrace:
[ 1720.811] 0: /usr/bin/X (xorg_backtrace+0x26) [0x7fb328fdf886]
[ 1720.811] 1: /usr/bin/X (0x7fb328e57000+0x18c72a) [0x7fb328fe372a]
[ 1720.811] 2: /lib/x86_64-linux-gnu/libpthread.so.0 (0x7fb32817d000+0xfcb0) [0x7fb32818ccb0]
[ 1720.811] 3: /usr/lib/xorg/modules/drivers/vmware_drv.so (0x7fb32523c000+0xf1d0) [0x7fb32524b1d0]
[ 1720.811] 4: /usr/bin/X (xf86RotateFreeShadow+0x44) [0x7fb328f22694]
[ 1720.811] 5: /usr/lib/xorg/modules/drivers/vmware_drv.so (0x7fb32523c000+0xd90c) [0x7fb32524990c]
[ 1720.811] 6: /usr/bin/X (0x7fb328e57000+0xc9640) [0x7fb328f20640]
[ 1720.811] 7: /usr/bin/X (ProcRRSetScreenSize+0x206) [0x7fb328f56cb6]
[ 1720.812] 8: /usr/bin/X (0x7fb328e57000+0x4e8a1) [0x7fb328ea58a1]
[ 1720.812] 9: /usr/bin/X (0x7fb328e57000+0x3d7ba) [0x7fb328e947ba]
[ 1720.812] 10: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xed) [0x7fb32700e76d]
[ 1720.812] 11: /usr/bin/X (0x7fb328e57000+0x3daad) [0x7fb328e94aad]
[ 1720.812] Segmentation fault at address 0x10
[ 1720.812]
Caught signal 11 (Segmentation fault). Server aborting

Note, this is only with version 8 hardware compatibility, if the guest is created with the older version 6.5-7.x hardware compatibility it works fine, no segfault using the same procedure listed above.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: xserver-xorg-video-vmware 1:12.0.1-1ubuntu1.1
ProcVersionSignature: Ubuntu 3.2.0-37.58-generic 3.2.35
Uname: Linux 3.2.0-37-generic x86_64
ApportVersion: 2.0.1-0ubuntu17.1
Architecture: amd64
Date: Fri Feb 8 12:14:50 2013
InstallationMedia: Xubuntu 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120817.3)
MarkForUpload: True
ProcEnviron:
 LANGUAGE=en_US:en
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: xserver-xorg-video-vmware
UpgradeStatus: No upgrade log present (probably fresh install)