libspice-server1 >=0.14.0-1 causes Win7 guest bsod in qxldd.sys

Bug #1814146 reported by John Mackoy on 2019-01-31
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
spice (Ubuntu)
Undecided
Unassigned

Bug Description

Summary:
After upgrade from 16.04 to 18.04, when attempting to start a Win7 VM whose state was saved via 'virsh managedsave', the VM will generate a bsod in qxldd.sys.

  Base Config:
    Ubuntu 18.04
    qemu-kvm_1:2.11+dfsg-1ubuntu7.9
    Win7 VM
      qxldd.sys v.6.1.0.10024 (via https://www.spice-space.org/download/windows/spice-guest-tools/spice-guest-tools-0.141/spice-guest-tools-0.141.exe)

  Broken:
    libspice-server1_0.14.0-1ubuntu2.4

  Working (but using older package):
    libspice-server1_0.12.6-4ubuntu0.3

Detail:
From my own research, I learned that there appears to be a problem in libspice-server (0.14.0-1) whereby memory is not managed properly between the spice agent/server and qemu (https://bugzilla.redhat.com/show_bug.cgi?id=1540919#c18 and https://bugzilla.redhat.com/show_bug.cgi?id=1540919#c34). Older versions of libspice-server (0.12.8) were not affected (https://bugzilla.redhat.com/show_bug.cgi?id=1540919#c25).

RedHat appeared to fix this problem (ref: https://bugzilla.redhat.com/show_bug.cgi?id=1567944) by releasing a newer version of the spice server library (https://bugzilla.redhat.com/show_bug.cgi?id=1567944) - 0.14.0-2 - with RHEL 7.5 via RHBA-2018:1378 (https://access.redhat.com/solutions/3461871). Their version of this package is spice-0.14.0-2.el7_5.3, or later.

I was able to work around the problem by reverting to the latest version of libspice-server1 used with Ubuntu 16.04 (libspice-server1_0.12.6-4ubuntu0.3_amd64.deb) via 'sudo apt install -f' and pinning this version of the package in apt via 'sudo apt-mark hold libspice-server1'.

I'm *guessing* that we need a newer version of libspice-server1 compiled from at least the commit/tag that RedHat used when creating binaries for their package to address this.

Related branches

Sebastien Bacher (seb128) wrote :

Looking to the git commit that one reference to the redhat bug
https://github.com/freedesktop/spice/commit/b5479191

Would you be able to rebuild a package with that patch for testing it? (or to test from a ppa otherwise?)

John Mackoy (jtmackoy) wrote :

Bear with me. I'm learning a lot from this experience and think I'm finally getting the idea of building from source on Linux - a new concept to me.

I started down the path of compiling a new version of spice from the latest commit, which informed me that it couldn't find spice-protocol (I believe this is named libspice-server1 in Ubuntu - could be wrong about that). I compiled spice-protocol using the latest commit from here: https://github.com/freedesktop/spice-protocol.

After wrangling some python errors to meet dependency requirements, I successfully built and installed the following:
lrwxrwxrwx root/root 0 2019-04-12 09:52 ./usr/local/lib/libspice-server.so -> libspice-server.so.1.12.5
lrwxrwxrwx root/root 0 2019-04-12 09:52 ./usr/local/lib/libspice-server.so.1 -> libspice-server.so.1.12.5

However, after firing up a VM (using virsh) it doesn't appear that QEMU is using that version:
sudo lsof -p 1746 | grep spice
qemu-syst 1746 root mem REG 8,17 1194088 4203571 /usr/lib/x86_64-linux-gnu/libspice-server.so.1.12.4

Am I missing a step to make QEMU use the new version of spice server I've built?

Also - if I'm waaay off of the reservation here, feel free to point me to a PPA.

Hi John,
Qemu links to libspice like this:

$ ldd /usr/bin/qemu-system-x86_64 | grep spice
        libspice-server.so.1 => /usr/lib/x86_64-linux-gnu/libspice-server.so.1 (0x00007fe884651000)

And that is a symlink usually as installed by the package:
  $ ll /usr/lib/x86_64-linux-gnu/libspice-server.so.1
  lrwxrwxrwx 1 root root 25 Jan 24 15:00 /usr/lib/x86_64-linux-gnu/libspice-server.so.1 -> libspice-server.so.1.12.4

So you could:
- rebuild qemu (useless and complex in this case)
- use a PPA that would keep the bionic version plus that fix
- change the symlink target (much easier)

I have created you an (untested) PPA with that change on top of the version in Bionic at [1].
You can use that for your current testing and if successful we can go on from there.

my recommendation would be to test the PPA (close to a real fix for Ubuntu), if that fails change the symlink to check if soemthing else in the latter version would have fixed it.

Note: the change will be in v0.14.1~93 and later as released from upstream. But versions between packages, protocol and upstream tags are a bit confusing here (between spice vs spice-protocol in particular).

Status: incomplete until testing on PPA was done

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1814146-spice-crash

Changed in spice (Ubuntu):
status: New → Incomplete
John Mackoy (jtmackoy) wrote :

Thank you very much for the explanation of how QEMU references libspice, Christian. One of the reasons my previous response took so long was because I went down the path of compiling QEMU along with libspice, spice-protocl, etc. Your assessment that it was complex was spot-on, although I learned quite a bit from the experience.

Moving on, I've installed the package from the PPA you've referenced and will attempt to repro the situation that seems to generate the BSOD on my Windows 7 guest.

More data points on reproducing the problem: while I will occasionally get BSODs during general usage, by far the best way to trigger is to perform the following:
1. Run VM while host is docked and tied to two external monitors - guest is showing two spice displays (multi-monitor) across each of the external monitors
2. virsh managedsave <guest name> --verbose
3. Shut host down. Undock.
4. Start host (undocked) and resume guest VM
5. virsh start <guest name>; nohup virt-viewer -v --attach Personal_Workstation &
6. Use VM for a meeting or 2 (30-90 mins)
7. virsh managedsave <guest name> --verbose
8. Shut host down
9. Dock host and start host
10. virsh start <guest name>; nohup virt-viewer -v --attach Personal_Workstation &
11. BSOD shortly after VM displays last screen before prior managedsave operation

John Mackoy (jtmackoy) wrote :
Download full text (5.7 KiB)

Unfortunately, after using the system for a couple of days, I received another BSOD in qxldd.sys.

I started to look into linking the binary I built when it looks like it is already using it. Here's what is currently being used:

ldd /usr/bin/qemu-system-x86_64 | grep spice
 libspice-server.so.1 => /usr/local/lib/libspice-server.so.1 (0x00007fac29f23000)

ll /usr/local/lib/libspice-server.so.1
lrwxrwxrwx 1 root root 25 Apr 12 09:52 /usr/local/lib/libspice-server.so.1 -> libspice-server.so.1.12.5

dpkg -S /usr/local/lib/libspice-server.so.1
spice: /usr/local/lib/libspice-server.so.1

sudo apt list spice
Listing... Done
spice/now 0.14.1.143-20190412-1 amd64 [installed,local]

And here is the contents of the package that I built before installing yours from the PPA:

dpkg --contents spice_0.14.1.143-20190412-1_amd64.deb | grep libspice-server.so
-rwxr-xr-x root/root 1202224 2019-04-12 09:52 ./usr/local/lib/libspice-server.so.1.12.5
lrwxrwxrwx root/root 0 2019-04-12 09:52 ./usr/local/lib/libspice-server.so -> libspice-server.so.1.12.5
lrwxrwxrwx root/root 0 2019-04-12 09:52 ./usr/local/lib/libspice-server.so.1 -> libspice-server.so.1.12.5

I've uninstalled the PPA, uninstalled my package, and then reinstalled the PPA.

sudo ppa-purge ppa:paelzer/bug-1814146-spice-crash
Updating packages lists
PPA to be removed: paelzer bug-1814146-spice-crash
Package revert list generated:
 libspice-server1:amd64/bionic

Disabling paelzer PPA from
/etc/apt/sources.list.d/paelzer-ubuntu-bug-1814146-spice-crash-bionic.list
Updating packages lists
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '0.14.0-1ubuntu2.4' (Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security [amd64]) for 'libspice-server1'
The following packages were automatically installed and are no longer required:
  android-libext4-utils android-libselinux android-libsepol libf2fs0
Use 'sudo apt autoremove' to remove them.
The following packages will be DOWNGRADED:
  libspice-server1
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 345 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libspice-server1 amd64 0.14.0-1ubuntu2.4 [345 kB]
Fetched 345 kB in 1s (497 kB/s)
dpkg: warning: downgrading libspice-server1:amd64 from 0.14.0-1ubuntu2.5~ppa1 to 0.14.0-1ubuntu2.4
(Reading database ... 302458 files and directories currently installed.)
Preparing to unpack .../libspice-server1_0.14.0-1ubuntu2.4_amd64.deb ...
Unpacking libspice-server1:amd64 (0.14.0-1ubuntu2.4) over (0.14.0-1ubuntu2.5~ppa1) ...
Setting up libspice-server1:amd64 (0.14.0-1ubuntu2.4) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
PPA purged successfully

sudo apt list spice
Listing... Done
spice/now 0.14.1.143-20190412-1 amd64 [installed,local]

My version is still installed. Removing that now.

sudo apt auto-remove spice
Reading package lists... Done
Building dependency tree
Reading...

Read more...

Andreas Hasenack (ahasenack) wrote :

I'll try to merge a newer spice into Eoan, then we will see if that helps as assumed.

Changed in spice (Ubuntu):
status: Incomplete → Triaged

FYI - there were a few test errors (and there might be more).
For now I have identified a few issue that won't work in the build env and I fixed [1]

Lets try again if 14.2 would now work on all architectures ...

[1]: https://gitlab.freedesktop.org/spice/spice/merge_requests/6

[1] breaks some projects rebuilds.
Most are good I think, but we probably need 0.36/0.37 of [2] spice-gtk.
Unfortunately this is complex as the fix is actually in spice-common path but packaged into spice-gtk.

spice-common/common/generated_client_marshallers.c:303: if (src->type == SPICE_TUNNEL_SERVICE_TYPE_IPP) {
spice-common/common/generated_server_demarshallers.c:1487: if (type__value == SPICE_TUNNEL_SERVICE_TYPE_IPP) {
spice-common/common/generated_server_demarshallers.c:1541: if (out->type == SPICE_TUNNEL_SERVICE_TYPE_IPP) {

This isn't in git at all, it only is in the 0.37 tarball [3] of spice-gtk but with a change to the paths that need changes to the build ...
Arrr, this should be so easy.

[1]: https://gitlab.freedesktop.org/spice/spice-protocol/commit/f7ba19224dc6632f1a5cef5209746adf15dbd6c3
[2]: https://gitlab.freedesktop.org/spice/spice-gtk
[3]: https://gitlab.freedesktop.org/spice/spice-gtk/uploads/b1235a9fde3aef6d461b5ef2d338d854/spice-gtk-0.37.tar.bz2

Ok, I also have a spice-gtk upload ready that seems to work, but I'm not triggering all that on a friday :-)

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package spice - 0.14.2-0ubuntu2

---------------
spice (0.14.2-0ubuntu2) eoan; urgency=medium

  * Fixup autpkgtest (LP: #1834286)
    These changes will make the test able to run again, but not output mismatch
    errors (this matches the behavior before 0.14.2). Upstream discussion
    started on how to resolve that as a next step, more details at the LP bug.
    - d/t/automated-tests: spice-common moved into dir subprojects
    - d/t/automated-tests: option --enable-automated-tests now is always on"
    - d/t/automated-tests, d/t/control: make tests more debuggable by allowing
      stderr
    - d/t/control: install new test dependency python-pil
    - d/t/base_test.ppm, d/t/regression-test.py: provide test resources from
      upstream git not part of the released tarball anymore
    - d/source/include-binaries: allow binary base_test.ppm in package

 -- Christian Ehrhardt <email address hidden> Tue, 25 Jun 2019 12:59:01 +0200

Changed in spice (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers