Activity log for bug #1697729

Date Who What changed Old value New value Message
2017-06-13 16:15:49 Aaron Johnson bug added bug
2017-06-13 16:17:06 Alexander J. Maidak bug added subscriber Alexander J. Maidak
2017-06-14 11:04:21 James Page bug task added libvirt (Ubuntu)
2017-06-14 11:09:07 James Page libvirt (Ubuntu): importance Undecided High
2017-06-14 11:09:07 James Page libvirt (Ubuntu): status New Confirmed
2017-06-14 11:09:31 James Page nominated for series cloud-archive/ocata
2017-06-14 11:09:31 James Page bug task added cloud-archive/ocata
2017-06-14 11:09:31 James Page nominated for series cloud-archive/pike
2017-06-14 11:09:31 James Page bug task added cloud-archive/pike
2017-06-14 11:09:36 James Page cloud-archive/ocata: status New Triaged
2017-06-14 11:09:38 James Page cloud-archive/pike: status New Triaged
2017-06-14 11:09:40 James Page cloud-archive/pike: importance Undecided High
2017-06-14 11:09:42 James Page cloud-archive/ocata: importance Undecided High
2017-06-14 11:10:40 James Page nominated for series Ubuntu Artful
2017-06-14 11:10:40 James Page bug task added libvirt (Ubuntu Artful)
2017-06-14 11:10:40 James Page nominated for series Ubuntu Zesty
2017-06-14 11:10:40 James Page bug task added libvirt (Ubuntu Zesty)
2017-06-14 11:10:47 James Page libvirt (Ubuntu Zesty): status New Confirmed
2017-06-14 11:10:49 James Page libvirt (Ubuntu Zesty): importance Undecided High
2017-06-14 11:11:06 James Page libvirt (Ubuntu Zesty): status Confirmed Triaged
2017-06-14 11:11:07 James Page libvirt (Ubuntu Artful): status Confirmed Triaged
2017-06-14 11:23:59 James Page tags 1397440 console horizon spice 1397440 console horizon openstack spice
2017-06-14 13:13:26 Christian Ehrhardt  libvirt (Ubuntu Artful): assignee ChristianEhrhardt (paelzer)
2017-06-14 13:13:29 Christian Ehrhardt  libvirt (Ubuntu Artful): status Triaged In Progress
2017-06-19 06:36:20 Christian Ehrhardt  description Using the UCA ocata release of libvirt we sporatically recieve this error message in nova-compute.log: 2017-06-12 14:32:54.359 19007 ERROR nova.compute.manager [instance: d1af2a13-0a53-4d9c-ada3-683e4973f28a] libvirtError: internal error: process exited while connecting to monitor: ((null):63256): Spice-Warning **: reds.c:2463:reds_init_socket: reds_init_socket: binding socket to 10.141.112.21:5900 failed Please backport the fix for the following bug into UCA ocata/pike releases: https://bugzilla.redhat.com/show_bug.cgi?id=1397440 The patch is documented here: https://www.spinics.net/linux/fedora/libvir/msg144093.html We've tested backporting this same fix using the ocata UCA libvirt 2.5.0-3ubuntu5~cloud0 source package and it fixes the problem for us. [Impact] * VMs start to fail depending on a race around spice port allocation * Solution is the Backport of an upstream fix that avoids a double release on the ports [Test Case] * TBD [Regression Potential] * It is race after all, so we might miss some corner cases in the testing, but reviewing the patch and given the verifications so far it should be safe. From the patch the change is like: Old: Spice-Init -> Fail -> Release [...] QemuStop -> Release ^ If new alloc in this time it was released unintentionally New: Spice-Init -> Fail [...] QemuStop -> Release This eliminates the race, but still releases the port as intended. * This change only affects users of spice ports. [Other Info] * n/a --- Using the UCA ocata release of libvirt we sporatically recieve this error message in nova-compute.log: 2017-06-12 14:32:54.359 19007 ERROR nova.compute.manager [instance: d1af2a13-0a53-4d9c-ada3-683e4973f28a] libvirtError: internal error: process exited while connecting to monitor: ((null):63256): Spice-Warning **: reds.c:2463:reds_init_socket: reds_init_socket: binding socket to 10.141.112.21:5900 failed Please backport the fix for the following bug into UCA ocata/pike releases: https://bugzilla.redhat.com/show_bug.cgi?id=1397440 The patch is documented here: https://www.spinics.net/linux/fedora/libvir/msg144093.html We've tested backporting this same fix using the ocata UCA libvirt 2.5.0-3ubuntu5~cloud0 source package and it fixes the problem for us.
2017-06-19 06:53:39 Christian Ehrhardt  description [Impact] * VMs start to fail depending on a race around spice port allocation * Solution is the Backport of an upstream fix that avoids a double release on the ports [Test Case] * TBD [Regression Potential] * It is race after all, so we might miss some corner cases in the testing, but reviewing the patch and given the verifications so far it should be safe. From the patch the change is like: Old: Spice-Init -> Fail -> Release [...] QemuStop -> Release ^ If new alloc in this time it was released unintentionally New: Spice-Init -> Fail [...] QemuStop -> Release This eliminates the race, but still releases the port as intended. * This change only affects users of spice ports. [Other Info] * n/a --- Using the UCA ocata release of libvirt we sporatically recieve this error message in nova-compute.log: 2017-06-12 14:32:54.359 19007 ERROR nova.compute.manager [instance: d1af2a13-0a53-4d9c-ada3-683e4973f28a] libvirtError: internal error: process exited while connecting to monitor: ((null):63256): Spice-Warning **: reds.c:2463:reds_init_socket: reds_init_socket: binding socket to 10.141.112.21:5900 failed Please backport the fix for the following bug into UCA ocata/pike releases: https://bugzilla.redhat.com/show_bug.cgi?id=1397440 The patch is documented here: https://www.spinics.net/linux/fedora/libvir/msg144093.html We've tested backporting this same fix using the ocata UCA libvirt 2.5.0-3ubuntu5~cloud0 source package and it fixes the problem for us. [Impact]  * VMs start to fail depending on a race around spice port allocation  * Solution is the Backport of an upstream fix that avoids a double    release on the ports [Test Case]  * Prepare a set of VMs using spice and start them concurrently. $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial $ sed 's/vnc/spice/' /usr/share/uvtool/libvirt/template.xml > spice-template.xml $ for idx in {1..20}; do uvt-kvm create --template spice-template.xml --password=ubuntu test-${idx} release=xenial arch=amd64 label=daily; done $ for idx in {1..20}; do virsh shutdown test-${idx}; done # wait until all are gone $ for idx in {1..20}; do (virsh start test-${idx} &); done $ for idx in {1..20}; do virsh domdisplay test-${idx} ; done | sort * expectation - all work, ports are used one by one * current status - failing to intialize: error: internal error: process exited while connecting to monitor: ((null):31733): Spice-Warning **: reds.c:2493:reds_init_socket: reds_init_socket: binding socket to 127.0.0.1:5901 failed [Regression Potential]  * It is race after all, so we might miss some corner cases in the    testing, but reviewing the patch and given the verifications so far it    should be safe. From the patch the change is like:      Old: Spice-Init -> Cleanup -> Release [...] QemuStop -> Release                                            ^                 If new alloc in this time it was released unintentionally      New: Spice-Init -> Fail [...] QemuStop -> Release    This eliminates the race, but still releases the port as intended.  * This change only affects users of spice ports. [Other Info]  * n/a --- Using the UCA ocata release of libvirt we sporatically recieve this error message in nova-compute.log: 2017-06-12 14:32:54.359 19007 ERROR nova.compute.manager [instance: d1af2a13-0a53-4d9c-ada3-683e4973f28a] libvirtError: internal error: process exited while connecting to monitor: ((null):63256): Spice-Warning **: reds.c:2463:reds_init_socket: reds_init_socket: binding socket to 10.141.112.21:5900 failed Please backport the fix for the following bug into UCA ocata/pike releases: https://bugzilla.redhat.com/show_bug.cgi?id=1397440 The patch is documented here: https://www.spinics.net/linux/fedora/libvir/msg144093.html We've tested backporting this same fix using the ocata UCA libvirt 2.5.0-3ubuntu5~cloud0 source package and it fixes the problem for us.
2017-06-19 07:46:45 James Page cloud-archive/pike: status Triaged Fix Committed
2017-06-19 07:46:46 James Page tags 1397440 console horizon openstack spice 1397440 console horizon openstack spice verification-pike-needed
2017-06-19 13:07:52 Launchpad Janitor libvirt (Ubuntu Artful): status In Progress Fix Released
2017-06-20 06:07:40 Christian Ehrhardt  libvirt (Ubuntu Zesty): status Triaged In Progress
2017-06-20 06:07:42 Christian Ehrhardt  libvirt (Ubuntu Zesty): assignee ChristianEhrhardt (paelzer)
2017-06-21 01:16:35 Chris Halse Rogers libvirt (Ubuntu Zesty): status In Progress Fix Committed
2017-06-21 01:16:39 Chris Halse Rogers bug added subscriber Ubuntu Stable Release Updates Team
2017-06-21 01:16:44 Chris Halse Rogers bug added subscriber SRU Verification
2017-06-21 01:16:50 Chris Halse Rogers tags 1397440 console horizon openstack spice verification-pike-needed 1397440 console horizon openstack spice verification-needed verification-pike-needed
2017-06-21 08:59:00 Christian Ehrhardt  tags 1397440 console horizon openstack spice verification-needed verification-pike-needed 1397440 console horizon openstack spice verification-done verification-pike-needed
2017-06-22 05:46:14 Christian Ehrhardt  tags 1397440 console horizon openstack spice verification-done verification-pike-needed 1397440 console horizon openstack spice verification-done verification-pike-done verification-zesty-done
2017-06-22 08:10:50 James Page cloud-archive/ocata: status Triaged Fix Committed
2017-06-22 08:10:52 James Page tags 1397440 console horizon openstack spice verification-done verification-pike-done verification-zesty-done 1397440 console horizon openstack spice verification-done verification-ocata-needed verification-pike-done verification-zesty-done
2017-06-22 08:13:47 James Page cloud-archive/pike: status Fix Committed Fix Released
2017-06-27 14:22:02 Łukasz Zemczak tags 1397440 console horizon openstack spice verification-done verification-ocata-needed verification-pike-done verification-zesty-done 1397440 console horizon openstack spice verification-ocata-needed verification-pike-done verification-zesty-done
2017-07-03 13:08:49 James Page cloud-archive/ocata: status Fix Committed Fix Released
2017-07-03 13:12:54 James Page tags 1397440 console horizon openstack spice verification-ocata-needed verification-pike-done verification-zesty-done 1397440 console horizon openstack spice verification-ocata-done verification-pike-done verification-zesty-done
2017-07-11 14:06:47 Christian Ehrhardt  tags 1397440 console horizon openstack spice verification-ocata-done verification-pike-done verification-zesty-done 1397440 console horizon openstack spice verification-done-zesty verification-ocata-done verification-pike-done
2017-07-11 14:08:53 Launchpad Janitor libvirt (Ubuntu Zesty): status Fix Committed Fix Released