| 2019-05-21 07:45:11 |
Matthew Ruffell |
bug |
|
|
added bug |
| 2019-05-21 07:54:03 |
Matthew Ruffell |
bug task added |
|
libvirt (Ubuntu) |
|
| 2019-05-21 07:54:27 |
Matthew Ruffell |
nominated for series |
|
Ubuntu Xenial |
|
| 2019-05-21 07:54:27 |
Matthew Ruffell |
bug task added |
|
libvirt (Ubuntu Xenial) |
|
| 2019-05-21 07:55:16 |
Matthew Ruffell |
libvirt (Ubuntu): assignee |
|
Matthew Ruffell (mruffell) |
|
| 2019-05-21 07:55:19 |
Matthew Ruffell |
libvirt (Ubuntu Xenial): assignee |
|
Matthew Ruffell (mruffell) |
|
| 2019-05-21 07:55:22 |
Matthew Ruffell |
cloud-archive: assignee |
|
Matthew Ruffell (mruffell) |
|
| 2019-05-21 08:00:49 |
Matthew Ruffell |
description |
[Impact]
* libvirt-bin, in: libvirt-1.3.1-1ubuntu10.24~cloud0 in trusty mitaka uca
and the parent package in xenial, libvirt-1.3.1-1ubuntu10.24 are effected.
* When you shutdown a system in trusty which is running some kvm virtual machines,
the libvirt-bin service is stopped before libvirt-guests. libvirt-guests tries
to connect to the libvirt socket to send shutdown commands to the running vms,
which cannot happen since libvirtd is not running.
* On some machines, the qemu processes behind the virtual machines are not killed
and are left behind as defunct processes, which can cause the system to hang
on them not being terminated.
* The bug is caused by the libvirt-bin upstart script [1] calling a non-existant
script, /usr/lib/libvirt/libvirt-stop-guests [2]. This script used to exist
in the upstart script itself in version 1.2.2-0ubuntu13.1.27 [3], the version
in the trusty archives. In liberty UCA, version 1.2.16-2ubuntu11.15.10.4~cloud0
[4], the script was seperated out into /usr/lib/libvirt/libvirt-stop-guests [2].
In the mitaka release, the libvirt-stop-guests script was removed and rewritten
as /etc/init.d/libvirt-guests [5], but the script in [1] was never updated to
point to it.
[1] http://paste.ubuntu.com/p/GxxBczkCmk
[2] http://paste.ubuntu.com/p/fKCDQh46vh
[3] http://paste.ubuntu.com/p/QrKXqK2Bvz
[4] http://paste.ubuntu.com/p/W8DgQwpYv3
[5] http://paste.ubuntu.com/p/Z28Sp2fPd6
* Since the upstart script was never updated to point to it, libvirt-bin stops
without stopping libvirt-guests first. When libvirt-guests is stopped later,
it cannot access the libvirt socket, cannot shut down the machines, causing
the bug.
* The fix is to change the upstart script to point to the new libvirt-guests
script.
[Test Case]
* You can reproduce this in trusty with the mitaka UCA enabled.
1) Enable mitaka UCA and install libvirt0 and libvirt-bin
$ sudo add-apt-repository cloud-archive:mitaka
$ sudo apt update
$ sudo apt install libvirt0 libvirt-bin
2) Install a virtual machine, either by using virt-install or virt-manager.
I used a bionic VM.
3) Enable debugging on libvirt-guests so you can see what is going on
Modify /etc/init.d/libvirt-guests and add "-x" to the end of "!/bin/sh"
4) With the vm running, shut down the system
$ sudo shutdown -h now
5) Check /var/log/upstart/libvirt-bin.log, on reboot. It will say
"No such file or directory: /usr/lib/libvirt/libvirt-stop-guests"
6) During that shutdown, you will see messages like:
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
What should happen:
If you follow the same steps with the fixed package, when you look at
/var/log/upstart/libvirt-bin.log, you will see output of libvirt-guests
connecting to and shutting down the virtual machines which lookes a little like
this: https://paste.ubuntu.com/p/s4jyJX2y9F/
[Regression Potential]
* There is only one file modified, the upstart script for libvirt-bin.
Currently this upstart file references a file which doesn't exist, so fixing
it will restore the behaviour in a way which aligns with excactly what took
place in previous versions.
* In xenial, there is no concern of stopping an already stopped service in the
event that the upstart script pre-stop section is called by systemd.
* This change only effects systems during shutdown while they still
have virtual machines running, and do not effect starting and stopping services
while the machine is running normally.
* I believe the regression potential is low.
[Other Info]
* Xenial is not effected by this bug even though it ships the exact same packages.
This is because xenial uses insserv to generate service dependency files
".depend.boot" ".depend.start" ".depend.stop" which parse the scripts in
/etc/init.d/ and systemd respects the dependency ordering in these files.
libvirt-guests reports a dependency on libvirt-bin in the script header,
so systemd will always stop libvirt-guests before libvirt-bin, avoiding the
problem seen in trusty.
* The fix is needed in trusty mitaka UCA and xenial will likely need the SRU
as part of the process. |
subject: libvirt-bin: during shutdown libvirt-bin is stopped before libvirt-guests causing hang
[Impact]
* libvirt-bin, in: libvirt-1.3.1-1ubuntu10.24~cloud0 in trusty mitaka uca
and the parent package in xenial, libvirt-1.3.1-1ubuntu10.24 are effected.
* When you shutdown a system in trusty which is running some kvm virtual
machines, the libvirt-bin service is stopped before libvirt-guests.
libvirt-guests tries to connect to the libvirt socket to send shutdown
commands to the running vms, which cannot happen since libvirtd is not
running.
* On some machines, the qemu processes behind the virtual machines are not
killed and are left behind as defunct processes, which can cause the
system to hang on them not being terminated.
* The bug is caused by the libvirt-bin upstart script [1] calling a
non-existant script, /usr/lib/libvirt/libvirt-stop-guests [2]. This script
used to exist in the upstart script itself in version 1.2.2-0ubuntu13.1.27
[3], the version in the trusty archives. In liberty UCA,
version 1.2.16-2ubuntu11.15.10.4~cloud0 [4], the script was seperated out
into /usr/lib/libvirt/libvirt-stop-guests [2]. In the mitaka release, the
libvirt-stop-guests script was removed and rewritten as
/etc/init.d/libvirt-guests [5], but the script in [1] was never updated to
point to it.
[1] http://paste.ubuntu.com/p/GxxBczkCmk
[2] http://paste.ubuntu.com/p/fKCDQh46vh
[3] http://paste.ubuntu.com/p/QrKXqK2Bvz
[4] http://paste.ubuntu.com/p/W8DgQwpYv3
[5] http://paste.ubuntu.com/p/Z28Sp2fPd6
* Since the upstart script was never updated to point to it, libvirt-bin
stops without stopping libvirt-guests first. When libvirt-guests is
stopped later, it cannot access the libvirt socket, cannot shut down the
machines, causing the bug.
* The fix is to change the upstart script to point to the new libvirt-guests
script.
[Test Case]
* You can reproduce this in trusty with the mitaka UCA enabled.
1) Enable mitaka UCA and install libvirt0 and libvirt-bin
$ sudo add-apt-repository cloud-archive:mitaka
$ sudo apt update
$ sudo apt install libvirt0 libvirt-bin
2) Install a virtual machine, either by using virt-install or virt-manager.
I used a bionic VM.
3) Enable debugging on libvirt-guests so you can see what is going on
Modify /etc/init.d/libvirt-guests and add "-x" to the end of "!/bin/sh"
4) With the vm running, shut down the system
$ sudo shutdown -h now
5) Check /var/log/upstart/libvirt-bin.log, on reboot. It will say
"No such file or directory: /usr/lib/libvirt/libvirt-stop-guests"
6) During that shutdown, you will see messages like:
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
No such file or directory
What should happen:
If you follow the same steps with the fixed package, when you look at
/var/log/upstart/libvirt-bin.log, you will see output of libvirt-guests
connecting to and shutting down the virtual machines which looks a little
like this: https://paste.ubuntu.com/p/s4jyJX2y9F/
[Regression Potential]
* There is only one file modified, the upstart script for libvirt-bin.
Currently this upstart file references a file which doesn't exist, so
fixing it will restore the behaviour in a way which aligns with exactly
what took place in previous versions.
* In xenial, there is no concern of stopping an already stopped service in
the event that the upstart script pre-stop section is called by systemd.
* This change only effects systems during shutdown while they still
have virtual machines running, and do not effect starting and stopping
services while the machine is running normally.
* I believe the regression potential is low.
[Other Info]
* Xenial is not effected by this bug even though it ships the exact same
packages. This is because xenial uses insserv to generate service
dependency files ".depend.boot" ".depend.start" ".depend.stop" which parse
the scripts in /etc/init.d/ and systemd respects the dependency ordering
in these files.
libvirt-guests reports a dependency on libvirt-bin in the script header,
so systemd will always stop libvirt-guests before libvirt-bin, avoiding
the problem seen in trusty.
* The fix is needed in trusty mitaka UCA and xenial will likely need the SRU
as part of the process. |
|
| 2019-05-21 08:00:54 |
Matthew Ruffell |
libvirt (Ubuntu): importance |
Undecided |
Medium |
|
| 2019-05-21 08:00:57 |
Matthew Ruffell |
libvirt (Ubuntu Xenial): importance |
Undecided |
Medium |
|
| 2019-05-21 08:01:04 |
Matthew Ruffell |
libvirt (Ubuntu): status |
New |
In Progress |
|
| 2019-05-21 08:01:07 |
Matthew Ruffell |
libvirt (Ubuntu Xenial): status |
New |
In Progress |
|
| 2019-05-21 08:01:10 |
Matthew Ruffell |
cloud-archive: status |
New |
In Progress |
|
| 2019-05-21 08:01:33 |
Matthew Ruffell |
description |
subject: libvirt-bin: during shutdown libvirt-bin is stopped before libvirt-guests causing hang
[Impact]
* libvirt-bin, in: libvirt-1.3.1-1ubuntu10.24~cloud0 in trusty mitaka uca
and the parent package in xenial, libvirt-1.3.1-1ubuntu10.24 are effected.
* When you shutdown a system in trusty which is running some kvm virtual
machines, the libvirt-bin service is stopped before libvirt-guests.
libvirt-guests tries to connect to the libvirt socket to send shutdown
commands to the running vms, which cannot happen since libvirtd is not
running.
* On some machines, the qemu processes behind the virtual machines are not
killed and are left behind as defunct processes, which can cause the
system to hang on them not being terminated.
* The bug is caused by the libvirt-bin upstart script [1] calling a
non-existant script, /usr/lib/libvirt/libvirt-stop-guests [2]. This script
used to exist in the upstart script itself in version 1.2.2-0ubuntu13.1.27
[3], the version in the trusty archives. In liberty UCA,
version 1.2.16-2ubuntu11.15.10.4~cloud0 [4], the script was seperated out
into /usr/lib/libvirt/libvirt-stop-guests [2]. In the mitaka release, the
libvirt-stop-guests script was removed and rewritten as
/etc/init.d/libvirt-guests [5], but the script in [1] was never updated to
point to it.
[1] http://paste.ubuntu.com/p/GxxBczkCmk
[2] http://paste.ubuntu.com/p/fKCDQh46vh
[3] http://paste.ubuntu.com/p/QrKXqK2Bvz
[4] http://paste.ubuntu.com/p/W8DgQwpYv3
[5] http://paste.ubuntu.com/p/Z28Sp2fPd6
* Since the upstart script was never updated to point to it, libvirt-bin
stops without stopping libvirt-guests first. When libvirt-guests is
stopped later, it cannot access the libvirt socket, cannot shut down the
machines, causing the bug.
* The fix is to change the upstart script to point to the new libvirt-guests
script.
[Test Case]
* You can reproduce this in trusty with the mitaka UCA enabled.
1) Enable mitaka UCA and install libvirt0 and libvirt-bin
$ sudo add-apt-repository cloud-archive:mitaka
$ sudo apt update
$ sudo apt install libvirt0 libvirt-bin
2) Install a virtual machine, either by using virt-install or virt-manager.
I used a bionic VM.
3) Enable debugging on libvirt-guests so you can see what is going on
Modify /etc/init.d/libvirt-guests and add "-x" to the end of "!/bin/sh"
4) With the vm running, shut down the system
$ sudo shutdown -h now
5) Check /var/log/upstart/libvirt-bin.log, on reboot. It will say
"No such file or directory: /usr/lib/libvirt/libvirt-stop-guests"
6) During that shutdown, you will see messages like:
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
No such file or directory
What should happen:
If you follow the same steps with the fixed package, when you look at
/var/log/upstart/libvirt-bin.log, you will see output of libvirt-guests
connecting to and shutting down the virtual machines which looks a little
like this: https://paste.ubuntu.com/p/s4jyJX2y9F/
[Regression Potential]
* There is only one file modified, the upstart script for libvirt-bin.
Currently this upstart file references a file which doesn't exist, so
fixing it will restore the behaviour in a way which aligns with exactly
what took place in previous versions.
* In xenial, there is no concern of stopping an already stopped service in
the event that the upstart script pre-stop section is called by systemd.
* This change only effects systems during shutdown while they still
have virtual machines running, and do not effect starting and stopping
services while the machine is running normally.
* I believe the regression potential is low.
[Other Info]
* Xenial is not effected by this bug even though it ships the exact same
packages. This is because xenial uses insserv to generate service
dependency files ".depend.boot" ".depend.start" ".depend.stop" which parse
the scripts in /etc/init.d/ and systemd respects the dependency ordering
in these files.
libvirt-guests reports a dependency on libvirt-bin in the script header,
so systemd will always stop libvirt-guests before libvirt-bin, avoiding
the problem seen in trusty.
* The fix is needed in trusty mitaka UCA and xenial will likely need the SRU
as part of the process. |
[Impact]
* libvirt-bin, in: libvirt-1.3.1-1ubuntu10.24~cloud0 in trusty mitaka uca
and the parent package in xenial, libvirt-1.3.1-1ubuntu10.24 are effected.
* When you shutdown a system in trusty which is running some kvm virtual
machines, the libvirt-bin service is stopped before libvirt-guests.
libvirt-guests tries to connect to the libvirt socket to send shutdown
commands to the running vms, which cannot happen since libvirtd is not
running.
* On some machines, the qemu processes behind the virtual machines are not
killed and are left behind as defunct processes, which can cause the
system to hang on them not being terminated.
* The bug is caused by the libvirt-bin upstart script [1] calling a
non-existant script, /usr/lib/libvirt/libvirt-stop-guests [2]. This script
used to exist in the upstart script itself in version 1.2.2-0ubuntu13.1.27
[3], the version in the trusty archives. In liberty UCA,
version 1.2.16-2ubuntu11.15.10.4~cloud0 [4], the script was seperated out
into /usr/lib/libvirt/libvirt-stop-guests [2]. In the mitaka release, the
libvirt-stop-guests script was removed and rewritten as
/etc/init.d/libvirt-guests [5], but the script in [1] was never updated to
point to it.
[1] http://paste.ubuntu.com/p/GxxBczkCmk
[2] http://paste.ubuntu.com/p/fKCDQh46vh
[3] http://paste.ubuntu.com/p/QrKXqK2Bvz
[4] http://paste.ubuntu.com/p/W8DgQwpYv3
[5] http://paste.ubuntu.com/p/Z28Sp2fPd6
* Since the upstart script was never updated to point to it, libvirt-bin
stops without stopping libvirt-guests first. When libvirt-guests is
stopped later, it cannot access the libvirt socket, cannot shut down the
machines, causing the bug.
* The fix is to change the upstart script to point to the new libvirt-guests
script.
[Test Case]
* You can reproduce this in trusty with the mitaka UCA enabled.
1) Enable mitaka UCA and install libvirt0 and libvirt-bin
$ sudo add-apt-repository cloud-archive:mitaka
$ sudo apt update
$ sudo apt install libvirt0 libvirt-bin
2) Install a virtual machine, either by using virt-install or virt-manager.
I used a bionic VM.
3) Enable debugging on libvirt-guests so you can see what is going on
Modify /etc/init.d/libvirt-guests and add "-x" to the end of "!/bin/sh"
4) With the vm running, shut down the system
$ sudo shutdown -h now
5) Check /var/log/upstart/libvirt-bin.log, on reboot. It will say
"No such file or directory: /usr/lib/libvirt/libvirt-stop-guests"
6) During that shutdown, you will see messages like:
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
No such file or directory
What should happen:
If you follow the same steps with the fixed package, when you look at
/var/log/upstart/libvirt-bin.log, you will see output of libvirt-guests
connecting to and shutting down the virtual machines which looks a little
like this: https://paste.ubuntu.com/p/s4jyJX2y9F/
[Regression Potential]
* There is only one file modified, the upstart script for libvirt-bin.
Currently this upstart file references a file which doesn't exist, so
fixing it will restore the behaviour in a way which aligns with exactly
what took place in previous versions.
* In xenial, there is no concern of stopping an already stopped service in
the event that the upstart script pre-stop section is called by systemd.
* This change only effects systems during shutdown while they still
have virtual machines running, and do not effect starting and stopping
services while the machine is running normally.
* I believe the regression potential is low.
[Other Info]
* Xenial is not effected by this bug even though it ships the exact same
packages. This is because xenial uses insserv to generate service
dependency files ".depend.boot" ".depend.start" ".depend.stop" which parse
the scripts in /etc/init.d/ and systemd respects the dependency ordering
in these files.
libvirt-guests reports a dependency on libvirt-bin in the script header,
so systemd will always stop libvirt-guests before libvirt-bin, avoiding
the problem seen in trusty.
* The fix is needed in trusty mitaka UCA and xenial will likely need the SRU
as part of the process. |
|
| 2019-05-21 08:02:23 |
Matthew Ruffell |
description |
[Impact]
* libvirt-bin, in: libvirt-1.3.1-1ubuntu10.24~cloud0 in trusty mitaka uca
and the parent package in xenial, libvirt-1.3.1-1ubuntu10.24 are effected.
* When you shutdown a system in trusty which is running some kvm virtual
machines, the libvirt-bin service is stopped before libvirt-guests.
libvirt-guests tries to connect to the libvirt socket to send shutdown
commands to the running vms, which cannot happen since libvirtd is not
running.
* On some machines, the qemu processes behind the virtual machines are not
killed and are left behind as defunct processes, which can cause the
system to hang on them not being terminated.
* The bug is caused by the libvirt-bin upstart script [1] calling a
non-existant script, /usr/lib/libvirt/libvirt-stop-guests [2]. This script
used to exist in the upstart script itself in version 1.2.2-0ubuntu13.1.27
[3], the version in the trusty archives. In liberty UCA,
version 1.2.16-2ubuntu11.15.10.4~cloud0 [4], the script was seperated out
into /usr/lib/libvirt/libvirt-stop-guests [2]. In the mitaka release, the
libvirt-stop-guests script was removed and rewritten as
/etc/init.d/libvirt-guests [5], but the script in [1] was never updated to
point to it.
[1] http://paste.ubuntu.com/p/GxxBczkCmk
[2] http://paste.ubuntu.com/p/fKCDQh46vh
[3] http://paste.ubuntu.com/p/QrKXqK2Bvz
[4] http://paste.ubuntu.com/p/W8DgQwpYv3
[5] http://paste.ubuntu.com/p/Z28Sp2fPd6
* Since the upstart script was never updated to point to it, libvirt-bin
stops without stopping libvirt-guests first. When libvirt-guests is
stopped later, it cannot access the libvirt socket, cannot shut down the
machines, causing the bug.
* The fix is to change the upstart script to point to the new libvirt-guests
script.
[Test Case]
* You can reproduce this in trusty with the mitaka UCA enabled.
1) Enable mitaka UCA and install libvirt0 and libvirt-bin
$ sudo add-apt-repository cloud-archive:mitaka
$ sudo apt update
$ sudo apt install libvirt0 libvirt-bin
2) Install a virtual machine, either by using virt-install or virt-manager.
I used a bionic VM.
3) Enable debugging on libvirt-guests so you can see what is going on
Modify /etc/init.d/libvirt-guests and add "-x" to the end of "!/bin/sh"
4) With the vm running, shut down the system
$ sudo shutdown -h now
5) Check /var/log/upstart/libvirt-bin.log, on reboot. It will say
"No such file or directory: /usr/lib/libvirt/libvirt-stop-guests"
6) During that shutdown, you will see messages like:
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
No such file or directory
What should happen:
If you follow the same steps with the fixed package, when you look at
/var/log/upstart/libvirt-bin.log, you will see output of libvirt-guests
connecting to and shutting down the virtual machines which looks a little
like this: https://paste.ubuntu.com/p/s4jyJX2y9F/
[Regression Potential]
* There is only one file modified, the upstart script for libvirt-bin.
Currently this upstart file references a file which doesn't exist, so
fixing it will restore the behaviour in a way which aligns with exactly
what took place in previous versions.
* In xenial, there is no concern of stopping an already stopped service in
the event that the upstart script pre-stop section is called by systemd.
* This change only effects systems during shutdown while they still
have virtual machines running, and do not effect starting and stopping
services while the machine is running normally.
* I believe the regression potential is low.
[Other Info]
* Xenial is not effected by this bug even though it ships the exact same
packages. This is because xenial uses insserv to generate service
dependency files ".depend.boot" ".depend.start" ".depend.stop" which parse
the scripts in /etc/init.d/ and systemd respects the dependency ordering
in these files.
libvirt-guests reports a dependency on libvirt-bin in the script header,
so systemd will always stop libvirt-guests before libvirt-bin, avoiding
the problem seen in trusty.
* The fix is needed in trusty mitaka UCA and xenial will likely need the SRU
as part of the process. |
[Impact]
* libvirt-bin, in: libvirt-1.3.1-1ubuntu10.24~cloud0 in trusty mitaka uca
and the parent package in xenial, libvirt-1.3.1-1ubuntu10.24 are effected.
* When you shutdown a system in trusty which is running some kvm virtual
machines, the libvirt-bin service is stopped before libvirt-guests.
libvirt-guests tries to connect to the libvirt socket to send shutdown
commands to the running vms, which cannot happen since libvirtd is not
running.
* On some machines, the qemu processes behind the virtual machines are not
killed and are left behind as defunct processes, which can cause the
system to hang on them not being terminated.
* The bug is caused by the libvirt-bin upstart script [1] calling a
non-existant script, /usr/lib/libvirt/libvirt-stop-guests [2]. This script
used to exist in the upstart script itself in version 1.2.2-0ubuntu13.1.27
[3], the version in the trusty archives. In liberty UCA,
version 1.2.16-2ubuntu11.15.10.4~cloud0 [4], the script was seperated out
into /usr/lib/libvirt/libvirt-stop-guests [2]. In the mitaka release, the
libvirt-stop-guests script was removed and rewritten as
/etc/init.d/libvirt-guests [5], but the script in [1] was never updated to
point to it.
[1] http://paste.ubuntu.com/p/GxxBczkCmk
[2] http://paste.ubuntu.com/p/fKCDQh46vh
[3] http://paste.ubuntu.com/p/QrKXqK2Bvz
[4] http://paste.ubuntu.com/p/W8DgQwpYv3
[5] http://paste.ubuntu.com/p/Z28Sp2fPd6
* Since the upstart script was never updated to point to it, libvirt-bin
stops without stopping libvirt-guests first. When libvirt-guests is
stopped later, it cannot access the libvirt socket, cannot shut down the
machines, causing the bug.
* The fix is to change the upstart script to point to the new libvirt-guests
script.
[Test Case]
* You can reproduce this in trusty with the mitaka UCA enabled.
1) Enable mitaka UCA and install libvirt0 and libvirt-bin
$ sudo add-apt-repository cloud-archive:mitaka
$ sudo apt update
$ sudo apt install libvirt0 libvirt-bin
2) Install a virtual machine, either by using virt-install or virt-manager.
I used a bionic VM.
3) Enable debugging on libvirt-guests so you can see what is going on
Modify /etc/init.d/libvirt-guests and add "-x" to the end of "!/bin/sh"
4) With the vm running, shut down the system
$ sudo shutdown -h now
5) Check /var/log/upstart/libvirt-bin.log, on reboot. It will say
"No such file or directory: /usr/lib/libvirt/libvirt-stop-guests"
6) During that shutdown, you will see messages like:
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
No such file or directory
What should happen:
If you follow the same steps with the fixed package, when you look at
/var/log/upstart/libvirt-bin.log, you will see output of libvirt-guests
connecting to and shutting down the virtual machines which looks a little
like this: https://paste.ubuntu.com/p/s4jyJX2y9F/
[Regression Potential]
* There is only one file modified, the upstart script for libvirt-bin.
Currently this upstart file references a file which doesn't exist, so
fixing it will restore the behaviour in a way which aligns with exactly
what took place in previous versions.
* In xenial, there is no concern of stopping an already stopped service in
the event that the upstart script pre-stop section is called by systemd.
* This change only effects systems during shutdown while they still
have virtual machines running, and do not effect starting and stopping
services while the machine is running normally.
* I believe the regression potential is low.
[Other Info]
* Xenial is not effected by this bug even though it ships the exact same
packages. This is because xenial uses insserv to generate service
dependency files ".depend.boot" ".depend.start" ".depend.stop" which parse
the scripts in /etc/init.d/ and systemd respects the dependency ordering
in these files.
libvirt-guests reports a dependency on libvirt-bin in the script header,
so systemd will always stop libvirt-guests before libvirt-bin, avoiding
the problem seen in trusty.
* The fix is needed in trusty mitaka UCA and xenial will likely need the SRU
as part of the process. |
|
| 2019-05-21 08:14:06 |
Christian Ehrhardt |
libvirt (Ubuntu): status |
In Progress |
Invalid |
|
| 2019-05-21 08:14:08 |
Christian Ehrhardt |
libvirt (Ubuntu Xenial): status |
In Progress |
Invalid |
|
| 2019-05-21 08:14:12 |
Christian Ehrhardt |
libvirt (Ubuntu): assignee |
Matthew Ruffell (mruffell) |
|
|
| 2019-05-21 08:14:14 |
Christian Ehrhardt |
libvirt (Ubuntu Xenial): assignee |
Matthew Ruffell (mruffell) |
|
|
| 2019-05-21 08:27:23 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2019-05-21 21:32:11 |
Dan Streetman |
bug |
|
|
added subscriber Dan Streetman |
| 2019-05-22 04:54:58 |
Matthew Ruffell |
attachment added |
|
libvirt debdiff for trusty-mitaka https://bugs.launchpad.net/cloud-archive/+bug/1829823/+attachment/5265459/+files/lp1829823_trusty-mitaka.debdiff |
|
| 2019-05-27 03:31:28 |
Matthew Ruffell |
attachment added |
|
libvirt debdiff for xenial https://bugs.launchpad.net/cloud-archive/+bug/1829823/+attachment/5266848/+files/lp1829823_xenial.debdiff |
|
| 2019-05-28 05:51:46 |
Christian Ehrhardt |
libvirt (Ubuntu Xenial): status |
Invalid |
Fix Released |
|
| 2019-05-28 05:57:18 |
Christian Ehrhardt |
description |
[Impact]
* libvirt-bin, in: libvirt-1.3.1-1ubuntu10.24~cloud0 in trusty mitaka uca
and the parent package in xenial, libvirt-1.3.1-1ubuntu10.24 are effected.
* When you shutdown a system in trusty which is running some kvm virtual
machines, the libvirt-bin service is stopped before libvirt-guests.
libvirt-guests tries to connect to the libvirt socket to send shutdown
commands to the running vms, which cannot happen since libvirtd is not
running.
* On some machines, the qemu processes behind the virtual machines are not
killed and are left behind as defunct processes, which can cause the
system to hang on them not being terminated.
* The bug is caused by the libvirt-bin upstart script [1] calling a
non-existant script, /usr/lib/libvirt/libvirt-stop-guests [2]. This script
used to exist in the upstart script itself in version 1.2.2-0ubuntu13.1.27
[3], the version in the trusty archives. In liberty UCA,
version 1.2.16-2ubuntu11.15.10.4~cloud0 [4], the script was seperated out
into /usr/lib/libvirt/libvirt-stop-guests [2]. In the mitaka release, the
libvirt-stop-guests script was removed and rewritten as
/etc/init.d/libvirt-guests [5], but the script in [1] was never updated to
point to it.
[1] http://paste.ubuntu.com/p/GxxBczkCmk
[2] http://paste.ubuntu.com/p/fKCDQh46vh
[3] http://paste.ubuntu.com/p/QrKXqK2Bvz
[4] http://paste.ubuntu.com/p/W8DgQwpYv3
[5] http://paste.ubuntu.com/p/Z28Sp2fPd6
* Since the upstart script was never updated to point to it, libvirt-bin
stops without stopping libvirt-guests first. When libvirt-guests is
stopped later, it cannot access the libvirt socket, cannot shut down the
machines, causing the bug.
* The fix is to change the upstart script to point to the new libvirt-guests
script.
[Test Case]
* You can reproduce this in trusty with the mitaka UCA enabled.
1) Enable mitaka UCA and install libvirt0 and libvirt-bin
$ sudo add-apt-repository cloud-archive:mitaka
$ sudo apt update
$ sudo apt install libvirt0 libvirt-bin
2) Install a virtual machine, either by using virt-install or virt-manager.
I used a bionic VM.
3) Enable debugging on libvirt-guests so you can see what is going on
Modify /etc/init.d/libvirt-guests and add "-x" to the end of "!/bin/sh"
4) With the vm running, shut down the system
$ sudo shutdown -h now
5) Check /var/log/upstart/libvirt-bin.log, on reboot. It will say
"No such file or directory: /usr/lib/libvirt/libvirt-stop-guests"
6) During that shutdown, you will see messages like:
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
No such file or directory
What should happen:
If you follow the same steps with the fixed package, when you look at
/var/log/upstart/libvirt-bin.log, you will see output of libvirt-guests
connecting to and shutting down the virtual machines which looks a little
like this: https://paste.ubuntu.com/p/s4jyJX2y9F/
[Regression Potential]
* There is only one file modified, the upstart script for libvirt-bin.
Currently this upstart file references a file which doesn't exist, so
fixing it will restore the behaviour in a way which aligns with exactly
what took place in previous versions.
* In xenial, there is no concern of stopping an already stopped service in
the event that the upstart script pre-stop section is called by systemd.
* This change only effects systems during shutdown while they still
have virtual machines running, and do not effect starting and stopping
services while the machine is running normally.
* I believe the regression potential is low.
[Other Info]
* Xenial is not effected by this bug even though it ships the exact same
packages. This is because xenial uses insserv to generate service
dependency files ".depend.boot" ".depend.start" ".depend.stop" which parse
the scripts in /etc/init.d/ and systemd respects the dependency ordering
in these files.
libvirt-guests reports a dependency on libvirt-bin in the script header,
so systemd will always stop libvirt-guests before libvirt-bin, avoiding
the problem seen in trusty.
* The fix is needed in trusty mitaka UCA and xenial will likely need the SRU
as part of the process. |
[Impact]
* libvirt-bin, in: libvirt-1.3.1-1ubuntu10.24~cloud0 in trusty mitaka uca
and the parent package in xenial, libvirt-1.3.1-1ubuntu10.24 are
affected.
* When you shutdown a system in trusty which is running some kvm virtual
machines, the libvirt-bin service is stopped before libvirt-guests.
libvirt-guests tries to connect to the libvirt socket to send shutdown
commands to the running vms, which cannot happen since libvirtd is not
running.
* On some machines, the qemu processes behind the virtual machines are
not killed and are left behind as defunct processes, which can cause
the system to hang on them not being terminated.
* The bug is caused by the libvirt-bin upstart script [1] calling a
non-existant script, /usr/lib/libvirt/libvirt-stop-guests [2]. This
script used to exist in the upstart script itself in version
1.2.2-0ubuntu13.1.27 [3], the version in the trusty archives. In
liberty UCA, version 1.2.16-2ubuntu11.15.10.4~cloud0 [4], the script
was separated out into /usr/lib/libvirt/libvirt-stop-guests [2].
In the mitaka release, the libvirt-stop-guests script was removed and
rewritten as /etc/init.d/libvirt-guests [5], but the script in [1] was
never updated to point to it.
[1] http://paste.ubuntu.com/p/GxxBczkCmk
[2] http://paste.ubuntu.com/p/fKCDQh46vh
[3] http://paste.ubuntu.com/p/QrKXqK2Bvz
[4] http://paste.ubuntu.com/p/W8DgQwpYv3
[5] http://paste.ubuntu.com/p/Z28Sp2fPd6
* Since the upstart script was never updated to point to it, libvirt-bin
stops without stopping libvirt-guests first. When libvirt-guests is
stopped later, it cannot access the libvirt socket, cannot shut down
the machines, causing the bug.
* The fix is to change the upstart script to point to the new libvirt-
guests script.
[Test Case]
* You can reproduce this in trusty with the mitaka UCA enabled.
1) Enable mitaka UCA and install libvirt0 and libvirt-bin
$ sudo add-apt-repository cloud-archive:mitaka
$ sudo apt update
$ sudo apt install libvirt0 libvirt-bin
2) Install a virtual machine, either by using virt-install or
virt-manager.
I used a bionic VM.
3) Enable debugging on libvirt-guests so you can see what is going on
Modify /etc/init.d/libvirt-guests and add "-x" to the end of "!/bin/sh"
4) With the vm running, shut down the system
$ sudo shutdown -h now
5) Check /var/log/upstart/libvirt-bin.log, on reboot. It will say
"No such file or directory: /usr/lib/libvirt/libvirt-stop-guests"
6) During that shutdown, you will see messages like:
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
No such file or directory
What should happen:
If you follow the same steps with the fixed package, when you look at
/var/log/upstart/libvirt-bin.log, you will see output of libvirt-guests
connecting to and shutting down the virtual machines which looks a little
like this: https://paste.ubuntu.com/p/s4jyJX2y9F/
[Regression Potential]
* There is only one file modified, the upstart script for libvirt-bin.
Currently this upstart file references a file which doesn't exist, so
fixing it will restore the behavior in a way which aligns with exactly
what took place in previous versions.
* In xenial, all of this isn't used at all - see below at "Other Info"
* This change only effects systems during shutdown while they still
have virtual machines running, and do not effect starting and stopping
services while the machine is running normally.
* I believe the regression potential is low.
[Other Info]
* Xenial is not effected by this bug even though it ships the exact same
packages. This is because xenial uses insserv to generate service
dependency files ".depend.boot" ".depend.start" ".depend.stop" which
parse the scripts in /etc/init.d/ and systemd respects the dependency
ordering in these files.
libvirt-guests reports a dependency on libvirt-bin in the script
header, so systemd will always stop libvirt-guests before libvirt-bin,
avoiding the problem seen in trusty.
* The fix is needed in trusty mitaka UCA and xenial will likely need the
SRU as part of the process.
* We'd never have uploaded that change alone for xenial (being a no-op
causing MBs to download and an upgrade. But we will bundle it with an
actual change - so it can "ride along" to eventually help
Trusty-Mitaka. |
|
| 2019-05-28 05:57:21 |
Christian Ehrhardt |
libvirt (Ubuntu Xenial): status |
Fix Released |
Triaged |
|
| 2019-05-28 05:59:00 |
Christian Ehrhardt |
libvirt (Ubuntu Xenial): assignee |
|
Christian Ehrhardt (paelzer) |
|
| 2019-05-28 18:21:49 |
Corey Bryant |
nominated for series |
|
cloud-archive/mitaka |
|
| 2019-05-28 18:21:49 |
Corey Bryant |
bug task added |
|
cloud-archive/mitaka |
|
| 2019-05-28 18:22:03 |
Corey Bryant |
cloud-archive/mitaka: status |
New |
In Progress |
|
| 2019-05-28 18:22:12 |
Corey Bryant |
cloud-archive/mitaka: assignee |
|
Matthew Ruffell (mruffell) |
|
| 2019-05-28 18:22:15 |
Corey Bryant |
cloud-archive: assignee |
Matthew Ruffell (mruffell) |
|
|
| 2019-05-28 18:22:17 |
Corey Bryant |
cloud-archive: status |
In Progress |
Invalid |
|
| 2019-06-03 12:52:07 |
Christian Ehrhardt |
libvirt (Ubuntu Xenial): importance |
Medium |
Wishlist |
|
| 2019-06-05 12:10:42 |
Christian Ehrhardt |
description |
[Impact]
* libvirt-bin, in: libvirt-1.3.1-1ubuntu10.24~cloud0 in trusty mitaka uca
and the parent package in xenial, libvirt-1.3.1-1ubuntu10.24 are
affected.
* When you shutdown a system in trusty which is running some kvm virtual
machines, the libvirt-bin service is stopped before libvirt-guests.
libvirt-guests tries to connect to the libvirt socket to send shutdown
commands to the running vms, which cannot happen since libvirtd is not
running.
* On some machines, the qemu processes behind the virtual machines are
not killed and are left behind as defunct processes, which can cause
the system to hang on them not being terminated.
* The bug is caused by the libvirt-bin upstart script [1] calling a
non-existant script, /usr/lib/libvirt/libvirt-stop-guests [2]. This
script used to exist in the upstart script itself in version
1.2.2-0ubuntu13.1.27 [3], the version in the trusty archives. In
liberty UCA, version 1.2.16-2ubuntu11.15.10.4~cloud0 [4], the script
was separated out into /usr/lib/libvirt/libvirt-stop-guests [2].
In the mitaka release, the libvirt-stop-guests script was removed and
rewritten as /etc/init.d/libvirt-guests [5], but the script in [1] was
never updated to point to it.
[1] http://paste.ubuntu.com/p/GxxBczkCmk
[2] http://paste.ubuntu.com/p/fKCDQh46vh
[3] http://paste.ubuntu.com/p/QrKXqK2Bvz
[4] http://paste.ubuntu.com/p/W8DgQwpYv3
[5] http://paste.ubuntu.com/p/Z28Sp2fPd6
* Since the upstart script was never updated to point to it, libvirt-bin
stops without stopping libvirt-guests first. When libvirt-guests is
stopped later, it cannot access the libvirt socket, cannot shut down
the machines, causing the bug.
* The fix is to change the upstart script to point to the new libvirt-
guests script.
[Test Case]
* You can reproduce this in trusty with the mitaka UCA enabled.
1) Enable mitaka UCA and install libvirt0 and libvirt-bin
$ sudo add-apt-repository cloud-archive:mitaka
$ sudo apt update
$ sudo apt install libvirt0 libvirt-bin
2) Install a virtual machine, either by using virt-install or
virt-manager.
I used a bionic VM.
3) Enable debugging on libvirt-guests so you can see what is going on
Modify /etc/init.d/libvirt-guests and add "-x" to the end of "!/bin/sh"
4) With the vm running, shut down the system
$ sudo shutdown -h now
5) Check /var/log/upstart/libvirt-bin.log, on reboot. It will say
"No such file or directory: /usr/lib/libvirt/libvirt-stop-guests"
6) During that shutdown, you will see messages like:
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
No such file or directory
What should happen:
If you follow the same steps with the fixed package, when you look at
/var/log/upstart/libvirt-bin.log, you will see output of libvirt-guests
connecting to and shutting down the virtual machines which looks a little
like this: https://paste.ubuntu.com/p/s4jyJX2y9F/
[Regression Potential]
* There is only one file modified, the upstart script for libvirt-bin.
Currently this upstart file references a file which doesn't exist, so
fixing it will restore the behavior in a way which aligns with exactly
what took place in previous versions.
* In xenial, all of this isn't used at all - see below at "Other Info"
* This change only effects systems during shutdown while they still
have virtual machines running, and do not effect starting and stopping
services while the machine is running normally.
* I believe the regression potential is low.
[Other Info]
* Xenial is not effected by this bug even though it ships the exact same
packages. This is because xenial uses insserv to generate service
dependency files ".depend.boot" ".depend.start" ".depend.stop" which
parse the scripts in /etc/init.d/ and systemd respects the dependency
ordering in these files.
libvirt-guests reports a dependency on libvirt-bin in the script
header, so systemd will always stop libvirt-guests before libvirt-bin,
avoiding the problem seen in trusty.
* The fix is needed in trusty mitaka UCA and xenial will likely need the
SRU as part of the process.
* We'd never have uploaded that change alone for xenial (being a no-op
causing MBs to download and an upgrade. But we will bundle it with an
actual change - so it can "ride along" to eventually help
Trusty-Mitaka. |
[Impact]
* libvirt-bin, in: libvirt-1.3.1-1ubuntu10.24~cloud0 in trusty mitaka uca
and the parent package in xenial, libvirt-1.3.1-1ubuntu10.24 are
affected.
* When you shutdown a system in trusty which is running some kvm virtual
machines, the libvirt-bin service is stopped before libvirt-guests.
libvirt-guests tries to connect to the libvirt socket to send shutdown
commands to the running vms, which cannot happen since libvirtd is not
running.
* On some machines, the qemu processes behind the virtual machines are
not killed and are left behind as defunct processes, which can cause
the system to hang on them not being terminated.
* The bug is caused by the libvirt-bin upstart script [1] calling a
non-existant script, /usr/lib/libvirt/libvirt-stop-guests [2]. This
script used to exist in the upstart script itself in version
1.2.2-0ubuntu13.1.27 [3], the version in the trusty archives. In
liberty UCA, version 1.2.16-2ubuntu11.15.10.4~cloud0 [4], the script
was separated out into /usr/lib/libvirt/libvirt-stop-guests [2].
In the mitaka release, the libvirt-stop-guests script was removed and
rewritten as /etc/init.d/libvirt-guests [5], but the script in [1] was
never updated to point to it.
[1] http://paste.ubuntu.com/p/GxxBczkCmk
[2] http://paste.ubuntu.com/p/fKCDQh46vh
[3] http://paste.ubuntu.com/p/QrKXqK2Bvz
[4] http://paste.ubuntu.com/p/W8DgQwpYv3
[5] http://paste.ubuntu.com/p/Z28Sp2fPd6
* Since the upstart script was never updated to point to it, libvirt-bin
stops without stopping libvirt-guests first. When libvirt-guests is
stopped later, it cannot access the libvirt socket, cannot shut down
the machines, causing the bug.
* The fix is to change the upstart script to point to the new libvirt-
guests script.
[Test Case]
* You can reproduce this in trusty with the mitaka UCA enabled.
1) Enable mitaka UCA and install libvirt0 and libvirt-bin
$ sudo add-apt-repository cloud-archive:mitaka
$ sudo apt update
$ sudo apt install libvirt0 libvirt-bin
2) Install a virtual machine, either by using virt-install or
virt-manager.
I used a bionic VM.
3) Enable debugging on libvirt-guests so you can see what is going on
Modify /etc/init.d/libvirt-guests and add "-x" to the end of "!/bin/sh"
4) With the vm running, shut down the system
$ sudo shutdown -h now
5) Check /var/log/upstart/libvirt-bin.log, on reboot. It will say
"No such file or directory: /usr/lib/libvirt/libvirt-stop-guests"
6) During that shutdown, you will see messages like:
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock':
No such file or directory
What should happen:
If you follow the same steps with the fixed package, when you look at
/var/log/upstart/libvirt-bin.log, you will see output of libvirt-guests
connecting to and shutting down the virtual machines which looks a little
like this: https://paste.ubuntu.com/p/s4jyJX2y9F/
[Regression Potential]
* There is only one file modified, the upstart script for libvirt-bin.
Currently this upstart file references a file which doesn't exist, so
fixing it will restore the behavior in a way which aligns with exactly
what took place in previous versions.
* In xenial, all of this isn't used at all - see below at "Other Info"
* This change only effects systems during shutdown while they still
have virtual machines running, and do not effect starting and stopping
services while the machine is running normally.
* I believe the regression potential is low.
[Other Info]
* Xenial is not effected by this bug even though it ships the exact same
packages. This is because xenial uses insserv to generate service
dependency files ".depend.boot" ".depend.start" ".depend.stop" which
parse the scripts in /etc/init.d/ and systemd respects the dependency
ordering in these files.
libvirt-guests reports a dependency on libvirt-bin in the script
header, so systemd will always stop libvirt-guests before libvirt-bin,
avoiding the problem seen in trusty.
* The fix is needed in trusty mitaka UCA and xenial will likely need the
SRU as part of the process.
* We'd never have uploaded that change alone for xenial (being a no-op
causing MBs to download and an upgrade. But we will bundle it with an
actual change - so it can "ride along" to eventually help
Trusty-Mitaka. Unfortunately there is no "current" Xenial SRU for
libvirt, hence we want to get it into xenial-proposed (which is enough
for the UCA tooling) but we do not want to release it to xenial-updates
until another another SRU comes by which we will generate with a -v
covering both then.
* That way also if e.g. a security fix comes by it will be based on what
is in proposed. |
|
| 2019-06-05 12:22:10 |
Corey Bryant |
cloud-archive/mitaka: importance |
Undecided |
High |
|
| 2019-07-18 11:27:46 |
Robie Basak |
bug |
|
|
added subscriber Robie Basak |
| 2019-07-24 16:08:11 |
Robie Basak |
bug |
|
|
added subscriber Corey Bryant |
| 2019-07-24 16:46:00 |
Robie Basak |
tags |
sts |
block-proposed sts |
|
| 2019-07-24 16:51:29 |
Robie Basak |
libvirt (Ubuntu Xenial): status |
Triaged |
Fix Committed |
|
| 2019-07-24 16:51:30 |
Robie Basak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-07-24 16:51:34 |
Robie Basak |
bug |
|
|
added subscriber SRU Verification |
| 2019-07-24 16:51:38 |
Robie Basak |
tags |
block-proposed sts |
block-proposed sts verification-needed verification-needed-xenial |
|
| 2019-07-26 05:06:58 |
Matthew Ruffell |
tags |
block-proposed sts verification-needed verification-needed-xenial |
block-proposed sts verification-done-xenial |
|
| 2019-08-08 06:46:35 |
Mathew Hodson |
bug |
|
|
added subscriber Mathew Hodson |
| 2019-09-04 15:07:44 |
Hari Kumar |
libvirt (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2019-09-04 21:11:07 |
Matthew Ruffell |
libvirt (Ubuntu Xenial): status |
Fix Released |
Fix Committed |
|
| 2019-09-23 09:45:42 |
Łukasz Zemczak |
tags |
block-proposed sts verification-done-xenial |
block-proposed-bionic sts verification-done-xenial |
|
| 2019-09-25 21:19:40 |
Robie Basak |
tags |
block-proposed-bionic sts verification-done-xenial |
block-proposed-xenial sts verification-done-xenial |
|
| 2019-10-22 16:18:14 |
Corey Bryant |
cloud-archive/mitaka: status |
In Progress |
Fix Committed |
|
| 2019-10-22 16:18:16 |
Corey Bryant |
tags |
block-proposed-xenial sts verification-done-xenial |
block-proposed-xenial sts verification-done-xenial verification-mitaka-needed |
|
| 2019-10-28 15:28:49 |
Fabio Augusto Miranda Martins |
tags |
block-proposed-xenial sts verification-done-xenial verification-mitaka-needed |
block-proposed-xenial sts verification-done-mitaka verification-done-xenial |
|
| 2019-10-30 23:46:09 |
Mathew Hodson |
cloud-archive: status |
Invalid |
Fix Released |
|
| 2019-10-30 23:46:12 |
Mathew Hodson |
libvirt (Ubuntu): status |
Invalid |
Fix Released |
|
| 2019-10-30 23:46:38 |
Mathew Hodson |
removed subscriber Mathew Hodson |
|
|
|
| 2019-10-31 12:25:39 |
Corey Bryant |
cloud-archive: status |
Fix Released |
Invalid |
|
| 2020-01-13 13:57:34 |
Łukasz Zemczak |
tags |
block-proposed-xenial sts verification-done-mitaka verification-done-xenial |
sts verification-done-mitaka verification-done-xenial |
|
| 2020-01-13 13:58:44 |
Launchpad Janitor |
libvirt (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2020-01-16 21:55:13 |
Corey Bryant |
cloud-archive/mitaka: status |
Fix Committed |
Fix Released |
|
