| 2024-03-27 15:00:41 |
Mauricio Faria de Oliveira |
bug |
|
|
added bug |
| 2024-03-27 15:00:59 |
Mauricio Faria de Oliveira |
nominated for series |
|
Ubuntu Focal |
|
| 2024-03-27 15:00:59 |
Mauricio Faria de Oliveira |
bug task added |
|
libvirt (Ubuntu Focal) |
|
| 2024-03-27 15:00:59 |
Mauricio Faria de Oliveira |
nominated for series |
|
Ubuntu Jammy |
|
| 2024-03-27 15:00:59 |
Mauricio Faria de Oliveira |
bug task added |
|
libvirt (Ubuntu Jammy) |
|
| 2024-03-27 15:01:05 |
Mauricio Faria de Oliveira |
libvirt (Ubuntu): status |
New |
Fix Released |
|
| 2024-03-27 15:01:12 |
Mauricio Faria de Oliveira |
libvirt (Ubuntu Focal): status |
New |
Confirmed |
|
| 2024-03-27 15:01:15 |
Mauricio Faria de Oliveira |
libvirt (Ubuntu Focal): importance |
Undecided |
Medium |
|
| 2024-03-27 15:01:18 |
Mauricio Faria de Oliveira |
libvirt (Ubuntu Focal): assignee |
|
Mauricio Faria de Oliveira (mfo) |
|
| 2024-03-27 15:01:20 |
Mauricio Faria de Oliveira |
libvirt (Ubuntu Jammy): status |
New |
Confirmed |
|
| 2024-03-27 15:01:27 |
Mauricio Faria de Oliveira |
libvirt (Ubuntu Jammy): importance |
Undecided |
Medium |
|
| 2024-03-27 15:01:29 |
Mauricio Faria de Oliveira |
libvirt (Ubuntu Jammy): assignee |
|
Mauricio Faria de Oliveira (mfo) |
|
| 2024-03-27 15:02:02 |
Mauricio Faria de Oliveira |
description |
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
|
| 2024-03-27 17:43:34 |
Mauricio Faria de Oliveira |
description |
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* Synthetic reproducer with GDB in comment #1.
* On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
* On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
[ Regression Potential ]
* The patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
|
| 2024-03-27 19:18:14 |
Mauricio Faria de Oliveira |
description |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* Synthetic reproducer with GDB in comment #1.
* On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
* On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
[ Regression Potential ]
* The patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* Synthetic reproducer with GDB in comment #1.
* On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
* On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
[ Regression Potential ]
* The patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Other Info]
* The fix commit [1] is included in Mantic and later,
and needed in Focal and Jammy.
$ git describe --contains 152770333449cd3b78b4f5a9f1148fc1f482d842
v9.3.0-rc1~90
$ rmadison -a source libvirt | sed -n '/focal/,$p'
libvirt | 6.0.0-0ubuntu8 | focal | source
libvirt | 6.0.0-0ubuntu8.16 | focal-security | source
libvirt | 6.0.0-0ubuntu8.16 | focal-updates | source
libvirt | 6.0.0-0ubuntu8.17 | focal-proposed | source
libvirt | 8.0.0-1ubuntu7 | jammy | source
libvirt | 8.0.0-1ubuntu7.5 | jammy-security | source
libvirt | 8.0.0-1ubuntu7.8 | jammy-updates | source
libvirt | 9.6.0-1ubuntu1 | mantic | source
libvirt | 10.0.0-2ubuntu1 | noble | source
libvirt | 10.0.0-2ubuntu5 | noble-proposed | source
[1] https://gitlab.com/libvirt/libvirt/-/commit/152770333449cd3b78b4f5a9f1148fc1f482d842
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
|
| 2024-03-27 19:19:39 |
Mauricio Faria de Oliveira |
libvirt (Ubuntu Jammy): status |
Confirmed |
In Progress |
|
| 2024-03-28 13:57:39 |
Andreas Hasenack |
libvirt (Ubuntu Jammy): status |
In Progress |
Fix Committed |
|
| 2024-03-28 13:57:40 |
Andreas Hasenack |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2024-03-28 13:57:47 |
Andreas Hasenack |
bug |
|
|
added subscriber SRU Verification |
| 2024-03-28 13:57:54 |
Andreas Hasenack |
tags |
|
verification-needed verification-needed-jammy |
|
| 2024-03-28 14:40:47 |
Mauricio Faria de Oliveira |
description |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* Synthetic reproducer with GDB in comment #1.
* On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
* On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
[ Regression Potential ]
* The patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Other Info]
* The fix commit [1] is included in Mantic and later,
and needed in Focal and Jammy.
$ git describe --contains 152770333449cd3b78b4f5a9f1148fc1f482d842
v9.3.0-rc1~90
$ rmadison -a source libvirt | sed -n '/focal/,$p'
libvirt | 6.0.0-0ubuntu8 | focal | source
libvirt | 6.0.0-0ubuntu8.16 | focal-security | source
libvirt | 6.0.0-0ubuntu8.16 | focal-updates | source
libvirt | 6.0.0-0ubuntu8.17 | focal-proposed | source
libvirt | 8.0.0-1ubuntu7 | jammy | source
libvirt | 8.0.0-1ubuntu7.5 | jammy-security | source
libvirt | 8.0.0-1ubuntu7.8 | jammy-updates | source
libvirt | 9.6.0-1ubuntu1 | mantic | source
libvirt | 10.0.0-2ubuntu1 | noble | source
libvirt | 10.0.0-2ubuntu5 | noble-proposed | source
[1] https://gitlab.com/libvirt/libvirt/-/commit/152770333449cd3b78b4f5a9f1148fc1f482d842
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* Synthetic reproducer with GDB in comment #1.
On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
* Normal 'restart' testing in comment #5.
[ Regression Potential ]
* The patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Other Info]
* The fix commit [1] is included in Mantic and later,
and needed in Focal and Jammy.
$ git describe --contains 152770333449cd3b78b4f5a9f1148fc1f482d842
v9.3.0-rc1~90
$ rmadison -a source libvirt | sed -n '/focal/,$p'
libvirt | 6.0.0-0ubuntu8 | focal | source
libvirt | 6.0.0-0ubuntu8.16 | focal-security | source
libvirt | 6.0.0-0ubuntu8.16 | focal-updates | source
libvirt | 6.0.0-0ubuntu8.17 | focal-proposed | source
libvirt | 8.0.0-1ubuntu7 | jammy | source
libvirt | 8.0.0-1ubuntu7.5 | jammy-security | source
libvirt | 8.0.0-1ubuntu7.8 | jammy-updates | source
libvirt | 9.6.0-1ubuntu1 | mantic | source
libvirt | 10.0.0-2ubuntu1 | noble | source
libvirt | 10.0.0-2ubuntu5 | noble-proposed | source
[1] https://gitlab.com/libvirt/libvirt/-/commit/152770333449cd3b78b4f5a9f1148fc1f482d842
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
|
| 2024-03-30 17:44:22 |
Mauricio Faria de Oliveira |
description |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* Synthetic reproducer with GDB in comment #1.
On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
* Normal 'restart' testing in comment #5.
[ Regression Potential ]
* The patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Other Info]
* The fix commit [1] is included in Mantic and later,
and needed in Focal and Jammy.
$ git describe --contains 152770333449cd3b78b4f5a9f1148fc1f482d842
v9.3.0-rc1~90
$ rmadison -a source libvirt | sed -n '/focal/,$p'
libvirt | 6.0.0-0ubuntu8 | focal | source
libvirt | 6.0.0-0ubuntu8.16 | focal-security | source
libvirt | 6.0.0-0ubuntu8.16 | focal-updates | source
libvirt | 6.0.0-0ubuntu8.17 | focal-proposed | source
libvirt | 8.0.0-1ubuntu7 | jammy | source
libvirt | 8.0.0-1ubuntu7.5 | jammy-security | source
libvirt | 8.0.0-1ubuntu7.8 | jammy-updates | source
libvirt | 9.6.0-1ubuntu1 | mantic | source
libvirt | 10.0.0-2ubuntu1 | noble | source
libvirt | 10.0.0-2ubuntu5 | noble-proposed | source
[1] https://gitlab.com/libvirt/libvirt/-/commit/152770333449cd3b78b4f5a9f1148fc1f482d842
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* Synthetic reproducer with GDB in comments #1 and #2.
On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
(comment #1)
On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
(comment #2)
* Normal 'restart' testing in comment #5.
[ Regression Potential ]
* The patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Other Info]
* The fix commit [1] is included in Mantic and later,
and needed in Focal and Jammy.
$ git describe --contains 152770333449cd3b78b4f5a9f1148fc1f482d842
v9.3.0-rc1~90
$ rmadison -a source libvirt | sed -n '/focal/,$p'
libvirt | 6.0.0-0ubuntu8 | focal | source
libvirt | 6.0.0-0ubuntu8.16 | focal-security | source
libvirt | 6.0.0-0ubuntu8.16 | focal-updates | source
libvirt | 6.0.0-0ubuntu8.17 | focal-proposed | source
libvirt | 8.0.0-1ubuntu7 | jammy | source
libvirt | 8.0.0-1ubuntu7.5 | jammy-security | source
libvirt | 8.0.0-1ubuntu7.8 | jammy-updates | source
libvirt | 9.6.0-1ubuntu1 | mantic | source
libvirt | 10.0.0-2ubuntu1 | noble | source
libvirt | 10.0.0-2ubuntu5 | noble-proposed | source
[1] https://gitlab.com/libvirt/libvirt/-/commit/152770333449cd3b78b4f5a9f1148fc1f482d842
* Test packages built successfully in all architectures
with -proposed enabled in Launchpad PPA mfo/lp2059272 [2]
[2] https://launchpad.net/~mfo/+archive/ubuntu/lp2059272
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
|
| 2024-03-30 18:04:43 |
Mauricio Faria de Oliveira |
tags |
verification-needed verification-needed-jammy |
verification-done-jammy verification-needed |
|
| 2024-03-30 23:53:16 |
Mauricio Faria de Oliveira |
description |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* Synthetic reproducer with GDB in comments #1 and #2.
On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
(comment #1)
On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
(comment #2)
* Normal 'restart' testing in comment #5.
[ Regression Potential ]
* The patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Other Info]
* The fix commit [1] is included in Mantic and later,
and needed in Focal and Jammy.
$ git describe --contains 152770333449cd3b78b4f5a9f1148fc1f482d842
v9.3.0-rc1~90
$ rmadison -a source libvirt | sed -n '/focal/,$p'
libvirt | 6.0.0-0ubuntu8 | focal | source
libvirt | 6.0.0-0ubuntu8.16 | focal-security | source
libvirt | 6.0.0-0ubuntu8.16 | focal-updates | source
libvirt | 6.0.0-0ubuntu8.17 | focal-proposed | source
libvirt | 8.0.0-1ubuntu7 | jammy | source
libvirt | 8.0.0-1ubuntu7.5 | jammy-security | source
libvirt | 8.0.0-1ubuntu7.8 | jammy-updates | source
libvirt | 9.6.0-1ubuntu1 | mantic | source
libvirt | 10.0.0-2ubuntu1 | noble | source
libvirt | 10.0.0-2ubuntu5 | noble-proposed | source
[1] https://gitlab.com/libvirt/libvirt/-/commit/152770333449cd3b78b4f5a9f1148fc1f482d842
* Test packages built successfully in all architectures
with -proposed enabled in Launchpad PPA mfo/lp2059272 [2]
[2] https://launchpad.net/~mfo/+archive/ubuntu/lp2059272
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
(The race condition details are in [Other Info].)
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* (Focal/Jammy) shutdown-on-runtime:
Synthetic reproducer/verification with GDB in comments #1 and #2 (Jammy) and #12 and #14 (Focal).
* (Focal-only) shutdown-on-init:
Synthetic reproducer/verification with GDB in comments #13 and #15.
* On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
* On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
* Normal 'restart' testing in comment #5.
* Test packages built successfully in all architectures
with -proposed enabled in Launchpad PPA mfo/lp2059272 [0]
[0] https://launchpad.net/~mfo/+archive/ubuntu/lp2059272
[ Regression Potential ]
* One patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* The other patch (Focal-only) introduces a bounded wait
(with configurable timeout via an environment variable)
in the (same) libvirt qemu driver's shutdown path.
By default, this waits for qemuProcessReconnect threads
for up to 30 seconds (expected to finish in less than
1 second, in practice), and gives up / continues with
shutdown anyway so not to introduce a behavior change
on this path (prevents impact in case of regressions).
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Other Info]
* In Focal, race windows exist if libvirtd shuts down
_after_ initialization and _during_ initialization
(which is unlikely in practice, but it's possible.)
Say, 'shutdown'on-runtime' and 'shutdown-on-init'.
* In Jammy, only 'shutdown-on-runtime' might happen,
due to the introduction of the '.stateShutdownWait'
driver callback (not available in Focal), which
indirectly prevents the 'shutdown-on-init' race
due to additional synchronization with locking.
* For 'shutdown-on-init' (Focal-only), we should use a
downstream-only patch (with configurable behavior),
since upstream addressed this issue indirectly with
the '.stateShutdownWait' callbacks and other changes
(which are not SRU material, ~10 patches, redesign [2]).
* For 'shutdown-on-runtime': use upstream commit [1].
It's needed in Focal and Jammy (included in Mantic).
$ git describe --contains 152770333449cd3b78b4f5a9f1148fc1f482d842
v9.3.0-rc1~90
$ rmadison -a source libvirt | sed -n '/focal/,$p'
libvirt | 6.0.0-0ubuntu8 | focal | source
libvirt | 6.0.0-0ubuntu8.16 | focal-security | source
libvirt | 6.0.0-0ubuntu8.16 | focal-updates | source
libvirt | 6.0.0-0ubuntu8.17 | focal-proposed | source
libvirt | 8.0.0-1ubuntu7 | jammy | source
libvirt | 8.0.0-1ubuntu7.5 | jammy-security | source
libvirt | 8.0.0-1ubuntu7.8 | jammy-updates | source
libvirt | 9.6.0-1ubuntu1 | mantic | source
libvirt | 10.0.0-2ubuntu1 | noble | source
libvirt | 10.0.0-2ubuntu5 | noble-proposed | source
[1] https://gitlab.com/libvirt/libvirt/-/commit/152770333449cd3b78b4f5a9f1148fc1f482d842
[2] https://listman.redhat.com/archives/libvir-list/2020-July/205291.html
PATCH 00/10] resolve hangs/crashes on libvirtd shutdown
commit 94e45d1042e21e03a15ce993f90fbef626f1ae41
Author: Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
Date: Thu Jul 23 09:53:04 2020 +0300
rpc: finish all threads before exiting main loop
$ git describe --contains 94e45d1042e21e03a15ce993f90fbef626f1ae41
v6.8.0-rc1~279
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
|
| 2024-03-30 23:57:32 |
Mauricio Faria de Oliveira |
description |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
(The race condition details are in [Other Info].)
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* (Focal/Jammy) shutdown-on-runtime:
Synthetic reproducer/verification with GDB in comments #1 and #2 (Jammy) and #12 and #14 (Focal).
* (Focal-only) shutdown-on-init:
Synthetic reproducer/verification with GDB in comments #13 and #15.
* On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
* On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
* Normal 'restart' testing in comment #5.
* Test packages built successfully in all architectures
with -proposed enabled in Launchpad PPA mfo/lp2059272 [0]
[0] https://launchpad.net/~mfo/+archive/ubuntu/lp2059272
[ Regression Potential ]
* One patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* The other patch (Focal-only) introduces a bounded wait
(with configurable timeout via an environment variable)
in the (same) libvirt qemu driver's shutdown path.
By default, this waits for qemuProcessReconnect threads
for up to 30 seconds (expected to finish in less than
1 second, in practice), and gives up / continues with
shutdown anyway so not to introduce a behavior change
on this path (prevents impact in case of regressions).
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Other Info]
* In Focal, race windows exist if libvirtd shuts down
_after_ initialization and _during_ initialization
(which is unlikely in practice, but it's possible.)
Say, 'shutdown'on-runtime' and 'shutdown-on-init'.
* In Jammy, only 'shutdown-on-runtime' might happen,
due to the introduction of the '.stateShutdownWait'
driver callback (not available in Focal), which
indirectly prevents the 'shutdown-on-init' race
due to additional synchronization with locking.
* For 'shutdown-on-init' (Focal-only), we should use a
downstream-only patch (with configurable behavior),
since upstream addressed this issue indirectly with
the '.stateShutdownWait' callbacks and other changes
(which are not SRU material, ~10 patches, redesign [2]).
* For 'shutdown-on-runtime': use upstream commit [1].
It's needed in Focal and Jammy (included in Mantic).
$ git describe --contains 152770333449cd3b78b4f5a9f1148fc1f482d842
v9.3.0-rc1~90
$ rmadison -a source libvirt | sed -n '/focal/,$p'
libvirt | 6.0.0-0ubuntu8 | focal | source
libvirt | 6.0.0-0ubuntu8.16 | focal-security | source
libvirt | 6.0.0-0ubuntu8.16 | focal-updates | source
libvirt | 6.0.0-0ubuntu8.17 | focal-proposed | source
libvirt | 8.0.0-1ubuntu7 | jammy | source
libvirt | 8.0.0-1ubuntu7.5 | jammy-security | source
libvirt | 8.0.0-1ubuntu7.8 | jammy-updates | source
libvirt | 9.6.0-1ubuntu1 | mantic | source
libvirt | 10.0.0-2ubuntu1 | noble | source
libvirt | 10.0.0-2ubuntu5 | noble-proposed | source
[1] https://gitlab.com/libvirt/libvirt/-/commit/152770333449cd3b78b4f5a9f1148fc1f482d842
[2] https://listman.redhat.com/archives/libvir-list/2020-July/205291.html
PATCH 00/10] resolve hangs/crashes on libvirtd shutdown
commit 94e45d1042e21e03a15ce993f90fbef626f1ae41
Author: Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
Date: Thu Jul 23 09:53:04 2020 +0300
rpc: finish all threads before exiting main loop
$ git describe --contains 94e45d1042e21e03a15ce993f90fbef626f1ae41
v6.8.0-rc1~279
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
(The race condition details are in [Other Info].)
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* (Focal/Jammy) shutdown-on-runtime:
Synthetic reproducer/verification with GDB in comments #1 and #2 (Jammy) and #12 and #14 (Focal).
* (Focal-only) shutdown-on-init:
Synthetic reproducer/verification with GDB in comments #13 and #15.
* On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
* On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
* Normal 'restart' testing in comment #5.
* Test packages built successfully in all architectures
with -proposed enabled in Launchpad PPA mfo/lp2059272 [0]
[0] https://launchpad.net/~mfo/+archive/ubuntu/lp2059272
[ Regression Potential ]
* One patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* The other patch (Focal-only) introduces a bounded wait
(with configurable timeout via an environment variable)
in the (same) libvirt qemu driver's shutdown path.
By default, this waits for qemuProcessReconnect threads
for up to 30 seconds (expected to finish in less than
1 second, in practice), and gives up / continues with
shutdown anyway so not to introduce a behavior change
on this path (prevents impact in case of regressions).
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Other Info]
* In Focal, race windows exist if libvirtd shuts down
_after_ initialization and _during_ initialization
(which is unlikely in practice, but it's possible.)
Say, 'shutdown'on-runtime' and 'shutdown-on-init'.
* In Jammy, only 'shutdown-on-runtime' might happen,
due to the introduction of the '.stateShutdownWait'
driver callback (not available in Focal), which
indirectly prevents the 'shutdown-on-init' race
due to additional synchronization with locking.
* For 'shutdown-on-runtime': use upstream commit [1].
It's needed in Focal and Jammy (included in Mantic).
* For 'shutdown-on-init' (Focal-only), we should use a
downstream-only patch (with configurable behavior),
since upstream addressed this issue indirectly with
the '.stateShutdownWait' callbacks and other changes
(which are not SRU material, ~10 patches, redesign [2])
in 6.8.0.
[1] https://gitlab.com/libvirt/libvirt/-/commit/152770333449cd3b78b4f5a9f1148fc1f482d842
$ git describe --contains 152770333449cd3b78b4f5a9f1148fc1f482d842
v9.3.0-rc1~90
$ rmadison -a source libvirt | sed -n '/focal/,$p'
libvirt | 6.0.0-0ubuntu8 | focal | source
libvirt | 6.0.0-0ubuntu8.16 | focal-security | source
libvirt | 6.0.0-0ubuntu8.16 | focal-updates | source
libvirt | 6.0.0-0ubuntu8.17 | focal-proposed | source
libvirt | 8.0.0-1ubuntu7 | jammy | source
libvirt | 8.0.0-1ubuntu7.5 | jammy-security | source
libvirt | 8.0.0-1ubuntu7.8 | jammy-updates | source
libvirt | 9.6.0-1ubuntu1 | mantic | source
libvirt | 10.0.0-2ubuntu1 | noble | source
libvirt | 10.0.0-2ubuntu5 | noble-proposed | source
[2] https://listman.redhat.com/archives/libvir-list/2020-July/205291.html
[PATCH 00/10] resolve hangs/crashes on libvirtd shutdown
commit 94e45d1042e21e03a15ce993f90fbef626f1ae41
Author: Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
Date: Thu Jul 23 09:53:04 2020 +0300
rpc: finish all threads before exiting main loop
$ git describe --contains 94e45d1042e21e03a15ce993f90fbef626f1ae41
v6.8.0-rc1~279
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
|
| 2024-04-01 21:15:56 |
Mauricio Faria de Oliveira |
libvirt (Ubuntu Focal): status |
Confirmed |
In Progress |
|
| 2024-04-03 02:06:45 |
Chris Halse Rogers |
libvirt (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2024-04-03 02:06:50 |
Chris Halse Rogers |
tags |
verification-done-jammy verification-needed |
verification-done-jammy verification-needed verification-needed-focal |
|
| 2024-04-10 00:27:51 |
Brian Murray |
tags |
verification-done-jammy verification-needed verification-needed-focal |
verification-done-jammy verification-failed-focal verification-needed |
|
| 2024-04-10 00:29:36 |
Launchpad Janitor |
libvirt (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2024-04-10 00:29:42 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2024-04-12 18:45:47 |
Mauricio Faria de Oliveira |
description |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
(The race condition details are in [Other Info].)
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* (Focal/Jammy) shutdown-on-runtime:
Synthetic reproducer/verification with GDB in comments #1 and #2 (Jammy) and #12 and #14 (Focal).
* (Focal-only) shutdown-on-init:
Synthetic reproducer/verification with GDB in comments #13 and #15.
* On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
* On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
* Normal 'restart' testing in comment #5.
* Test packages built successfully in all architectures
with -proposed enabled in Launchpad PPA mfo/lp2059272 [0]
[0] https://launchpad.net/~mfo/+archive/ubuntu/lp2059272
[ Regression Potential ]
* One patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* The other patch (Focal-only) introduces a bounded wait
(with configurable timeout via an environment variable)
in the (same) libvirt qemu driver's shutdown path.
By default, this waits for qemuProcessReconnect threads
for up to 30 seconds (expected to finish in less than
1 second, in practice), and gives up / continues with
shutdown anyway so not to introduce a behavior change
on this path (prevents impact in case of regressions).
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Other Info]
* In Focal, race windows exist if libvirtd shuts down
_after_ initialization and _during_ initialization
(which is unlikely in practice, but it's possible.)
Say, 'shutdown'on-runtime' and 'shutdown-on-init'.
* In Jammy, only 'shutdown-on-runtime' might happen,
due to the introduction of the '.stateShutdownWait'
driver callback (not available in Focal), which
indirectly prevents the 'shutdown-on-init' race
due to additional synchronization with locking.
* For 'shutdown-on-runtime': use upstream commit [1].
It's needed in Focal and Jammy (included in Mantic).
* For 'shutdown-on-init' (Focal-only), we should use a
downstream-only patch (with configurable behavior),
since upstream addressed this issue indirectly with
the '.stateShutdownWait' callbacks and other changes
(which are not SRU material, ~10 patches, redesign [2])
in 6.8.0.
[1] https://gitlab.com/libvirt/libvirt/-/commit/152770333449cd3b78b4f5a9f1148fc1f482d842
$ git describe --contains 152770333449cd3b78b4f5a9f1148fc1f482d842
v9.3.0-rc1~90
$ rmadison -a source libvirt | sed -n '/focal/,$p'
libvirt | 6.0.0-0ubuntu8 | focal | source
libvirt | 6.0.0-0ubuntu8.16 | focal-security | source
libvirt | 6.0.0-0ubuntu8.16 | focal-updates | source
libvirt | 6.0.0-0ubuntu8.17 | focal-proposed | source
libvirt | 8.0.0-1ubuntu7 | jammy | source
libvirt | 8.0.0-1ubuntu7.5 | jammy-security | source
libvirt | 8.0.0-1ubuntu7.8 | jammy-updates | source
libvirt | 9.6.0-1ubuntu1 | mantic | source
libvirt | 10.0.0-2ubuntu1 | noble | source
libvirt | 10.0.0-2ubuntu5 | noble-proposed | source
[2] https://listman.redhat.com/archives/libvir-list/2020-July/205291.html
[PATCH 00/10] resolve hangs/crashes on libvirtd shutdown
commit 94e45d1042e21e03a15ce993f90fbef626f1ae41
Author: Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
Date: Thu Jul 23 09:53:04 2020 +0300
rpc: finish all threads before exiting main loop
$ git describe --contains 94e45d1042e21e03a15ce993f90fbef626f1ae41
v6.8.0-rc1~279
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
[ Impact ]
* If a race condition occurs on libvirtd shutdown,
a QEMU domain status XML (/run/libvirt/qemu/*.xml)
might lose the QEMU-driver specific information,
such as '<monitor path=.../>'.
(The race condition details are in [Other Info].)
* On the next libvirtd startup, the parsing of that
QEMU domain's status XML fails as '<monitor path='
is not found:
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
* As a result, the domain is not listed in `virsh list`,
and `virsh` commands to it fail.
$ virsh list
Id Name State
--------------------
* The domain is still running, but libvirt considers
it as shutdown, which might cause conflicts/issues
with higher-level tools (e.g., openstack nova).
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm,
[ Test Plan ]
* (Focal/Jammy) shutdown-on-runtime:
Synthetic reproducer/verification with GDB in comments #1 and #2 (Jammy) and #12 and #14 (Focal).
* (Focal-only) shutdown-on-init:
Synthetic reproducer/verification with GDB in comments #13 and #15.
* On failure, the XML is saved *without* '<monitor path='
and libvirt fails to parse the domain on startup.
The domain is *not* listed in `virsh list`.
* On success, the XML is saved *with* '<monitor path='
and libvirt correctly parses the domain on startup.
The domain is listed in `virsh list`.
* Normal 'restart' testing in comment #5.
* Test packages built successfully in all architectures
with -proposed enabled in Launchpad PPA mfo/lp2059272 [0]
[0] https://launchpad.net/~mfo/+archive/ubuntu/lp2059272
[ Regression Potential ]
* One patch changes *where* in the libvirt qemu driver's
shutdown path the worker thread pool is stopped/freed:
from _after_ releasing other data to _before_ doing so.
* The other patch (Focal-only) skips the update of the
QEMU domain status XML file during initialization if
libvirt is shutting down. (This is OK since the file
is not going to be used anyway in the current run as
it is shutting down, and it will be updated again in
the next run anyway.)
* Therefore, the potential for regression is limited to
the libvirt qemu driver's shutdown path, and would be
observed when stopping/restarting libvirtd.service.
* The behavior during normal operation is not affected.
[Other Info]
* In Focal, race windows exist if libvirtd shuts down
_after_ initialization and _during_ initialization
(which is unlikely in practice, but it's possible.)
Say, 'shutdown'on-runtime' and 'shutdown-on-init'.
* In Jammy, only 'shutdown-on-runtime' might happen,
due to the introduction of the '.stateShutdownWait'
driver callback (not available in Focal), which
indirectly prevents the 'shutdown-on-init' race
due to additional synchronization with locking.
* For 'shutdown-on-runtime': use upstream commit [1].
It's needed in Focal and Jammy (included in Mantic).
* For 'shutdown-on-init' (Focal-only), we should use a
downstream-only patch (with conservative behavior),
since upstream addressed this issue indirectly with
the '.stateShutdownWait' callbacks and other changes
(which are not SRU material, ~10 patches, redesign [2])
in 6.8.0.
[1] https://gitlab.com/libvirt/libvirt/-/commit/152770333449cd3b78b4f5a9f1148fc1f482d842
$ git describe --contains 152770333449cd3b78b4f5a9f1148fc1f482d842
v9.3.0-rc1~90
$ rmadison -a source libvirt | sed -n '/focal/,$p'
libvirt | 6.0.0-0ubuntu8 | focal | source
libvirt | 6.0.0-0ubuntu8.16 | focal-security | source
libvirt | 6.0.0-0ubuntu8.16 | focal-updates | source
libvirt | 6.0.0-0ubuntu8.17 | focal-proposed | source
libvirt | 8.0.0-1ubuntu7 | jammy | source
libvirt | 8.0.0-1ubuntu7.5 | jammy-security | source
libvirt | 8.0.0-1ubuntu7.8 | jammy-updates | source
libvirt | 9.6.0-1ubuntu1 | mantic | source
libvirt | 10.0.0-2ubuntu1 | noble | source
libvirt | 10.0.0-2ubuntu5 | noble-proposed | source
[2] https://listman.redhat.com/archives/libvir-list/2020-July/205291.html
[PATCH 00/10] resolve hangs/crashes on libvirtd shutdown
commit 94e45d1042e21e03a15ce993f90fbef626f1ae41
Author: Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
Date: Thu Jul 23 09:53:04 2020 +0300
rpc: finish all threads before exiting main loop
$ git describe --contains 94e45d1042e21e03a15ce993f90fbef626f1ae41
v6.8.0-rc1~279
[Original Description]
There's a race condition on libvirtd shutdown
that might cause the domain status XML file(s)
to lose the '<monitor path=...'> tag/field.
This causes an error on libvirtd startup, and
the domain is not listed/managed, despite it
is still running.
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ sudo systemctl restart libvirtd.service
$ journalctl -b -u libvirtd.service | tail
...
... libvirtd[2789]: internal error: no monitor path
... libvirtd[2789]: Failed to load config for domain 'test-vm'
$ virsh list
Id Name State
--------------------
$ virsh list --all
Id Name State
--------------------------
- test-vm shut off
$ pgrep -af qemu-system-x86_64 | cut -d, -f1
2638 /usr/bin/qemu-system-x86_64 -name guest=test-vm, |
|
| 2024-04-18 17:28:39 |
Robie Basak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2024-04-18 17:28:41 |
Robie Basak |
tags |
verification-done-jammy verification-failed-focal verification-needed |
verification-done-jammy verification-needed verification-needed-focal |
|
| 2024-04-24 13:19:47 |
Mauricio Faria de Oliveira |
tags |
verification-done-jammy verification-needed verification-needed-focal |
verification-done verification-done-focal verification-done-jammy |
|
| 2024-04-24 21:03:16 |
Fabio Augusto Miranda Martins |
bug |
|
|
added subscriber Fabio Augusto Miranda Martins |
| 2024-04-25 13:40:23 |
Launchpad Janitor |
libvirt (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
