| 2018-08-29 14:22:02 |
Silvio Moioli |
bug |
|
|
added bug |
| 2018-08-29 14:39:22 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Bionic |
|
| 2018-08-29 14:39:22 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Bionic) |
|
| 2018-08-29 14:39:28 |
Christian Ehrhardt |
libvirt (Ubuntu): status |
New |
Fix Released |
|
| 2018-08-29 14:48:27 |
Christian Ehrhardt |
description |
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2], a tool to automate infrastructure deployments, together with terraform-provider-libvirt[3], a Terraform provider (plugin) that allow interaction with libvirt.
By default, this setup will issue several libvirt API calls concurrently in an hard-to-predict order, as demonstrated by logs that I am attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during iteration
According to analysis of a similar problem by Red Hat in an OpenStack scenario[4], this is has been fixed in upstream libvirt via commit 4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package shipping with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5] https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago) |
[Impact]
* An explanation of the effects of the bug on users and
* justification for backporting the fix to the stable release.
* In addition, it is helpful, but not required, to include an
explanation of how the upload fixes this bug.
[Test Case]
* detailed instructions how to reproduce the bug
* these should allow someone who is not familiar with the affected
package to reproduce the bug and verify that the updated package fixes
the problem.
[Regression Potential]
* discussion of how regressions are most likely to manifest as a result of this change.
* It is assumed that any SRU candidate patch is well-tested before
upload and has a low overall risk of regression, but it's important
to make the effort to think about what ''could'' happen in the
event of a regression.
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
[Other Info]
* Anything else you think is useful to include
* Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
* and address these questions in advance
----
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2], a tool to automate infrastructure deployments, together with terraform-provider-libvirt[3], a Terraform provider (plugin) that allow interaction with libvirt.
By default, this setup will issue several libvirt API calls concurrently in an hard-to-predict order, as demonstrated by logs that I am attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during iteration
According to analysis of a similar problem by Red Hat in an OpenStack scenario[4], this is has been fixed in upstream libvirt via commit 4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package shipping with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5] https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago) |
|
| 2018-08-29 14:48:47 |
Christian Ehrhardt |
libvirt (Ubuntu Bionic): status |
New |
Incomplete |
|
| 2018-09-04 07:05:09 |
Silvio Moioli |
bug watch added |
|
https://bugzilla.redhat.com/show_bug.cgi?id=1581364 |
|
| 2018-09-04 08:00:08 |
Christian Ehrhardt |
libvirt (Ubuntu Bionic): status |
Incomplete |
Triaged |
|
| 2018-09-04 08:00:10 |
Christian Ehrhardt |
libvirt (Ubuntu Bionic): assignee |
|
Christian Ehrhardt (paelzer) |
|
| 2018-09-04 08:01:12 |
Christian Ehrhardt |
bug |
|
|
added subscriber Ubuntu Server |
| 2018-09-04 10:05:32 |
Christian Ehrhardt |
description |
[Impact]
* An explanation of the effects of the bug on users and
* justification for backporting the fix to the stable release.
* In addition, it is helpful, but not required, to include an
explanation of how the upload fixes this bug.
[Test Case]
* detailed instructions how to reproduce the bug
* these should allow someone who is not familiar with the affected
package to reproduce the bug and verify that the updated package fixes
the problem.
[Regression Potential]
* discussion of how regressions are most likely to manifest as a result of this change.
* It is assumed that any SRU candidate patch is well-tested before
upload and has a low overall risk of regression, but it's important
to make the effort to think about what ''could'' happen in the
event of a regression.
* This both shows the SRU team that the risks have been considered,
and provides guidance to testers in regression-testing the SRU.
[Other Info]
* Anything else you think is useful to include
* Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
* and address these questions in advance
----
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2], a tool to automate infrastructure deployments, together with terraform-provider-libvirt[3], a Terraform provider (plugin) that allow interaction with libvirt.
By default, this setup will issue several libvirt API calls concurrently in an hard-to-predict order, as demonstrated by logs that I am attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during iteration
According to analysis of a similar problem by Red Hat in an OpenStack scenario[4], this is has been fixed in upstream libvirt via commit 4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package shipping with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5] https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago) |
[Impact]
* Libvirt does an overzealous check on concurrent hash usage which breaks
some automation like for example Terraform. Upstream moved the need to
lock up the stack where applicable and dropped the checks as they were
superfluous.
* Backport the upstream change dropping the extra checks on hash usage
[Test Case]
* 1.Start a guest:
$ virsh start test1
* 2.Do 'virsh list' in a loop:
$ for i in {1..1000}; do virsh list; done
* 3.Open another terminal, do 'virsh domstats' in a loop:
$ for i in {1..1000}; do virsh domstats; done
* 4.Check the libvirtd.log:
$ cat /var/log/libvirt/libvirtd.log | grep -i "Hash operation not allowed during"
2018-09-04 06:57:00.761+0000: 28687: error : virHashForEach:597 : Hash operation not allowed during iteration
From: https://bugzilla.redhat.com/show_bug.cgi?id=1581364#c7
With the aforementioned patch, this error is not produced any more.
[Regression Potential]
* If the assumption that all remaining callers are safe isn't correct
there could be code using invalid hashes accessing guest information.
This needs thorough tests and regression checks as well as an extra
review if it can be applied to the libvirt 4.0 we have in Bionic.
[Other Info]
* n/a
----
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2], a tool to automate infrastructure deployments, together with terraform-provider-libvirt[3], a Terraform provider (plugin) that allow interaction with libvirt.
By default, this setup will issue several libvirt API calls concurrently in an hard-to-predict order, as demonstrated by logs that I am attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during iteration
According to analysis of a similar problem by Red Hat in an OpenStack scenario[4], this is has been fixed in upstream libvirt via commit 4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package shipping with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5] https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago) |
|
| 2018-09-04 15:42:08 |
Christian Ehrhardt |
libvirt (Ubuntu Bionic): importance |
Undecided |
High |
|
| 2018-09-04 15:51:01 |
GOVINDA TATTI |
bug |
|
|
added subscriber GOVINDA TATTI |
| 2018-09-04 18:15:53 |
Andrew Cloke |
bug |
|
|
added subscriber Andrew Cloke |
| 2018-09-05 01:16:18 |
GOVINDA TATTI |
bug |
|
|
added subscriber Anish Gupta |
| 2018-09-05 01:16:46 |
GOVINDA TATTI |
bug |
|
|
added subscriber Raaghav Hebbar |
| 2018-09-05 07:41:11 |
Christian Ehrhardt |
description |
[Impact]
* Libvirt does an overzealous check on concurrent hash usage which breaks
some automation like for example Terraform. Upstream moved the need to
lock up the stack where applicable and dropped the checks as they were
superfluous.
* Backport the upstream change dropping the extra checks on hash usage
[Test Case]
* 1.Start a guest:
$ virsh start test1
* 2.Do 'virsh list' in a loop:
$ for i in {1..1000}; do virsh list; done
* 3.Open another terminal, do 'virsh domstats' in a loop:
$ for i in {1..1000}; do virsh domstats; done
* 4.Check the libvirtd.log:
$ cat /var/log/libvirt/libvirtd.log | grep -i "Hash operation not allowed during"
2018-09-04 06:57:00.761+0000: 28687: error : virHashForEach:597 : Hash operation not allowed during iteration
From: https://bugzilla.redhat.com/show_bug.cgi?id=1581364#c7
With the aforementioned patch, this error is not produced any more.
[Regression Potential]
* If the assumption that all remaining callers are safe isn't correct
there could be code using invalid hashes accessing guest information.
This needs thorough tests and regression checks as well as an extra
review if it can be applied to the libvirt 4.0 we have in Bionic.
[Other Info]
* n/a
----
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2], a tool to automate infrastructure deployments, together with terraform-provider-libvirt[3], a Terraform provider (plugin) that allow interaction with libvirt.
By default, this setup will issue several libvirt API calls concurrently in an hard-to-predict order, as demonstrated by logs that I am attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during iteration
According to analysis of a similar problem by Red Hat in an OpenStack scenario[4], this is has been fixed in upstream libvirt via commit 4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package shipping with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5] https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago) |
[Impact]
* Libvirt does an overzealous check on concurrent hash usage which breaks
some automation like for example Terraform. Upstream moved the need to
lock up the stack where applicable and dropped the checks as they were
superfluous.
* Backport the upstream change dropping the extra checks on hash usage
[Test Case]
* 1.Start (at least) one guest:
$ virsh start test1
* 2.Do 'virsh list' in a loop:
$ for i in {1..1000}; do virsh list; done
* 3.Open another terminal, do 'virsh domstats' in a loop:
$ for i in {1..1000}; do virsh domstats; done
# It is a race after all, so more guests and more of the loops above
concurrently help to raise the chance. With 4 guests and 4 concurrent
of each loops above I hit it 4 times of which 3 where almost at the
same moment.
* 4.Check the libvirtd.log:
$ cat /var/log/libvirt/libvirtd.log | grep -i "Hash operation not allowed during"
2018-09-04 06:57:00.761+0000: 28687: error : virHashForEach:597 : Hash operation not allowed during iteration
From: https://bugzilla.redhat.com/show_bug.cgi?id=1581364#c7
With the aforementioned patch, this error is not produced any more.
[Regression Potential]
* If the assumption that all remaining callers are safe isn't correct
there could be code using invalid hashes accessing guest information.
This needs thorough tests and regression checks as well as an extra
review if it can be applied to the libvirt 4.0 we have in Bionic.
[Other Info]
* n/a
----
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2], a tool to automate infrastructure deployments, together with terraform-provider-libvirt[3], a Terraform provider (plugin) that allow interaction with libvirt.
By default, this setup will issue several libvirt API calls concurrently in an hard-to-predict order, as demonstrated by logs that I am attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during iteration
According to analysis of a similar problem by Red Hat in an OpenStack scenario[4], this is has been fixed in upstream libvirt via commit 4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package shipping with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5] https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago) |
|
| 2018-09-05 08:12:31 |
Christian Ehrhardt |
libvirt (Ubuntu Bionic): status |
Triaged |
In Progress |
|
| 2018-09-05 08:20:17 |
Christian Ehrhardt |
description |
[Impact]
* Libvirt does an overzealous check on concurrent hash usage which breaks
some automation like for example Terraform. Upstream moved the need to
lock up the stack where applicable and dropped the checks as they were
superfluous.
* Backport the upstream change dropping the extra checks on hash usage
[Test Case]
* 1.Start (at least) one guest:
$ virsh start test1
* 2.Do 'virsh list' in a loop:
$ for i in {1..1000}; do virsh list; done
* 3.Open another terminal, do 'virsh domstats' in a loop:
$ for i in {1..1000}; do virsh domstats; done
# It is a race after all, so more guests and more of the loops above
concurrently help to raise the chance. With 4 guests and 4 concurrent
of each loops above I hit it 4 times of which 3 where almost at the
same moment.
* 4.Check the libvirtd.log:
$ cat /var/log/libvirt/libvirtd.log | grep -i "Hash operation not allowed during"
2018-09-04 06:57:00.761+0000: 28687: error : virHashForEach:597 : Hash operation not allowed during iteration
From: https://bugzilla.redhat.com/show_bug.cgi?id=1581364#c7
With the aforementioned patch, this error is not produced any more.
[Regression Potential]
* If the assumption that all remaining callers are safe isn't correct
there could be code using invalid hashes accessing guest information.
This needs thorough tests and regression checks as well as an extra
review if it can be applied to the libvirt 4.0 we have in Bionic.
[Other Info]
* n/a
----
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2], a tool to automate infrastructure deployments, together with terraform-provider-libvirt[3], a Terraform provider (plugin) that allow interaction with libvirt.
By default, this setup will issue several libvirt API calls concurrently in an hard-to-predict order, as demonstrated by logs that I am attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during iteration
According to analysis of a similar problem by Red Hat in an OpenStack scenario[4], this is has been fixed in upstream libvirt via commit 4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package shipping with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5] https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago) |
[Impact]
* Libvirt does an overzealous check on concurrent hash usage which breaks
some automation like for example Terraform. Upstream moved the need to
lock up the stack where applicable and dropped the checks as they were
superfluous.
* Backport the upstream change dropping the extra checks on hash usage
[Test Case]
* 1.Start (at least) one guest:
$ virsh start test1
* 2.Do 'virsh list' in a loop:
$ for i in {1..1000}; do virsh list; done
* 3.Open another terminal, do 'virsh domstats' in a loop:
$ for i in {1..1000}; do virsh domstats; done
# It is a race after all, so more guests and more of the loops above
concurrently help to raise the chance. With 4 guests and 4 concurrent
of each loops above I hit it 4 times of which 3 where almost at the
same moment.
* 4.Check the libvirtd.log:
$ cat /var/log/libvirt/libvirtd.log | grep -i "Hash operation not allowed during"
2018-09-04 06:57:00.761+0000: 28687: error : virHashForEach:597 : Hash operation not allowed during iteration
From: https://bugzilla.redhat.com/show_bug.cgi?id=1581364#c7
With the aforementioned patch, this error is not produced any more.
[Regression Potential]
* If the assumption that all remaining callers are safe isn't correct
there could be code using invalid hashes accessing guest information.
This needs thorough tests and regression checks as well as an extra
review if it can be applied to the libvirt 4.0 we have in Bionic.
This was done by me and also upstream did the same.
Quote from the patch: "Most important uses seem to be covered. Callers
have now a greater responsability, notably the ability to execute some
operations while iterating were reliably forbidden before are now
accepted."
It is important to see that the internal locking was inherently racy
(boolean) so we are replaceing a broken lock-ckeck with no check as
upper layers were supposed to do it correctly. That is true for what is
in the archive but there could be out-of archive things not doing so
calling directly into the python api bindings for example.
Never the less those are today just as affected by the locking being
broken, so those do not "loose" a lot.
All major exploiters we know of are good - the rest is a regression
risk we are willing to take in favor of fixing all the valid use cases
being affected by the broken lock handling.
[Other Info]
* n/a
----
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2], a tool to automate infrastructure deployments, together with terraform-provider-libvirt[3], a Terraform provider (plugin) that allow interaction with libvirt.
By default, this setup will issue several libvirt API calls concurrently in an hard-to-predict order, as demonstrated by logs that I am attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during iteration
According to analysis of a similar problem by Red Hat in an OpenStack scenario[4], this is has been fixed in upstream libvirt via commit 4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package shipping with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5] https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago) |
|
| 2018-09-05 09:42:56 |
Christian Ehrhardt |
description |
[Impact]
* Libvirt does an overzealous check on concurrent hash usage which breaks
some automation like for example Terraform. Upstream moved the need to
lock up the stack where applicable and dropped the checks as they were
superfluous.
* Backport the upstream change dropping the extra checks on hash usage
[Test Case]
* 1.Start (at least) one guest:
$ virsh start test1
* 2.Do 'virsh list' in a loop:
$ for i in {1..1000}; do virsh list; done
* 3.Open another terminal, do 'virsh domstats' in a loop:
$ for i in {1..1000}; do virsh domstats; done
# It is a race after all, so more guests and more of the loops above
concurrently help to raise the chance. With 4 guests and 4 concurrent
of each loops above I hit it 4 times of which 3 where almost at the
same moment.
* 4.Check the libvirtd.log:
$ cat /var/log/libvirt/libvirtd.log | grep -i "Hash operation not allowed during"
2018-09-04 06:57:00.761+0000: 28687: error : virHashForEach:597 : Hash operation not allowed during iteration
From: https://bugzilla.redhat.com/show_bug.cgi?id=1581364#c7
With the aforementioned patch, this error is not produced any more.
[Regression Potential]
* If the assumption that all remaining callers are safe isn't correct
there could be code using invalid hashes accessing guest information.
This needs thorough tests and regression checks as well as an extra
review if it can be applied to the libvirt 4.0 we have in Bionic.
This was done by me and also upstream did the same.
Quote from the patch: "Most important uses seem to be covered. Callers
have now a greater responsability, notably the ability to execute some
operations while iterating were reliably forbidden before are now
accepted."
It is important to see that the internal locking was inherently racy
(boolean) so we are replaceing a broken lock-ckeck with no check as
upper layers were supposed to do it correctly. That is true for what is
in the archive but there could be out-of archive things not doing so
calling directly into the python api bindings for example.
Never the less those are today just as affected by the locking being
broken, so those do not "loose" a lot.
All major exploiters we know of are good - the rest is a regression
risk we are willing to take in favor of fixing all the valid use cases
being affected by the broken lock handling.
[Other Info]
* n/a
----
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2], a tool to automate infrastructure deployments, together with terraform-provider-libvirt[3], a Terraform provider (plugin) that allow interaction with libvirt.
By default, this setup will issue several libvirt API calls concurrently in an hard-to-predict order, as demonstrated by logs that I am attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during iteration
According to analysis of a similar problem by Red Hat in an OpenStack scenario[4], this is has been fixed in upstream libvirt via commit 4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package shipping with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5] https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago) |
[Impact]
* Libvirt does an overzealous check on concurrent hash usage which breaks
some automation like for example Terraform. Upstream moved the need to
lock up the stack where applicable and dropped the checks as they were
superfluous.
* Backport the upstream change dropping the extra checks on hash usage
[Test Case]
* Run the attached script test-lp1789659.sh
(based on ideas of https://bugzilla.redhat.com/show_bug.cgi?id=1581364#c7)
With the aforementioned patch, this error is not produced any more.
[Regression Potential]
* If the assumption that all remaining callers are safe isn't correct
there could be code using invalid hashes accessing guest information.
This needs thorough tests and regression checks as well as an extra
review if it can be applied to the libvirt 4.0 we have in Bionic.
This was done by me and also upstream did the same.
Quote from the patch: "Most important uses seem to be covered. Callers
have now a greater responsability, notably the ability to execute some
operations while iterating were reliably forbidden before are now
accepted."
It is important to see that the internal locking was inherently racy
(boolean) so we are replaceing a broken lock-ckeck with no check as
upper layers were supposed to do it correctly. That is true for what is
in the archive but there could be out-of archive things not doing so
calling directly into the python api bindings for example.
Never the less those are today just as affected by the locking being
broken, so those do not "loose" a lot.
All major exploiters we know of are good - the rest is a regression
risk we are willing to take in favor of fixing all the valid use cases
being affected by the broken lock handling.
[Other Info]
* n/a
----
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2], a tool to automate infrastructure deployments, together with terraform-provider-libvirt[3], a Terraform provider (plugin) that allow interaction with libvirt.
By default, this setup will issue several libvirt API calls concurrently in an hard-to-predict order, as demonstrated by logs that I am attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during iteration
According to analysis of a similar problem by Red Hat in an OpenStack scenario[4], this is has been fixed in upstream libvirt via commit 4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package shipping with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5] https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago) |
|
| 2018-09-05 09:46:32 |
Christian Ehrhardt |
attachment added |
|
Helper to test and verify the issue https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1789659/+attachment/5184890/+files/test-lp1789659.sh |
|
| 2018-09-05 09:54:03 |
Christian Ehrhardt |
attachment removed |
Helper to test and verify the issue https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1789659/+attachment/5184890/+files/test-lp1789659.sh |
|
|
| 2018-09-05 09:54:35 |
Christian Ehrhardt |
attachment added |
|
Helper to test and verify the issue https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1789659/+attachment/5184891/+files/test-lp1789659.sh |
|
| 2018-09-06 15:57:32 |
Andy Whitcroft |
libvirt (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2018-09-06 15:57:34 |
Andy Whitcroft |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2018-09-06 15:57:37 |
Andy Whitcroft |
bug |
|
|
added subscriber SRU Verification |
| 2018-09-06 15:57:40 |
Andy Whitcroft |
tags |
amd64 apport-bug bionic |
amd64 apport-bug bionic verification-needed verification-needed-bionic |
|
| 2018-09-10 10:22:59 |
Christian Ehrhardt |
tags |
amd64 apport-bug bionic verification-needed verification-needed-bionic |
amd64 apport-bug bionic verification-done verification-done-bionic |
|
| 2018-09-13 08:00:33 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2018-09-13 08:10:33 |
Launchpad Janitor |
libvirt (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2018-09-27 20:12:53 |
Michael Craft |
bug |
|
|
added subscriber Michael Craft |
| 2018-09-28 06:44:25 |
Christian Ehrhardt |
bug task added |
|
cloud-archive |
|
| 2018-09-28 06:44:31 |
Christian Ehrhardt |
cloud-archive: status |
New |
Incomplete |
|
| 2018-09-28 14:25:04 |
Michael Craft |
cloud-archive: status |
Incomplete |
New |
|
| 2018-09-28 14:53:29 |
Adam Vinsh |
bug |
|
|
added subscriber Adam Vinsh |
| 2020-06-23 09:22:06 |
Chris MacNaughton |
nominated for series |
|
cloud-archive/pike |
|
| 2020-06-23 09:22:06 |
Chris MacNaughton |
bug task added |
|
cloud-archive/pike |
|
| 2020-06-23 10:44:19 |
Chris MacNaughton |
cloud-archive/pike: status |
New |
Confirmed |
|
| 2020-07-22 14:22:00 |
Corey Bryant |
cloud-archive: status |
New |
Fix Released |
|
| 2020-07-22 14:22:05 |
Corey Bryant |
cloud-archive/pike: status |
Confirmed |
Won't Fix |
|
