| 2015-04-27 13:15:59 |
Stephen Finucane |
bug |
|
|
added bug |
| 2015-04-27 13:19:25 |
Stephen Finucane |
bug |
|
|
added subscriber Przemyslaw Czesnowicz |
| 2015-04-27 17:00:45 |
Przemyslaw Czesnowicz |
information type |
Public |
Public Security |
|
| 2015-04-28 08:47:09 |
Stephen Finucane |
description |
#1438226 reported that CPU pinning was broken in select versions of libvirt. Further investigation has highlighted issues with NUMA tuning in general on these versions. The same error messages seen with when configuring CPU pinning are seen when configuring NUMA tuning. The results from testing, mostly duplicated from the aforementioned bug report, are given below. Note that v1.2.10 is still being tested at this time.
This is somewhat related to #1422775 ("nova libvirt driver assumes qemu support for NUMA pinning").
---
# Testing Configuration
Testing was conducted in a container which provided a single-node, Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built with devstack). The yum-provided libvirt and its dependencies were removed and libvirt and libvirt-python were built and installed from source.
# Results
The results are as follows (currently incomplete):
versions status
-------- ------
1.2.9 ok
1.2.9.1 ok
1.2.9.2 fail
1.2.10 ???
1.2.11 ok
1.2.12 ok
v1.2.9.2 is broken by this (backported) patch:
https://www.redhat.com/archives/libvir-list/2014-November/msg00275.html
This can be seen as commit
e226772 (qemu: fix domain startup failing with 'strict' mode in numatune)
# Error logs
v1.2.9.2 produces the following exception:
Traceback (most recent call last):
File "/opt/stack/nova/nova/compute/manager.py", line 2301, in _build_resources
yield resources
File "/opt/stack/nova/nova/compute/manager.py", line 2171, in _build_and_run_instance
flavor=flavor)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2357, in spawn
block_device_info=block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4376, in _create_domain_and_network
power_on=power_on)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4307, in _create_domain
LOG.error(err)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 82, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4297, in _create_domain
domain.createWithFlags(launch_flags)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
six.reraise(c, e, tb)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
rv = meth(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: Failed to create controller cpu for group: No such file or directory |
#1438226 reported that CPU pinning was broken in select versions of libvirt. Further investigation has highlighted issues with NUMA tuning in general on these versions. On some versions of libvirt, the same error messages seen when configuring CPU pinning are seen when configuring NUMA tuning (e.g. with use of the 'hw:numa-nodes' flavor key). This would suggest that the entire NUMA tuning feature is broken on these versions, rather than just CPU pinning. The results from testing, mostly duplicated from the aforementioned bug report, are given below.
This is somewhat related to #1422775 ("nova libvirt driver assumes qemu support for NUMA pinning").
---
# Testing Configuration
Testing was conducted in a container which provided a single-node, Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built with devstack). The yum-provided libvirt and its dependencies were removed and libvirt and libvirt-python were built and installed from source.
# Results
The results are as follows (currently incomplete):
versions status
-------- ------
1.2.9 ok
1.2.9.1 ok
1.2.9.2 fail
1.2.10 ok
1.2.11 ok
1.2.12 ok
v1.2.9.2 is broken by this (backported) patch:
https://www.redhat.com/archives/libvir-list/2014-November/msg00275.html
This can be seen as commit
e226772 (qemu: fix domain startup failing with 'strict' mode in numatune)
# Error logs
v1.2.9.2 produces the following exception:
Traceback (most recent call last):
File "/opt/stack/nova/nova/compute/manager.py", line 2301, in _build_resources
yield resources
File "/opt/stack/nova/nova/compute/manager.py", line 2171, in _build_and_run_instance
flavor=flavor)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2357, in spawn
block_device_info=block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4376, in _create_domain_and_network
power_on=power_on)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4307, in _create_domain
LOG.error(err)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 82, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4297, in _create_domain
domain.createWithFlags(launch_flags)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
six.reraise(c, e, tb)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
rv = meth(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: Failed to create controller cpu for group: No such file or directory |
|
| 2015-04-28 08:51:35 |
Przemyslaw Czesnowicz |
information type |
Public Security |
Public |
|
| 2015-04-28 13:29:07 |
OpenStack Infra |
nova: status |
New |
In Progress |
|
| 2015-04-28 13:29:07 |
OpenStack Infra |
nova: assignee |
|
Stephen Finucane (stephen-finucane) |
|
| 2015-05-18 09:49:29 |
Stephen Finucane |
description |
#1438226 reported that CPU pinning was broken in select versions of libvirt. Further investigation has highlighted issues with NUMA tuning in general on these versions. On some versions of libvirt, the same error messages seen when configuring CPU pinning are seen when configuring NUMA tuning (e.g. with use of the 'hw:numa-nodes' flavor key). This would suggest that the entire NUMA tuning feature is broken on these versions, rather than just CPU pinning. The results from testing, mostly duplicated from the aforementioned bug report, are given below.
This is somewhat related to #1422775 ("nova libvirt driver assumes qemu support for NUMA pinning").
---
# Testing Configuration
Testing was conducted in a container which provided a single-node, Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built with devstack). The yum-provided libvirt and its dependencies were removed and libvirt and libvirt-python were built and installed from source.
# Results
The results are as follows (currently incomplete):
versions status
-------- ------
1.2.9 ok
1.2.9.1 ok
1.2.9.2 fail
1.2.10 ok
1.2.11 ok
1.2.12 ok
v1.2.9.2 is broken by this (backported) patch:
https://www.redhat.com/archives/libvir-list/2014-November/msg00275.html
This can be seen as commit
e226772 (qemu: fix domain startup failing with 'strict' mode in numatune)
# Error logs
v1.2.9.2 produces the following exception:
Traceback (most recent call last):
File "/opt/stack/nova/nova/compute/manager.py", line 2301, in _build_resources
yield resources
File "/opt/stack/nova/nova/compute/manager.py", line 2171, in _build_and_run_instance
flavor=flavor)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2357, in spawn
block_device_info=block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4376, in _create_domain_and_network
power_on=power_on)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4307, in _create_domain
LOG.error(err)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 82, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4297, in _create_domain
domain.createWithFlags(launch_flags)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
six.reraise(c, e, tb)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
rv = meth(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: Failed to create controller cpu for group: No such file or directory |
#1438226 reported that CPU pinning was broken in select versions of libvirt. Further investigation has highlighted issues with NUMA tuning in general on these versions. On some versions of libvirt, the same error messages seen when configuring CPU pinning are seen when configuring NUMA tuning (e.g. with use of the 'hw:numa-nodes' flavor key). This would suggest that the entire NUMA tuning feature is broken on these versions, rather than just CPU pinning. The results from testing, mostly duplicated from the aforementioned bug report, are given below.
This is somewhat related to #1422775 ("nova libvirt driver assumes qemu support for NUMA pinning").
---
# Testing Configuration
Testing was conducted in a container which provided a single-node, Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built with devstack). The yum-provided libvirt and its dependencies were removed and libvirt and libvirt-python were built and installed from source.
# Results
The results are as follows (currently incomplete):
versions status
-------- ------
1.2.9 ok
1.2.9.1 ok
1.2.9.2 fail
1.2.9.3 ok
1.2.10 ok
1.2.11 ok
1.2.12 ok
v1.2.9.2 is broken by this (backported) patch:
https://www.redhat.com/archives/libvir-list/2014-November/msg00275.html
This can be seen as commit
e226772 (qemu: fix domain startup failing with 'strict' mode in numatune)
# Error logs
v1.2.9.2 produces the following exception:
Traceback (most recent call last):
File "/opt/stack/nova/nova/compute/manager.py", line 2301, in _build_resources
yield resources
File "/opt/stack/nova/nova/compute/manager.py", line 2171, in _build_and_run_instance
flavor=flavor)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2357, in spawn
block_device_info=block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4376, in _create_domain_and_network
power_on=power_on)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4307, in _create_domain
LOG.error(err)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 82, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4297, in _create_domain
domain.createWithFlags(launch_flags)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
six.reraise(c, e, tb)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
rv = meth(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: Failed to create controller cpu for group: No such file or directory |
|
| 2015-06-04 14:53:29 |
OpenStack Infra |
nova: assignee |
Stephen Finucane (stephen-finucane) |
Matt Riedemann (mriedem) |
|
| 2015-06-05 00:12:55 |
OpenStack Infra |
nova: status |
In Progress |
Fix Committed |
|
| 2015-06-24 12:18:18 |
Thierry Carrez |
nova: status |
Fix Committed |
Fix Released |
|
| 2015-06-24 12:18:18 |
Thierry Carrez |
nova: milestone |
|
liberty-1 |
|
| 2015-10-15 08:54:13 |
Thierry Carrez |
nova: milestone |
liberty-1 |
12.0.0 |
|
