Icehouse on CentOS - launch instance failed with nova

[Boon Lee]

Hi there,

I am unable to launch any instance after recent yum upgrade(10 April). I have installed icehouse on CentOS-6.5.

Dashboard:

Message: Virtual Interface creation failed
Code: 500
Details
File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 296, in decorated_function return function(self, context, *args, **kwargs) File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2073, in run_instance do_run_instance() File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner return f(*args, **kwargs) File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2072, in do_run_instance legacy_bdm_in_spec) File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1205, in _run_instance notify("error", fault=e) # notify that build failed File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1189, in _run_instance instance, image_meta, legacy_bdm_in_spec) File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1333, in _build_instance network_info.wait(do_raise=False) File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1309, in _build_instance set_access_ip=set_access_ip) File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 399, in decorated_function return function(self, context, *args, **kwargs) File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1721, in _spawn LOG.exception(_('Instance failed to spawn'), instance=instance) File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1718, in _spawn block_device_info) File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2251, in spawn block_device_info) File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3654, in _create_domain_and_network raise exception.VirtualInterfaceCreateException()

api.log:

2014-04-11 10:38:36.383 6386 ERROR nova.api.openstack.wsgi [-] Exception handling resource: multi() got an unexpected keyword argument 'body'
2014-04-11 10:38:36.383 6386 TRACE nova.api.openstack.wsgi Traceback (most recent call last):
2014-04-11 10:38:36.383 6386 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 983, in _process_stack
2014-04-11 10:38:36.383 6386 TRACE nova.api.openstack.wsgi action_result = self.dispatch(meth, request, action_args)
2014-04-11 10:38:36.383 6386 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1070, in dispatch
2014-04-11 10:38:36.383 6386 TRACE nova.api.openstack.wsgi return method(req=request, **action_args)
2014-04-11 10:38:36.383 6386 TRACE nova.api.openstack.wsgi TypeError: multi() got an unexpected keyword argument 'body'

compute.log:

2014-04-11 10:38:32.869 17049 ERROR nova.compute.manager [req-7c8f2ee8-f957-4f1e-b188-2f136fcb7cdb 1cfbc71fddb848668295a2bb1895dfda 9f03371fd9fb4ef8ba5760a3ae1021ba] [instance: 45a6f90d-3c76-4740-bacb-f3ea73c1b78c] Instance failed to spawn
2014-04-11 10:38:32.869 17049 TRACE nova.compute.manager [instance: 45a6f90d-3c76-4740-bacb-f3ea73c1b78c] Traceback (most recent call last):
2014-04-11 10:38:32.869 17049 TRACE nova.compute.manager [instance: 45a6f90d-3c76-4740-bacb-f3ea73c1b78c] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1718, in _spawn
2014-04-11 10:38:32.869 17049 TRACE nova.compute.manager [instance: 45a6f90d-3c76-4740-bacb-f3ea73c1b78c] block_device_info)
2014-04-11 10:38:32.869 17049 TRACE nova.compute.manager [instance: 45a6f90d-3c76-4740-bacb-f3ea73c1b78c] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2251, in spawn
2014-04-11 10:38:32.869 17049 TRACE nova.compute.manager [instance: 45a6f90d-3c76-4740-bacb-f3ea73c1b78c] block_device_info)
2014-04-11 10:38:32.869 17049 TRACE nova.compute.manager [instance: 45a6f90d-3c76-4740-bacb-f3ea73c1b78c] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3654, in _create_domain_and_network
2014-04-11 10:38:32.869 17049 TRACE nova.compute.manager [instance: 45a6f90d-3c76-4740-bacb-f3ea73c1b78c] raise exception.VirtualInterfaceCreateException()
2014-04-11 10:38:32.869 17049 TRACE nova.compute.manager [instance: 45a6f90d-3c76-4740-bacb-f3ea73c1b78c] VirtualInterfaceCreateException: Virtual Interface creation failed

Regards,
Boon Lee

[root@controller nova]# rpm -qa | grep openstack
openstack-nova-console-2014.1-0.12.rc1.el6.noarch
openstack-nova-scheduler-2014.1-0.12.rc1.el6.noarch
openstack-utils-2013.2-3.el6.noarch
openstack-selinux-0.1.3-2.el6ost.noarch
openstack-dashboard-2014.1-0.14.rc2.el6.noarch
openstack-nova-conductor-2014.1-0.12.rc1.el6.noarch
openstack-glance-2014.1-0.5.rc1.el6.noarch
openstack-nova-common-2014.1-0.12.rc1.el6.noarch
openstack-neutron-2014.1-0.13.rc1.el6.noarch
openstack-neutron-openvswitch-2014.1-0.13.rc1.el6.noarch
openstack-nova-api-2014.1-0.12.rc1.el6.noarch
openstack-keystone-2014.1-0.7.rc1.el6.noarch
openstack-nova-cert-2014.1-0.12.rc1.el6.noarch
python-django-openstack-auth-1.1.5-1.el6.noarch
openstack-nova-novncproxy-2014.1-0.12.rc1.el6.noarch

-----------------------------------
Built: 2014-04-10T22:07:30 00:00
git SHA: 289c27051a4cccca84ce52f872013b77b2411c59
URL: http://docs.openstack.org/trunk/install-guide/install/yum/content/launch-instance.html
source File: file:/home/jenkins/workspace/openstack-install-deploy-guide-fedora/doc/install-guide/ch_launch-instance.xml
xml:id: launch-instance

[Tom Fifield (fifieldt)]

Boon,

This is very likely a bug either in packaging or in the code, rather than a problem in the manuals. I'm going to send this to the nova project.

[Boon Lee (eupub)]

Hi Tom,

I fixed the problem by updating neutron.conf at controller to include neutron-nova interactions config which is missing at Open vSwitch OVS plugin - configure controller node

# openstack-config --set /etc/neutron/neutron.conf DEFAULT \
  notify_nova_on_port_status_changes True
# openstack-config --set /etc/neutron/neutron.conf DEFAULT \
  notify_nova_on_port_data_changes True
# openstack-config --set /etc/neutron/neutron.conf DEFAULT \
  nova_url http://controller:8774/v2
# openstack-config --set /etc/neutron/neutron.conf DEFAULT \
  nova_admin_username nova
# openstack-config --set /etc/neutron/neutron.conf DEFAULT \
  nova_admin_tenant_id SERVICE_TENANT_ID
# openstack-config --set /etc/neutron/neutron.conf DEFAULT \
  nova_admin_password NOVA_PASS
# openstack-config --set /etc/neutron/neutron.conf DEFAULT \
  nova_admin_auth_url http://controller:35357/v2.0

After adding these, the problem went away. FYI.

Regards,
Boon Lee

[Trinath Somanchi (trinath-somanchi)]

The same problem is fixed when I create br-eth1 in my controller node in ovs.

[Seb Astian (callmeseb)]

Solution #2 didn't work for me but workaround from RDO worked: http://openstack.redhat.com/Workarounds

Edit /etc/nova/nova.conf
vif_plugging_is_fatal: false
vif_plugging_timeout: 0

...but I'm struggling to understand why?

Rgds,
S

[Matt Riedemann (mriedem)]

This is setting making vif_plugging_is_fatal=False makes it work:

http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3716

Before the timeout / vif plugging event code, any cleanup failures are wrapped and swallowed:

http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3707

With the eventlet timeout block, if the domain isn't running and libvirt pukes on that, the libvirtError isn't swallowed as before and you get the libvirtError up the stack rather than the VirtualInterfaceCreateException.

FWIW I'm setting vif_plugging_is_fatal=False in my deployment but leaving the default 5 minute timeout. This way if it does fail, the reliability is no worse off than it was between nova and neutron in Havana.

I'll push a fix to wrap the libvirtError in that timeout case though.

[Matt Riedemann (mriedem)]

Sorry, I went on a bit of a tangent with the eventlet timeout and the libvirt error on a non-running domain, I'm seeing that in a migration flow when finish_migration is called, the bug reported here is for spawn.

This is probably what you're seeing on spawn when it times out:

http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3636

And changing the flag makes it a no-op so you don't get the exception.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/94476

[Matt Riedemann (mriedem)]

This might be useful if you're on Icehouse: https://review.openstack.org/#/c/89870/

[Matt Riedemann (mriedem)]

I've moved the resize/migrate from stopped instance issue with the neutron timeout to new bug 1321872.

[Matt Riedemann (mriedem)]

Marking this bug as invalid since it's a configuration failure, nova/neutron have to be configured correctly for event callbacks during spawn with the libvirt driver.