Juniper Openstack

On Icehouse, nova interface-attach cmd fails but port does get bound to the VM

Bug #1377419 reported by Vedamurthy Joshi
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R1.1
Incomplete
High
Hampapur Ajay
Juniper Openstack r1.20
Trunk
Fix Committed
High
Atul Moghe

Bug Description

R1.10 Build 44 (Ubuntu Icehouse multi-node setup)

root@nodec22:~# neutron net-list
+--------------------------------------+-------------------------+-----------------------------------------------------+
| id | name | subnets |
+--------------------------------------+-------------------------+-----------------------------------------------------+
| dd32ac7e-527c-486f-addb-4118a65ed43a | default-virtual-network | |
| eae8c123-451b-43cb-948d-d37ec2c827d5 | vn1-31619381 | 0bc7104c-1a59-47ec-ad19-6fbc6dbab0b5 51.6.68.0/24 |
| 1651d8d2-b9ec-47dd-9e72-b02c05725b9c | vn1-04559766 | 125d5498-1b78-445b-ae91-fa714f053efd 41.202.19.0/24 |
| d9efaf6b-d60c-4b48-b5e7-89ce4b29647c | ip-fabric | |
| 2993df96-842a-4f5f-b7f8-93fa36e37a04 | __link_local__ | |
+--------------------------------------+-------------------------+-----------------------------------------------------+
root@nodec22:~# neutron port-create 1651d8d2-b9ec-47dd-9e72-b02c05725b9c
Created a new port:
+-----------------+------------------------------------------------------------------------------------+
| Field | Value |
+-----------------+------------------------------------------------------------------------------------+
| admin_state_up | True |
| device_id | |
| device_owner | |
| fixed_ips | {"subnet_id": "125d5498-1b78-445b-ae91-fa714f053efd", "ip_address": "41.202.19.2"} |
| id | 02abb855-cb06-4825-9ae8-b051a417b7ce |
| mac_address | 02:02:ab:b8:55:cb |
| name | 02abb855-cb06-4825-9ae8-b051a417b7ce |
| network_id | 1651d8d2-b9ec-47dd-9e72-b02c05725b9c |
| security_groups | 5f169107-09ce-4f35-8db0-9d6452ed004b |
| status | DOWN |
| tenant_id | 812d66ad27bd40d1bf81761afd59eeb9 |
+-----------------+------------------------------------------------------------------------------------+
root@nodec22:~# root@nodec22:~# nova interface-attach --port-id 02abb855-cb06-4825-9ae8-b051a417b7ce 7c9396c3-0748-418a-a5bf-f8241c8df274
ERROR: Failed to attach interface (HTTP 500) (Request-ID: req-59399617-ab6c-4d7e-8aa1-87dd44d1b9d9) <<<<<<<<<<<
root@nodec22:~# neutron port-list |grep 02abb
| 02abb855-cb06-4825-9ae8-b051a417b7ce | 02abb855-cb06-4825-9ae8-b051a417b7ce | 02:02:ab:b8:55:cb | {"subnet_id": "125d5498-1b78-445b-ae91-fa714f053efd", "ip_address": "41.202.19.2"} |
root@nodec22:~# neutron port-show 02abb855-cb06-4825-9ae8-b051a417b7ce --device_id
+-----------------+------------------------------------------------------------------------------------+
| Field | Value |
+-----------------+------------------------------------------------------------------------------------+
| admin_state_up | True |
| device_id | 7c9396c3-0748-418a-a5bf-f8241c8df274 |
| device_owner | compute:nova |
| fixed_ips | {"subnet_id": "125d5498-1b78-445b-ae91-fa714f053efd", "ip_address": "41.202.19.2"} |
| id | 02abb855-cb06-4825-9ae8-b051a417b7ce |
| mac_address | 02:02:ab:b8:55:cb |
| name | 02abb855-cb06-4825-9ae8-b051a417b7ce |
| network_id | 1651d8d2-b9ec-47dd-9e72-b02c05725b9c |
| security_groups | 5f169107-09ce-4f35-8db0-9d6452ed004b |
| status | ACTIVE |
| tenant_id | 812d66ad27bd40d1bf81761afd59eeb9 |
+-----------------+------------------------------------------------------------------------------------+
root@nodec22:~#

nova-api.log:
==============

2014-10-04 04:25:51.641 5579 ERROR nova.api.openstack.compute.contrib.attach_interfaces [req-59399617-ab6c-4d7e-8aa1-87dd44d1b9d9 e54aa038ff494a9094a1c00ee0161bd4 812d66ad27bd40d1bf81761afd59eeb9] <nova.objects.instance.Instance object at 0x450af10>
Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))

  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 396, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4347, in attach_interface
    self.driver.attach_interface(instance, image_meta, network_info[0])

  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1422, in attach_interface
    raise exception.InterfaceAttachFailed(instance)

InterfaceAttachFailed: <nova.objects.instance.Instance object at 0x450af10>
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces Traceback (most recent call last):
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/attach_interfaces.py", line 106, in create
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces instance, network_id, port_id, req_ip)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 201, in wrapped
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces return func(self, context, target, *args, **kwargs)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2864, in attach_interface
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces requested_ip=requested_ip)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 322, in attach_interface
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces port_id=port_id, requested_ip=requested_ip)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces wait_for_reply=True, timeout=timeout)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces timeout=timeout)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces raise result
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces InterfaceAttachFailed_Remote: <nova.objects.instance.Instance object at 0x450af10>
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces Traceback (most recent call last):
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/attach_interfaces.py", line 106, in create2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces instance, network_id, port_id, req_ip)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/c
ompute/api.py", line 201, in wrapped
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces return func(self, context, target, *args, **kwargs)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2864, in attach_interface
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces requested_ip=requested_ip)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 322, in attach_interface
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces port_id=port_id, requested_ip=requested_ip)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces wait_for_reply=True, timeout=timeout)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces timeout=timeout)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces raise result
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces InterfaceAttachFailed_Remote: <nova.objects.instance.Instance object at 0x450af10>
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces Traceback (most recent call last):
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces incoming.message))
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces return self._do_dispatch(endpoint, method, ctxt, args)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces result = getattr(endpoint, method)(ctxt, **new_args)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 396, in decorated_function
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces return function(self, context, *args, **kwargs)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4347, in attach_interface
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces self.driver.attach_interface(instance, image_meta, network_info[0])
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1422, in attach_interface
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces raise exception.InterfaceAttachFailed(instance)
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces InterfaceAttachFailed: <nova.objects.instance.Instance object at 0x450af10>
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces

2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces InterfaceAttachFailed: <nova.objects.instance.Instance object at 0x450af10>
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-10-04 04:25:51.641 5579 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-10-04 04:25:51.645 5579 INFO nova.api.openstack.wsgi [req-59399617-ab6c-4d7e-8aa1-87dd44d1b9d9 e54aa038ff494a9094a1c00ee0161bd4 812d66ad27bd40d1bf81761afd59eeb9] HTTP exception thrown: Failed to attach interface
2014-10-04 04:25:51.646 5579 INFO nova.osapi_compute.wsgi.server [req-59399617-ab6c-4d7e-8aa1-87dd44d1b9d9 e54aa038ff494a9094a1c00ee0161bd4 812d66ad27bd40d1bf81761afd59eeb9] 10.204.217.7 "POST /v1.1/812d66ad27bd40d1bf81761afd59eeb9/servers/7c9396c3-0748-418a-a5bf-f8241c8df274/os-interface HTTP/1.1" status: 500 len: 278 time: 0.9511430

Tags: openstack
Revision history for this message
Hampapur Ajay (hajay) wrote :

I tried recreating in single node and didn't encounter this. Vedu, could you comment on whether this occurs always on multi-node setup or happens inconsistently? If it occurs inconsistently would appreciate access to system to debug further. Also it seems that there was a problem in nova-compute with libvirt which leads to nova code to call vif unplug (but it won't do neutron port delete) so it should occur upstream too if there is a problem.

Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :

Was unable to repro it on Build 52...will monitor this

tags: removed: blocker
Revision history for this message
Hampapur Ajay (hajay) wrote :

The failure to attach is at libvirt layer, see
https://bugs.launchpad.net/juniperopenstack/+bug/1367220/comments/1

Revision history for this message
Atul Moghe (moghea) wrote :

This should work with 14.04 icehouse

Revision history for this message
Atul Moghe (moghea) wrote :

QA verified that, this works with 2.0 icehosue 14.04 and 12.04.03 based systems.

Revision history for this message
Atul Moghe (moghea) wrote :

Fix for both 2.0 and trunk came from syncing nova to latest openstack release tag.

Revision history for this message
Atul Moghe (moghea) wrote :

restoring 1.1 status to incomplete. It is fixed in 2.0 and trunk

Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :

I seem to have done a mistake while validating this Bug 1377419 to say that it works on Ubuntu 12.4.3 Icehouse. It does not.

Like it was mentioned in Comment#3 by Ajay, the libvirt version 1.2.2-0ubuntu13.1.2 has this bug.

To summarize :

Ubuntu 12.04.3 1.10 34 Havana has libvirt 0.9.8-2ubuntu17.17 (Works)
Ubuntu 12.04.3 2.0 20 icehouse has libvirt 1.2.2-0ubuntu13.1.2 (Fails)
Ubuntu 14.04 3.0 2509 icehouse has libvirt 1.2.2-0ubuntu13.1.6 (Works)
Ubuntu 14.04 2.0 20 icehouse has libvirt 1.2.2-0ubuntu13.1.6 (Works)
Ubuntu 14.04 3.0 2509 juno has libvirt 1.2.2-0ubuntu13.1.5 (Works)

Centos 6.5 2.0 18 icehouse has libvirt 0.10.2-29.el6 (Works)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.