VM not deleted due to port-delete failing (failure in interface-route-table's deletion)

Bug #1520201 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Fix Committed
High
Sachin Bansal
R2.20
Fix Committed
Medium
Sachin Bansal

Bug Description

2.20 Build 108 Ubuntu 14.04 Juno multi-node HA setup

nodec1,nodec2,nodec3 form the controller cluster

We were running a case from a test script where one of the VM deletions failed . It was due to port-delete returning an error.
The port-delete error seems to be due to failure in deleting interface-route-table . The interface-route-table and the port did get deleted though

Full logs of the nodes and cassandra config log will be in http://10.204.216.50/Docs/bugs/#

root@nodec1:~/logs# nova show 627f27c7-4e64-4501-a922-95f92b97d907
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | nodek2 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | nodek2.englab.juniper.net |
| OS-EXT-SRV-ATTR:instance_name | instance-0000036d |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | 2015-11-26T10:51:56.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-11-26T10:51:53Z |
                                                    |
| fault | {"message": "An unknown exception occurred.", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 326, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2543, in terminate_instance |
| | do_terminate_instance(instance, bdms) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py\", line 272, in inner |
| | return f(*args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2541, in do_terminate_instance |
| | self._set_instance_error_state(context, instance) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2531, in do_terminate_instance |
| | self._delete_instance(context, instance, bdms, quotas) |
| | File \"/usr/lib/python2.7/dist-packages/nova/hooks.py\", line 131, in inner |
| | rv = f(*args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2500, in _delete_instance |
| | quotas.rollback() |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2477, in _delete_instance |
| | self._shutdown_instance(context, instance, bdms) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2410, in _shutdown_instance |
| | self._try_deallocate_network(context, instance, requested_networks) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2340, in _try_deallocate_network
                                                     |
| | self._set_instance_error_state(context, instance) |
| | File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py\", line 82, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 2335, in _try_deallocate_network |
| | self._deallocate_network(context, instance, requested_networks) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 1939, in _deallocate_network |
| | context, instance, requested_networks=requested_networks) |
| | File \"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 552, in deallocate_for_instance |
| | self._delete_ports(neutron, instance, ports, raise_if_fail=True) |
| | File \"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py\", line 525, in _delete_ports |
| | raise exceptions[0] |
| | ", "created": "2015-11-26T11:36:04Z"} |
| flavor | contrail_flavor_small (ec0eea8e-aac7-4a65-8914-1c3650a24a5a) |
| hostId | c5bcd8ed77a74ce293ccc56219f5f7f1d4180988f3249002d948ad4f |
| id | 627f27c7-4e64-4501-a922-95f92b97d907 |
| image | ubuntu-traffic (7799b586-4266-4f30-a984-e3f4c9c32cce) |
| key_name | juniperkey1 |
| metadata | {} |
| name | vmc4 |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | 4551763b91b649a8a402c1fbc54e97f9 |
| updated | 2015-11-26T11:36:04Z |
| user_id | 6888f8e3070d4a9e9a26e661f053608e
                                                    |
| vnet2 network | 10.2.1.4 |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@nodec1:~/logs#

Related contrail api errors :

127.0.0.1 - - [2015-11-26 17:06:04] "DELETE /virtual-machine-interface/b115c6cf-d0b5-4796-b8f5-37e04cb06e39 HTTP/1.1" 200 115 0.024795
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 856, in _handle
    return route.call(**args)
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 1721, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 536, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py", line 17428, in interface_route_table_http_delete
    (ok, del_result) = self._delete_common(request, 'interface_route_table', id, parent_type)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1348, in _http_delete_common
    obj_dict = self._db_conn.uuid_to_obj_dict(uuid)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1574, in uuid_to_obj_dict
    return self._cassandra_db.uuid_to_obj_dict(obj_uuid)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 692, in uuid_to_obj_dict
    raise NoIdError(id)
NoIdError: Unknown id: c0906f29-fb8f-460f-b4b5-4b2575ad715d
127.0.0.1 - - [2015-11-26 17:06:04] "DELETE /interface-route-table/c0906f29-fb8f-460f-b4b5-4b2575ad715d HTTP/1.1" 500 156 0.007171

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 422, in plugin_delete_port
    cfgdb.port_delete(port['id'])
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 2227, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 3707, in port_delete
    self._vnc_lib.interface_route_table_delete(id=rt_ref['uuid'])
  File "/usr/lib/python2.7/dist-packages/vnc_api/gen/vnc_api_client_gen.py", line 3652, in interface_route_table_delete
    content = self._request_server(rest.OP_DELETE, uri)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 497, in _request_server
    retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 558, in _request
    raise HttpError(status, content)
HttpError: HTTP Status: 500 Content: Internal Server Error

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 960, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 473, in plugin_http_post_port
    return self.plugin_delete_port(context, port)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 422, in plugin_delete_port
    cfgdb.port_delete(port['id'])
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 2227, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 3707, in port_delete
    self._vnc_lib.interface_route_table_delete(id=rt_ref['uuid'])
  File "/usr/lib/python2.7/dist-packages/vnc_api/gen/vnc_api_client_gen.py", line 3652, in interface_route_table_delete
    content = self._request_server(rest.OP_DELETE, uri)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 497, in _request_server
    retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 558, in _request
    raise HttpError(status, content)
HttpError: HTTP Status: 500 Content: Internal Server Error

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 856, in _handle
    return route.call(**args)
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 1721, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 960, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 473, in plugin_http_post_port
    return self.plugin_delete_port(context, port)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 422, in plugin_delete_port
    cfgdb.port_delete(port['id'])
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 2227, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 3707, in port_delete
    self._vnc_lib.interface_route_table_delete(id=rt_ref['uuid'])
  File "/usr/lib/python2.7/dist-packages/vnc_api/gen/vnc_api_client_gen.py", line 3652, in interface_route_table_delete
    content = self._request_server(rest.OP_DELETE, uri)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 497, in _request_server
    retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 558, in _request
    raise HttpError(status, content)
HttpError: HTTP Status: 500 Content: Internal Server Error
10.204.216.58 - - [2015-11-26 17:06:04] "POST /neutron/port HTTP/1.1" 500 156 0.129024
10.204.216.58 - - [2015-11-26 17:06:10] "GET /virtual-machine/627f27c7-4e64-4501-a922-95f92b97d907 HTTP/1.1" 200 850 0.005240

Tags: config
Changed in juniperopenstack:
milestone: r3.0-fcs → r3.1.0.0-fcs
assignee: Hampapur Ajay (hajay) → Sachin Bansal (sbansal)
Sachin Bansal (sbansal)
Changed in juniperopenstack:
status: New → Fix Committed
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/19158
Submitter: Sachin Bansal (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19158
Committed: http://github.org/Juniper/contrail-controller/commit/b62f1f780882432b65e66c84e8964052baff6dff
Submitter: Zuul
Branch: R2.20

commit b62f1f780882432b65e66c84e8964052baff6dff
Author: Sachin Bansal <email address hidden>
Date: Thu Apr 7 11:53:59 2016 -0700

Do not read object just to get fq_name

fq_name of the object was already obtained in the same function. We should not
read it again.

Change-Id: I353233af380fc262f5f07ed52d223809baf98f87
Closes-Bug: 1520201

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.