[2.1-Build 40] instance IP cannot be deleted due to "VncApiError: <type 'exceptions.KeyError'>"

[Daisuke Nakajima]

When we try to delete "/instance-ip/a4deb962-3907-41a0-86f9-064c0e478e7b", we get 'internal server error"

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 856, in _handle
    return route.call(**args)
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 1721, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 497, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py", line 2135, in instance_ip_http_delete
    (ok, del_result) = r_class.http_delete(id, read_result, db_conn)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_types.py", line 260, in http_delete
    ip_addr = obj_dict['instance_ip_address']
KeyError: 'instance_ip_address'
172.23.10.193 - - [2015-05-15 10:20:40] "DELETE /instance-ip/a4deb962-3907-41a0-86f9-064c0e478e7b HTTP/1.1" 500 156 0.017596
ERROR:openc-12:contrail-api:Config:0:__default__ [SYS_ERR]: VncApiError: <type 'exceptions.KeyError'>
Python 2.7.6: /usr/bin/python
Fri May 15 10:20:40 2015

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in handler_trap_exception(*args=(), **kwargs={'id': 'a4deb962-3907-41a0-86f9-064c0e478e7b'})
  495 trace = self._generate_rest_api_request_trace()
  496 try:
  497 response = handler(*args, **kwargs)
  498 self._generate_rest_api_response_trace(trace, response)
  499 return response
response undefined
handler = <bound method VncApiServer.instance_ip_http_dele...i_server.vnc_cfg_api_server.VncApiServer object>>
args = ()
kwargs = {'id': 'a4deb962-3907-41a0-86f9-064c0e478e7b'}

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py in instance_ip_http_delete(self=<vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>, id='a4deb962-3907-41a0-86f9-064c0e478e7b')
 2133 self._instance_ip_delete_default_children(read_result)
 2134
 2135 (ok, del_result) = r_class.http_delete(id, read_result, db_conn)
 2136 if not ok:
 2137 (code, msg) = del_result
ok = True
del_result = ''
r_class = <class 'vnc_cfg_api_server.vnc_cfg_types.InstanceIpServer'>
r_class.http_delete = <bound method type.http_delete of <class 'vnc_cfg_api_server.vnc_cfg_types.InstanceIpServer'>>
id = 'a4deb962-3907-41a0-86f9-064c0e478e7b'
read_result = {'display_name': u'a4deb962-3907-41a0-86f9-064c0e478e7b', 'fq_name': [u'a4deb962-3907-41a0-86f9-064c0e478e7b'], 'id_perms': {u'created': u'2015-05-13T02:29:41.263960', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2015-05-13T02:29:41.263960', u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': {u'uuid_lslong': 9725811794010148475L, u'uuid_mslong': 11880136698564657568L}}, 'subnet_uuid': u'481ccd61-3718-41a0-9d1c-4d7ea82e801b', 'uuid': 'a4deb962-3907-41a0-86f9-064c0e478e7b', 'virtual_machine_interface_refs': [{'attr': None, 'href': 'http://172.23.10.194:9100/virtual-machine-interface/22302759-fafa-4a67-b8b1-dd27fdde32c3', 'to': [u'default-domain', u'common', u'22302759-fafa-4a67-b8b1-dd27fdde32c3'], 'uuid': '22302759-fafa-4a67-b8b1-dd27fdde32c3'}], 'virtual_network_refs': [{'attr': None, 'href': 'http://172.23.10.194:9100/virtual-network/1ccd6de8-c11d-45f9-b5d9-51c15e2a31a4', 'to': [u'default-domain', u'common', u'vrouter_test_30'], 'uuid': '1ccd6de8-c11d-45f9-b5d9-51c15e2a31a4'}]}
db_conn = <vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object>

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_types.py in http_delete(cls=<class 'vnc_cfg_api_server.vnc_cfg_types.InstanceIpServer'>, id='a4deb962-3907-41a0-86f9-064c0e478e7b', obj_dict={'display_name': u'a4deb962-3907-41a0-86f9-064c0e478e7b', 'fq_name': [u'a4deb962-3907-41a0-86f9-064c0e478e7b'], 'id_perms': {u'created': u'2015-05-13T02:29:41.263960', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2015-05-13T02:29:41.263960', u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': {u'uuid_lslong': 9725811794010148475L, u'uuid_mslong': 11880136698564657568L}}, 'subnet_uuid': u'481ccd61-3718-41a0-9d1c-4d7ea82e801b', 'uuid': 'a4deb962-3907-41a0-86f9-064c0e478e7b', 'virtual_machine_interface_refs': [{'attr': None, 'href': 'http://172.23.10.194:9100/virtual-machine-interface/22302759-fafa-4a67-b8b1-dd27fdde32c3', 'to': [u'default-domain', u'common', u'22302759-fafa-4a67-b8b1-dd27fdde32c3'], 'uuid': '22302759-fafa-4a67-b8b1-dd27fdde32c3'}], 'virtual_network_refs': [{'attr': None, 'href': 'http://172.23.10.194:9100/virtual-network/1ccd6de8-c11d-45f9-b5d9-51c15e2a31a4', 'to': [u'default-domain', u'common', u'vrouter_test_30'], 'uuid': '1ccd6de8-c11d-45f9-b5d9-51c15e2a31a4'}]}, db_conn=<vnc_cfg_api_server.vnc_cfg_ifmap.VncDbClient object>)
  258 return True, ""
  259
  260 ip_addr = obj_dict['instance_ip_address']
  261 db_conn.config_log('AddrMgmt: free IP %s, vn=%s'
  262 % (ip_addr, vn_fq_name),
ip_addr undefined
obj_dict = {'display_name': u'a4deb962-3907-41a0-86f9-064c0e478e7b', 'fq_name': [u'a4deb962-3907-41a0-86f9-064c0e478e7b'], 'id_perms': {u'created': u'2015-05-13T02:29:41.263960', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2015-05-13T02:29:41.263960', u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': {u'uuid_lslong': 9725811794010148475L, u'uuid_mslong': 11880136698564657568L}}, 'subnet_uuid': u'481ccd61-3718-41a0-9d1c-4d7ea82e801b', 'uuid': 'a4deb962-3907-41a0-86f9-064c0e478e7b', 'virtual_machine_interface_refs': [{'attr': None, 'href': 'http://172.23.10.194:9100/virtual-machine-interface/22302759-fafa-4a67-b8b1-dd27fdde32c3', 'to': [u'default-domain', u'common', u'22302759-fafa-4a67-b8b1-dd27fdde32c3'], 'uuid': '22302759-fafa-4a67-b8b1-dd27fdde32c3'}], 'virtual_network_refs': [{'attr': None, 'href': 'http://172.23.10.194:9100/virtual-network/1ccd6de8-c11d-45f9-b5d9-51c15e2a31a4', 'to': [u'default-domain', u'common', u'vrouter_test_30'], 'uuid': '1ccd6de8-c11d-45f9-b5d9-51c15e2a31a4'}]}
<type 'exceptions.KeyError'>: 'instance_ip_address'
    __class__ = <type 'exceptions.KeyError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.KeyError object>
    __dict__ = {}
    __doc__ = 'Mapping key not found.'
    __format__ = <built-in method __format__ of exceptions.KeyError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.KeyError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.KeyError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.KeyError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.KeyError object>
    __init__ = <method-wrapper '__init__' of exceptions.KeyError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.KeyError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.KeyError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.KeyError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.KeyError object>
    __setstate__ = <built-in method __setstate__ of exceptions.KeyError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.KeyError object>
    __str__ = <method-wrapper '__str__' of exceptions.KeyError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.KeyError object>
    args = ('instance_ip_address',)
    message = 'instance_ip_address'

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 497, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py", line 2135, in instance_ip_http_delete
    (ok, del_result) = r_class.http_delete(id, read_result, db_conn)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_types.py", line 260, in http_delete
    ip_addr = obj_dict['instance_ip_address']
KeyError: 'instance_ip_address'

[Sachin Bansal (sbansal)]

Fixed in master with this commit:
https://github.com/Juniper/contrail-controller/commit/b9b9e4258847840351eb2b2d432637d8cd96736b

[Prakash Bailkeri (prakashmb)]

To recreate the issue (Create a subnet with allocation pool and create port with fixed_ip by specifying both ip and subnet_id):

neutron net-create vn22
# Create subnet with allocation pool
neutron subnet-create vn22 --allocation-pool start=1.2.3.34,end=1.2.3.36 1.2.3.0/24
# Requested ip is outside the allocation pool
neutron port-create vn22 --fixed-ip subnet_id=c0644fe6-eb22-4620-a58c-fb99ccafb528,ip_address=1.2.3.44

Port create returns error and from then on port-list will fail and also instance ip delete would thrown exception.

This bug is duplicate of https://bugs.launchpad.net/juniperopenstack/+bug/1451701.
Commit needs to be back ported to R2.20.

Additionally, code will be added to avoid exception in case instance_ip_address field is missing in cassandra.

Fix mentioned in comment #7 doesn't fix the issue.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/10922
Submitter: Prakash Bailkeri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11018
Submitter: Prakash Bailkeri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11019
Submitter: Prakash Bailkeri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11018
Committed: http://github.org/Juniper/contrail-controller/commit/e91754ea9e82b7982a999333f2f2a2218214dbc1
Submitter: Zuul
Branch: master

commit e91754ea9e82b7982a999333f2f2a2218214dbc1
Author: Prakash Bailkeri <email address hidden>
Date: Wed May 27 08:48:00 2015 -0700

Ignore missing instance_ip_address in InstanceIp object

When ip address is allocated by specifying both subnet id and fixed ip addres outside allocation pool, instance ip object is created without instance ip address field.

This subsequently causes exception in port-list, port-delete and instance ip delete.

Fix: ignore KeyError exception while accessing instance ip address field.

Change-Id: I6ead9f6d427269fc5391dcf85469a54eb6a964da
Closes-bug: #1455321

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11019
Committed: http://github.org/Juniper/contrail-controller/commit/2f37afc1a9b680cde1a7d2d5edac6a6b68f8e9c4
Submitter: Zuul
Branch: R2.20

commit 2f37afc1a9b680cde1a7d2d5edac6a6b68f8e9c4
Author: Prakash Bailkeri <email address hidden>
Date: Wed May 27 08:48:00 2015 -0700

Ignore missing instance_ip_address in InstanceIp object

When ip address is allocated by specifying both subnet id and fixed ip addres outside allocation pool, instance ip object is created without instance ip address field.

This subsequently causes exception in port-list, port-delete and instance ip delete.

Fix: ignore KeyError exception while accessing instance ip address field.

Change-Id: I6ead9f6d427269fc5391dcf85469a54eb6a964da
Closes-bug: #1455321