Unable to delete port - instance-ip deletion throws error

[Harish Kumar]

During running a rally scenario, I have seen contrail-api got stuck (not responding on time), so restarted contrail-api - then on, one of the port is not getting deleted - its instance-ip deletion is causing exception because its virtual_network_refs are recorded as "to: [ERROR]" rather than id (see below output).

It is three node setup with three control nodes as VMs (three VMs). But I only enabled harishku-vm3 for config api for easy troubleshooting. contrail version - 2.21-105

curl -u admin:secret http://localhost:8095/instance-ip/04c016bd-b9c1-412e-9eed-6ee09504f3b4 ; echo
{"instance-ip": {"display_name": "default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left", "uuid": "04c016bd-b9c1-412e-9eed-6ee09504f3b4", "instance_ip_address": "100.64.0.4", "virtual_machine_interface_refs": [{"to": ["default-domain", "c_rally_d68a4cf1_ZGNDTqOJ", "default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9__2__left__2"], "href": "http://localhost:8095/virtual-machine-interface/a4de7e38-872c-4aab-9a04-b24dbe6f3ae3", "attr": null, "uuid": "a4de7e38-872c-4aab-9a04-b24dbe6f3ae3"}], "instance_ip_mode": "active-standby", "href": "http://localhost:8095/instance-ip/04c016bd-b9c1-412e-9eed-6ee09504f3b4", "id_perms": {"enable": true, "description": null, "created": "2015-12-01T09:36:35.060840", "creator": null, "uuid": {"uuid_mslong": 342298575801237806, "uuid_lslong": 11451931338324898740}, "user_visible": true, "last_modified": "2015-12-01T10:23:31.651271", "permissions": {"owner": "cloud-admin", "owner_access": 7, "other_access": 7, "group": "cloud-admin-group", "group_access": 7}}, "fq_name": ["default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left"], "virtual_network_refs": [{"to": ["ERROR"], "href": "http://localhost:8095/virtual-network/9f0f9931-1f21-4152-ba4f-51a38c28f703", "attr": null, "uuid": "9f0f9931-1f21-4152-ba4f-51a38c28f703"}, {"to": ["default-domain", "c_rally_d68a4cf1_ZGNDTqOJ", "snat-si-left_si_6840042f-351b-4726-93d4-8bd4f41848d9"], "href": "http://localhost:8095/virtual-network/faf035ec-9bc6-4abd-9644-ee53a9339dbe", "attr": null, "uuid": "faf035ec-9bc6-4abd-9644-ee53a9339dbe"}], "name": "default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left"}}

[Harish Kumar (hkumarmk)]

Logs available at http://10.204.216.50/Docs/bugs/1521853/

[Sachin Bansal (sbansal)]

ERROR:harishku-vm1:contrail-api:Config:0:__default__ [SYS_ERR]: VncApiError: <class 'cfgm_common.exceptions.NoIdError'>
Python 2.7.6: /usr/bin/python
Tue Dec 1 15:10:23 2015

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in handler_trap_exception(*args=(), **kwargs={'id': '04c016bd-b9c1-412e-9eed-6ee09504f3b4'})
  534
  535 trace = self._generate_rest_api_request_trace()
  536 response = handler(*args, **kwargs)
  537 self._generate_rest_api_response_trace(trace, response)
  538
response undefined
handler = <bound method VncApiServer.instance_ip_http_dele...i_server.vnc_cfg_api_server.VncApiServer object>>
args = ()
kwargs = {'id': '04c016bd-b9c1-412e-9eed-6ee09504f3b4'}

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py in instance_ip_http_delete(self=<vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>, id='04c016bd-b9c1-412e-9eed-6ee09504f3b4')
 2379 self._instance_ip_delete_default_children(read_result)
 2380
 2381 (ok, del_result) = r_class.http_delete(id, read_result, db_conn)
 2382 if not ok:
 2383 (code, msg) = del_result
ok = True
del_result = ''
r_class = <class 'vnc_cfg_api_server.vnc_cfg_types.InstanceIpServer'>
r_class.http_delete = <bound method type.http_delete of <class 'vnc_cfg_api_server.vnc_cfg_types.InstanceIpServer'>>
id = '04c016bd-b9c1-412e-9eed-6ee09504f3b4'
read_result = {'display_name': u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left', 'fq_name': [u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9-left'], 'id_perms': {u'created': u'2015-12-01T09:36:35.060840', u'creator': None, u'description': None, u'enable': True, u'last_modified': u'2015-12-01T09:36:39.131312', u'permissions': {u'group': u'cloud-admin-group', u'group_access': 7, u'other_access': 7, u'owner': u'cloud-admin', u'owner_access': 7}, u'user_visible': True, u'uuid': {u'uuid_lslong': 11451931338324898740L, u'uuid_mslong': 342298575801237806}}, 'instance_ip_address': u'100.64.0.4', 'instance_ip_mode': u'active-standby', 'uuid': '04c016bd-b9c1-412e-9eed-6ee09504f3b4', 'virtual_machine_interface_refs': [{'attr': None, 'href': 'http://10.204.217.89:8082/virtual-machine-interface/a4de7e38-872c-4aab-9a04-b24dbe6f3ae3', 'to': [u'default-domain', u'c_rally_d68a4cf1_ZGNDTqOJ', u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9__2__left__2'], 'uuid': 'a4de7e38-872c-4aab-9a04-b24dbe6f3ae3'}, {'attr': None, 'href': 'http://10.204.217.89:8082/virtual-machine-interface/caf74340-7ce3-4f8d-bbdd-9292416aeb4b', 'to': [u'default-domain', u'c_rally_d68a4cf1_ZGNDTqOJ', u'default-domain__c_rally_d68a4cf1_ZGNDTqOJ__si_6840042f-351b-4726-93d4-8bd4f41848d9__1__left__2'], 'uuid': 'caf74340-7ce3-4f8d-bbdd-9292416aeb4b'}], 'virtual_network_refs': [{'attr': None, 'href': 'http://10.204.217.89:8082/virtual-network/9f0f9931-1f21-4152-ba4f-51a38c28f703', 'to': ['ERRO...

[Atul Moghe (moghea)]

Assigning to Jeba to reproduce with some later branches like 3.2 or 4.0/mainline.. 2.21 logs may not give much clue..