K8S: Nested: VM reference still present in agent after deletion of POD

[chhandak]

Description
-----------
In a Nested Kubernetes cluster after deletion of the POD, VM UUID still present in agent introspect. Kube-manager is having traceback while trying to delete the same as the reference still exists.

In Agent
---------
root@59c61034e315:/tmp/contrail-test-ci# curl http://10.87.66.153:8085/Snh_VmListReq?x=3f907200-f7e6-4178-b9a6-b1c6fbbc6996
<?xml-stylesheet type="text/xsl" href="/universal_parse.xsl"?><__VmListResp_list type="slist"><VmListResp type="sandesh"><vm_list type="list" identifier="1"><list type="struct" size="1"><VmSandeshData><uuid type="string" identifier="1">3f907200-f7e6-4178-b9a6-b1c6fbbc6996</uuid><drop_new_flows type="bool" identifier="3">false</drop_new_flows></VmSandeshData></list></vm_list><more type="bool" identifier="0">true</more></VmListResp><Pagination type="sandesh"><req type="struct" identifier="1"><PageReqData><prev_page type="string" identifier="1" link="PageReq"></prev_page><next_page type="string" identifier="2" link="PageReq"></next_page><first_page type="string" identifier="3" link="PageReq">begin:0,end:99,table:db.vm.0,name:3f907200-f7e6-4178-b9a6-b1c6fbbc6996</first_page><all type="string" identifier="4" link="PageReq">begin:-1,end:-1,table:db.vm.0,name:3f907200-f7e6-4178-b9a6-b1c6fbbc6996</all><table_size type="u32" identifier="5">6</table_size><entries type="string" identifier="6">0-0/1</entries></PageReqData></req><more ty

In API Server
--------------
root@5b8s38:~# curl -u admin:D70DA4079430 : http://127.0.0.1:8095/virtual-machine/3745da9e-341a-11e7-96bb-0276478b654b | python -m json.tool
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host:
100 1466 100 1466 0 0 308k 0 --:--:-- --:--:-- --:--:-- 308k
{
    "virtual-machine": {
        "annotations": {
            "key_value_pair": [
                {
                    "key": "cluster",
                    "value": "k8s"
                },
                {
                    "key": "kind",
                    "value": "Pod"
                },
                {
                    "key": "namespace",
                    "value": "default"
                },
                {
                    "key": "k8s_uuid",
                    "value": "3745da9e-341a-11e7-96bb-0276478b654b"
                },
                {
                    "key": "labels",
                    "value": "{}"
                },
                {
                    "key": "project",
                    "value": "admin"
                },
                {
                    "key": "name",
                    "value": "ctest-nginx-pod-29976947"
                },
                {
                    "key": "owner",
                    "value": "k8s"
                }
            ]
        },
        "display_name": "default__ctest-nginx-pod-29976947",
        "fq_name": [
            "ctest-nginx-pod-29976947__3745da9e-341a-11e7-96bb-0276478b654b"
        ],
        "href": "http://127.0.0.1:8095/virtual-machine/3745da9e-341a-11e7-96bb-0276478b654b",
        "id_perms": {
            "created": "2017-05-08T18:14:46.152002",
            "creator": null,
            "description": null,
            "enable": true,
            "last_modified": "2017-05-08T18:14:46.172724",
            "permissions": {
                "group": "cloud-admin-group",
                "group_access": 7,
                "other_access": 7,
                "owner": "cloud-admin",
                "owner_access": 7
            },
            "user_visible": true,
            "uuid": {
                "uuid_lslong": 10861277633363797323,
                "uuid_mslong": 3982829818492883431
            }
        },
        "name": "ctest-nginx-pod-29976947__3745da9e-341a-11e7-96bb-0276478b654b",
        "perms2": {
            "global_access": 0,
            "owner": "bb3c1dccb44941ae814d592dbb03f8d0",
            "owner_access": 7,
            "share": []
        },
        "uuid": "3745da9e-341a-11e7-96bb-0276478b654b",
        "virtual_router_back_refs": [
            {
                "attr": null,
                "href": "http://127.0.0.1:8095/virtual-router/0022d08e-0c5a-44db-952d-ff38abbaf433",
                "to": [
                    "default-global-system-config",
                    "5b8s38"
                ],
                "uuid": "0022d08e-0c5a-44db-952d-ff38abbaf433"
            }
        ]
    }
}
root@5b8s38:~#

COntrail-kube-manager Traceback
---------------------------------
The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/vnc_kubernetes.py", line 353, in vnc_process
    self.pod_mgr.process(event)
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/vnc_pod.py", line 480, in process
    self.vnc_pod_delete(pod_id)
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/vnc_pod.py", line 409, in vnc_pod_delete
    self._vnc_lib.virtual_machine_delete(id=pod_id)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 41, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 641, in _object_delete
    self._request_server(rest.OP_DELETE, uri)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 857, in _request_server
    retry_after_authn=retry_after_authn, retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 941, in _request
    raise RefsExistError(content)
RefsExistError: Delete when resource still referred: ['http://10.10.10.1:8082/virtual-router/0022d08e-0c5a-44db-952d-ff38abbaf433']

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31507
Submitter: Dinesh Bakiaraj (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31508
Submitter: Dinesh Bakiaraj (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31508
Committed: http://github.com/Juniper/contrail-controller/commit/aeeb311eb0c0ff02016bc797d9ba67641021bdf6
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit aeeb311eb0c0ff02016bc797d9ba67641021bdf6
Author: dineshb-jnpr <email address hidden>
Date: Fri May 12 16:02:11 2017 -0700

Fix VM-Vrouter ref update race.

When there is an add followed by delete, back-to-back, it is possible
VNC layer may not have updated the VM with vrouter info, as that ref
update is a seperate event.

Since VM's are always expected to vrouter info, if the ref is missing,
update that vrouter record in config db by reading from vnc.

Change-Id: I1771bad79ea6f58a768d9481224456fe7089534e
Closes-Bug: #1689905

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31507
Committed: http://github.com/Juniper/contrail-controller/commit/d96f298e001c002de35d9e3b7e1958cb11a721c5
Submitter: Zuul (<email address hidden>)
Branch: master

commit d96f298e001c002de35d9e3b7e1958cb11a721c5
Author: dineshb-jnpr <email address hidden>
Date: Fri May 12 16:02:11 2017 -0700

Fix VM-Vrouter ref update race.

When there is an add followed by delete, back-to-back, it is possible
VNC layer may not have updated the VM with vrouter info, as that ref
update is a seperate event.

Since VM's are always expected to vrouter info, if the ref is missing,
update that vrouter record in config db by reading from vnc.

Change-Id: I1771bad79ea6f58a768d9481224456fe7089534e
Closes-Bug: #1689905