[R4.1-15]:Lots of NoIdError seen in api logs while resource deletion

[alok kumar]

Lots of "NoIdError: Unknown id" errors are seen in api logs while project/sg deletion.

sanity report: http://10.204.216.50/Docs/logs/4.1.0.0-15_jenkins-ubuntu-14-04_mitaka_Multi_Node_Sanity-446_1507839596.84/junit-noframes.html

sample traceback logs collected: http://10.204.216.50/Docs/logs/4.1.0.0-15_jenkins-ubuntu-14-04_mitaka_Multi_Node_Sanity-446_1507839596.84/logs/nodec57-controller-tracebacks.log

contrail logs:/cs-shared/test_runs/nodec7/jenkins-ubuntu-14-04_mitaka_Multi_Node_Sanity-446/

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 818, in http_resource_delete
    'pre_%s_delete' %(obj_type), id)
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 252, in map_method
    method_name, *args, **kwds)
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 225, in map
    self._invoke_one_plugin(response.append, func, e, args, kwds)
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 256, in _invoke_one_plugin
    response_callback(func(e, *args, **kwds))
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 230, in _call_extension_method
    return getattr(extension.obj, method_name)(*args, **kwds)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 888, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 942, in pre_project_delete
    proj_obj = self._vnc_lib.project_read(id=proj_uuid)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 544, in _object_read
    response = self._request_server(rest.OP_GET, uri, query_params)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 855, in _request_server
    retry_after_authn=retry_after_authn, retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 933, in _request
    % (op, url, data, content))
NoIdError: Unknown id: Error: oper 2 url /project/390df54b-da13-497b-9f38-e213d009deda body {'exclude_back_refs': True, 'exclude_children': True} response Unknown id: 390df54b-da13-497b-9f38-e213d009deda

This is probably because of multiple configs(3 nodes) and on one of the config deletion goes through fine and then other configs may get this error(timing issue).

please have a look if traceback can be avoided as deletion of the resource was successful.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36627
Submitter: Édouard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36634
Submitter: Édouard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36635
Submitter: Édouard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36635
Committed: http://github.com/Juniper/contrail-controller/commit/5864ab70bc8a36d5598cf81375545f445e0448ca
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 5864ab70bc8a36d5598cf81375545f445e0448ca
Author: Édouard Thuleau <email address hidden>
Date: Wed Oct 18 16:09:39 2017 +0200

[VNC OpenStack] Fix concurrency project deletion

In mutli API server setup, the sync domains/projects VNC OpenStack
greenthread of one of the servers can try to remove and purge resource related
to a project which another server already purged. That results to not found
API exceptions. That patch catches that exceptions and abandons to purge
the project.
That also reveals an issue in permission code that checks if the
deletion is authorized which fails to read the resource as it was
already deleted.

Change-Id: I140cf5b7f4bd4d7798db18a7afd3b0d6168ea6b1
Closes-Bug: #1724179
Closes-Bug: #1724194

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36634
Committed: http://github.com/Juniper/contrail-controller/commit/78242e8c5b2a67acb6deadea78a97c5c365095db
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 78242e8c5b2a67acb6deadea78a97c5c365095db
Author: Édouard Thuleau <email address hidden>
Date: Wed Oct 18 16:09:39 2017 +0200

[VNC OpenStack] Fix concurrency project deletion

In mutli API server setup, the sync domains/projects VNC OpenStack
greenthread of one of the servers can try to remove and purge resource related
to a project which another server already purged. That results to not found
API exceptions. That patch catches that exceptions and abandons to purge
the project.
That also reveals an issue in permission code that checks if the
deletion is authorized which fails to read the resource as it was
already deleted.

Change-Id: I140cf5b7f4bd4d7798db18a7afd3b0d6168ea6b1
Closes-Bug: #1724179
Closes-Bug: #1724194

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36627
Committed: http://github.com/Juniper/contrail-controller/commit/342ca4044ecc7e4eca01bbb045a080a27386b915
Submitter: Zuul (<email address hidden>)
Branch: master

commit 342ca4044ecc7e4eca01bbb045a080a27386b915
Author: Édouard Thuleau <email address hidden>
Date: Wed Oct 18 16:09:39 2017 +0200

[VNC OpenStack] Fix concurrency project deletion

In mutli API server setup, the sync domains/projects VNC OpenStack
greenthread of one of the servers can try to remove and purge resource related
to a project which another server already purged. That results to not found
API exceptions. That patch catches that exceptions and abandons to purge
the project.
That also reveals an issue in permission code that checks if the
deletion is authorized which fails to read the resource as it was
already deleted.

Change-Id: I140cf5b7f4bd4d7798db18a7afd3b0d6168ea6b1
Closes-Bug: #1724179
Closes-Bug: #1724194