Juniper Openstack

[4.0.3.0-104]: Internal Server Error while deleting the project

Series trunk
Bug #1724194

Bug #1724194 reported by alok kumar on 2017-10-17

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R4.0	Fix Committed	Undecided	Édouard Thuleau	Juniper Openstack r4.0.3.0
R4.1	Fix Committed	Undecided	Édouard Thuleau	Juniper Openstack r4.1.0.0-fcs "r4.1"
Trunk	Fix Committed	Undecided	Édouard Thuleau	Juniper Openstack r5.0.0

Bug Description

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 426, in _ksv2_del_project_from_vnc
    self._vnc_lib.project_delete(id=project_id)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 647, in _object_delete
    self._request_server(rest.OP_DELETE, uri)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 873, in _request_server
    retry_after_authn=retry_after_authn, retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 976, in _request
    raise HttpError(status, content)
HttpError: HTTP Status: 500 Content: Internal Server Error

ERROR:contrail-api:Failed to delete project 8b79f48d-edce-43a5-91e3-37c08e7fed4c: HTTP Status: 500 Content: Internal Server Error

Sanity report: http://10.204.216.50/Docs/logs/4.0.3.0-104_jenkins-SMLite_ubuntu-14-04_mitaka_Openstack_HA_Sanity-253_1508173849.37/junit-noframes.html

http://10.204.216.50/Docs/logs/4.0.3.0-104_jenkins-SMLite_ubuntu-14-04_mitaka_Openstack_HA_Sanity-253_1508173849.37/logs/nodem6-controller-tracebacks.log

api log is attached for reference.
contrail logs: /cs-shared/test_runs/nodem14/jenkins-SMLite_ubuntu-14-04_mitaka_Openstack_HA_Sanity-253

Tags:

Revision history for this message

alok kumar (kalok) wrote on 2017-10-17:

contrail-api-0-stdout.log Edit (3.5 MiB, text/plain)

summary:	- [R4.1-15]: Internal Server Error while deleting the project + [4.0.3.0-104]: Internal Server Error while deleting the project
information type:	Proprietary → Public

Revision history for this message

alok kumar (kalok) wrote on 2017-10-17:

same error is seen while one of SG deletion in same log file:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 832, in http_resource_delete
    'pre_%s_delete' %(obj_type), id)
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 252, in map_method
    method_name, *args, **kwds)
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 225, in map
    self._invoke_one_plugin(response.append, func, e, args, kwds)
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 256, in _invoke_one_plugin
    response_callback(func(e, *args, **kwds))
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 230, in _call_extension_method
    return getattr(extension.obj, method_name)(*args, **kwds)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 886, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 944, in pre_project_delete
    self._vnc_lib.security_group_delete(id=group['uuid'])
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 647, in _object_delete
    self._request_server(rest.OP_DELETE, uri) File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 873, in _request_server
    retry_after_authn=retry_after_authn, retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 976, in _request
    raise HttpError(status, content)
HttpError: HTTP Status: 500 Content: Internal Server Error

same error is seen while one of SG deletion in same log file:

Traceback (most recent call last):                                              
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 832, in http_resource_delete
    'pre_%s_delete' %(obj_type), id)                                            
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 252, in map_method
    method_name, *args, **kwds)                                                 
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 225, in map
    self._invoke_one_plugin(response.append, func, e, args, kwds)               
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 256, in _invoke_one_plugin
    response_callback(func(e, *args, **kwds))                                   
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 230, in _call_extension_method
    return getattr(extension.obj, method_name)(*args, **kwds)                   
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 886, in wrapper
    return func(self, *args, **kwargs)                                          
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 944, in pre_project_delete
    self._vnc_lib.security_group_delete(id=group['uuid'])                       
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)                                          
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 647, in _object_delete
    self._request_server(rest.OP_DELETE, uri)                                     File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 873, in _request_server
    retry_after_authn=retry_after_authn, retry_count=retry_count)               
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 976, in _request
    raise HttpError(status, content)                                            
HttpError: HTTP Status: 500 Content: Internal Server Error

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-10-18: [Review update] master

Review in progress for https://review.opencontrail.org/36627
Submitter: Édouard Thuleau (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-10-18: [Review update] R4.1

Review in progress for https://review.opencontrail.org/36634
Submitter: Édouard Thuleau (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-10-18: [Review update] R4.0

Review in progress for https://review.opencontrail.org/36635
Submitter: Édouard Thuleau (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-10-19: A change has been merged

Reviewed: https://review.opencontrail.org/36635
Committed: http://github.com/Juniper/contrail-controller/commit/5864ab70bc8a36d5598cf81375545f445e0448ca
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 5864ab70bc8a36d5598cf81375545f445e0448ca
Author: Édouard Thuleau <email address hidden>
Date: Wed Oct 18 16:09:39 2017 +0200

[VNC OpenStack] Fix concurrency project deletion

In mutli API server setup, the sync domains/projects VNC OpenStack
greenthread of one of the servers can try to remove and purge resource related
to a project which another server already purged. That results to not found
API exceptions. That patch catches that exceptions and abandons to purge
the project.
That also reveals an issue in permission code that checks if the
deletion is authorized which fails to read the resource as it was
already deleted.

Change-Id: I140cf5b7f4bd4d7798db18a7afd3b0d6168ea6b1
Closes-Bug: #1724179
Closes-Bug: #1724194

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-10-19:

#10

Reviewed: https://review.opencontrail.org/36634
Committed: http://github.com/Juniper/contrail-controller/commit/78242e8c5b2a67acb6deadea78a97c5c365095db
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 78242e8c5b2a67acb6deadea78a97c5c365095db
Author: Édouard Thuleau <email address hidden>
Date: Wed Oct 18 16:09:39 2017 +0200

[VNC OpenStack] Fix concurrency project deletion

Change-Id: I140cf5b7f4bd4d7798db18a7afd3b0d6168ea6b1
Closes-Bug: #1724179
Closes-Bug: #1724194

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-10-23:

#11

Reviewed: https://review.opencontrail.org/36627
Committed: http://github.com/Juniper/contrail-controller/commit/342ca4044ecc7e4eca01bbb045a080a27386b915
Submitter: Zuul (<email address hidden>)
Branch: master