nova security group extension doesn't handle neutron exception properly

[Miguel Lavalle]

When networking is provided by Neutron, and a security group create request exceeds the tenants security group quota in Neutron, the security groups API extension (/nova/nova/api/openstack/compute/contrib/security_groups.py) doesn't manage properly the exception. As a consequence, the user gets a 500, "The server has either erred or is incapable of performing the requested operation", instead of a quota exceeded error.

2013-08-03 17:21:45.418 29258 DEBUG neutronclient.v2_0.client [-] Error message: {"NeutronError": "Quota exceeded for resources: ['security_group']"} _handle_fault_response /opt/stack/python-neutronclient/neutronclient/v2_0/client.py:756
2013-08-03 17:21:45.419 ERROR nova.network.security_group.neutron_driver [req-31ec340c-a595-4819-b7ec-887b89bc108b demo demo] Neutron Error creating security group sg-overlimit
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver Traceback (most recent call last):
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver File "/opt/stack/nova/nova/network/security_group/neutron_driver.py", line 54, in create_security_group
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver body).get('security_group')
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 108, in with_params
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver ret = self.function(instance, *args, **kwargs)
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 422, in create_security_group
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver return self.post(self.security_groups_path, body=body)
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 872, in post
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver headers=headers, params=params)
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 795, in do_request
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver self._handle_fault_response(status_code, replybody)
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 765, in _handle_fault_response
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver exception_handler_v20(status_code, des_error_body)
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 81, in exception_handler_v20
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver message=error_dict)
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver NeutronClientException: Quota exceeded for resources: ['security_group']
2013-08-03 17:21:45.419 29258 TRACE nova.network.security_group.neutron_driver
2013-08-03 17:21:45.419 ERROR nova.api.openstack [req-31ec340c-a595-4819-b7ec-887b89bc108b demo demo] Caught error: Quota exceeded for resources: ['security_group']
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack Traceback (most recent call last):
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/__init__.py", line 111, in __call__
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack return req.get_response(self.application)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack application, catch_exc_info=False)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack return resp(environ, start_response)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 473, in __call__
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack return self.app(env, start_response)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack return resp(environ, start_response)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack return resp(environ, start_response)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack response = self.app(environ, start_response)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack return resp(environ, start_response)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 904, in __call__
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack content_type, body, accept)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 963, in _process_stack
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1044, in dispatch
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack return method(req=request, **action_args)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/contrib/security_groups.py", line 316, in create
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack context, group_name, group_description)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/nova/nova/network/security_group/neutron_driver.py", line 54, in create_security_group
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack body).get('security_group')
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 108, in with_params
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack ret = self.function(instance, *args, **kwargs)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 422, in create_security_group
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack return self.post(self.security_groups_path, body=body)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 872, in post
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack headers=headers, params=params)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 795, in do_request
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack self._handle_fault_response(status_code, replybody)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 765, in _handle_fault_response
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack exception_handler_v20(status_code, des_error_body)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack File "/opt/stack/python-neutronclient/neutronclient/v2_0/client.py", line 81, in exception_handler_v20
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack message=error_dict)
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack NeutronClientException: Quota exceeded for resources: ['security_group']
2013-08-03 17:21:45.419 29258 TRACE nova.api.openstack
2013-08-03 17:21:45.421 INFO nova.api.openstack [req-31ec340c-a595-4819-b7ec-887b89bc108b demo demo] http://172.16.0.2:8774/v2/ddae52eadcfb403ca90a0f524f1263da/os-security-groups returned with HTTP 500
2013-08-03 17:21:45.421 DEBUG nova.api.openstack.wsgi [req-31ec340c-a595-4819-b7ec-887b89bc108b demo demo] Returning 500 to user: The server has either erred or is incapable of performing the requested operation. __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1188

[Matt Riedemann (mriedem)]

I've posted a related fix to neutron for one of the exceptions in neutron that's a 409 error which probably shouldn't be:

https://review.openstack.org/#/c/40954/

[Matt Riedemann (mriedem)]

Also posted a patch to remove the unused AlreadyAttached exception from Neutron:

https://bugs.launchpad.net/neutron/+bug/1210276

And this one changes InvalidQuotaValue to a 403 rather than 409:

https://review.openstack.org/#/c/40969/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/41177

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/41329

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/41329
Committed: http://github.com/openstack/nova/commit/862bf3dfb2058c62eeb5c82544e66d3be267ec8c
Submitter: Jenkins
Branch: master

commit 862bf3dfb2058c62eeb5c82544e66d3be267ec8c
Author: Miguel Lavalle <email address hidden>
Date: Mon Jul 29 07:32:26 2013 -0500

    Handle NeutronClientException in secgroup create

    When networking is provided by Neutron and a security group or a security group
    rules quota is exceeded, we get a NeutronClientException with status_code 409
    and message "NeutronClientException: Quota exceeded for resources"

    This patch updates the neutron security group driver to handle this exception
    and translate it to nova's SecurityGroupLimitExceeded

    test_neutron_driver is also updated to provide unit tests for this patch

    Change-Id: I0fa57b02694221507885ab4bfdb5805e936450e2
    Fixes: bug #1209446

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/41177
Committed: http://github.com/openstack/nova/commit/2d20b87aef5a7d00cb36826b8907032912fb17fc
Submitter: Jenkins
Branch: master

commit 2d20b87aef5a7d00cb36826b8907032912fb17fc
Author: Matt Riedemann <email address hidden>
Date: Fri Aug 9 11:15:26 2013 -0700

    Handle port over-quota when allocating network for instance

    This patch adds a check in the neutron API for a port create failure due
    to over-quota in neutron and raises the new exception PortLimitExceeded.

    Also moves the port-create block of code into it's own method to
    try and clean up some of the large allocate_for_instance method.

    Closes-Bug: #1207914
    Related-Bug: #1209446

    Change-Id: I4000c8ab550e032363f138a86e1b87f6ab2f5ff2