PortLimit thrown when specifying used fixed ip

[Drew Thorstensen]

PortLimit thrown when specifying used fixed ip

The nova/network/neutronv2/api.py _create_port will catch errors thrown from the neutron client. The code asserts that a 409 error is an over-quota error. However, this will hide other errors that may be occurring within the system.

The code is currently taking a calculated risk by assuming that all 409 errors that come through this code path will be over quota's. However, another high traffic code path is specifying a fixed_ip. If a user specifies a fixed ip address, this code will now incorrectly throw a PortLimitExceeded error. This leads the users to believe that they have run out of their quota limit.

Example exception (then wrapped exception):

NV-6FC38FD Neutron error creating port on network 7d360984-e12c-4bb3-819d-4b93c4ca4269
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 182, in _create_port
    port_id = port_client.create_port(port_req_body)['port']['id']
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 108, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 308, in create_port
    return self.post(self.ports_path, body=body)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1188, in post
    headers=headers, params=params)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1111, in do_request
    self._handle_fault_response(status_code, replybody)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1081, in _handle_fault_response
    exception_handler_v20(status_code, des_error_body)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
    message=msg)
NeutronClientException: 409-{u'NeutronError': {u'message': u'Unable to complete operation for network 7d360984-e12c-4bb3-819d-4b93c4ca4269. The IP address 10.0.0.2 is in use.', u'type': u'IpAddressInUse', u'detail': u''}}

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1236, in _allocate_network_async
    dhcp_options=dhcp_options)
  File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 49, in wrapper
    res = f(self, context, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 358, in allocate_for_instance
    LOG.exception(msg, port_id)
  File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
    security_group_ids, available_macs, dhcp_opts))
  File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 191, in _create_port
    raise exception.PortLimitExceeded()
PortLimitExceeded: Maximum number of ports exceeded

[Matt Riedemann (mriedem)]

Patch is here: https://review.openstack.org/#/c/55165/

[Matt Riedemann (mriedem)]

So bug 1207914 affects python-neutronclient I think to make the exceptions coming out of the python-neutronclient more specific for the various 409 conflicts.

[Matt Riedemann (mriedem)]

OK, so you're getting an IpAddressInUse exception from neutron and the client isn't handling it specifically so it just sends a generic NeutronClientException with the 409 code, I get that.

Looking at the neutron client code, there is a mapping for handling PortInUse specifically:

https://github.com/openstack/python-neutronclient/blob/2.3.1/neutronclient/v2_0/client.py#L51

I think we could easily do the same for IpAddressInUse and then check for that in nova once a version of python-neutronclient is released that contains the new exception class (and nova can require that version).

It would push fixing this out 'the right way' for awhile, which is not fun and might require a workaround in the nova code (with a TODO in there to clean it up).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/55190

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/55190
Committed: http://github.com/openstack/python-neutronclient/commit/2dd8f7afe2480ff25d58b4e4021c4e562697a167
Submitter: Jenkins
Branch: master

commit 2dd8f7afe2480ff25d58b4e4021c4e562697a167
Author: Matt Riedemann <email address hidden>
Date: Mon Nov 4 14:05:09 2013 -0800

    Handle IpAddressInUse with client exception

    Nova is not able to reliably distinguish between port OverQuota 409
    errors and IpAddressInUse 409 errors when setting up networks for
    instances. The client needs to provide a more granular exception so
    nova can translate the error to it's own set of exceptions.

    Also adds a test case for the exception_handler_v20 utility method which
    was lacking explicit test coverage before this change.

    Partial-Bug: #1247844

    Change-Id: I71589097ae660c3385b3f4ff93ffae99ac7042b9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/79776
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7070e4a66f576b2e6e4a4f1121445200e7854942
Submitter: Jenkins
Branch: master

commit 7070e4a66f576b2e6e4a4f1121445200e7854942
Author: James Carey <email address hidden>
Date: Tue Apr 29 00:53:18 2014 +0000

    Support detection of fixed ip already in use

    Update the validate_networks method to detect when a fixed ip is
    already in use and raise the FixedIpAlreadyInUse exception

    Change-Id: I07f83ce654c909b1158315dfc21a40654bc34a47
    Closes-Bug: #1247844