PortLimitExceeded results in 500 error from POST /servers/{server_id}/os-interface

[Matt Riedemann]

This is based on code inspection at the moment, but attaching an interface (port) to a server by providing a network and having nova-compute create the port on that network can result in an over-quota failure from neutron:

https://github.com/openstack/nova/blob/6ebb2c4cae65cb437e17a8c02fe5174a9825d8e0/nova/network/neutronv2/api.py#L565

Which is not handled in the API controller:

https://github.com/openstack/nova/blob/6ebb2c4cae65cb437e17a8c02fe5174a9825d8e0/nova/api/openstack/compute/attach_interfaces.py#L168

So it will result in a 500 response from the API when it should really be something like 403.

[Matt Riedemann (mriedem)]

It should be relatively easy to recreate this in devstack by setting [quotas]/quota_port=1 in neutron.conf, create a server and then try to attach a new port to the server via the same network.

[Matt Riedemann (mriedem)]

Yup I was able to easily recreate this in devstack:

g [None req-124eeb28-2f0f-4b47-b2c0-3889d65dd931 admin admin] 199.204.45.68 "POST /compute/v2.1/servers/8e90f1f0-e8dd-4783-8bb3-ec8d594e60f1/os-interface" status: 500 len: 1
9.204.45.68 () {64 vars in 1385 bytes} [Thu Apr 4 18:23:37 2019] POST /compute/v2.1/servers/8e90f1f0-e8dd-4783-8bb3-ec8d594e60f1/os-interface => generated 198 bytes in 2046
 ESCOD
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/manager.py", line 201, in decorated_function
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi return function(self, context, *args, **kwargs)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/manager.py", line 6081, in attach_interface
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi bind_host_id=bind_host_id, tag=tag)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1697, in allocate_port_for_in
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi bind_host_id=bind_host_id, attach=True)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1106, in allocate_for_instanc
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi security_group_ids)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1013, in _create_ports_for_in
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi neutron, instance, created_port_ids)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi self.force_reraise()
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi six.reraise(self.type_, self.value, self.tb)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.a...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/650097

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/650097
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8ff0fae42bce9f98fd5897888c3c4c739b4b5f45
Submitter: Zuul
Branch: master

commit 8ff0fae42bce9f98fd5897888c3c4c739b4b5f45
Author: Matt Riedemann <email address hidden>
Date: Thu Apr 4 14:44:02 2019 -0400

    Handle PortLimitExceeded in POST /servers/{server_id}/os-interface

    When attaching an interface to a server, if an existing port is not
    specified, nova-compute will attempt to create a port on either the
    user-specified network or the network that is available to the tenant.

    If the tenant exceeds their port quota in neutron, a PortLimitExceeded
    exception is raised up from nova-compute [1] which is not being handled
    in the API controller code - which is fixed in this change.

    Note that this is one of the few synchronous RPC call operations [2]
    in the compute API so exceptions from the compute service will leak
    back to the API like in this case and need to be handled to avoid a
    500 response to the user.

    The 403 response used here matches how PortLimitExceeded is handled in
    the server create API [3].

    [1] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/network/neutronv2/api.py#L565
    [2] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/compute/rpcapi.py#L489
    [3] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/api/openstack/compute/servers.py#L688

    Change-Id: I5687480a22542eea31d299442837bd64bf731285
    Closes-Bug: #1823203

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/665513

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/665513
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=43926ebca0f9a7f59be96e965e5be70b84501fa3
Submitter: Zuul
Branch: stable/stein

commit 43926ebca0f9a7f59be96e965e5be70b84501fa3
Author: Matt Riedemann <email address hidden>
Date: Thu Apr 4 14:44:02 2019 -0400

    Handle PortLimitExceeded in POST /servers/{server_id}/os-interface

    When attaching an interface to a server, if an existing port is not
    specified, nova-compute will attempt to create a port on either the
    user-specified network or the network that is available to the tenant.

    If the tenant exceeds their port quota in neutron, a PortLimitExceeded
    exception is raised up from nova-compute [1] which is not being handled
    in the API controller code - which is fixed in this change.

    Note that this is one of the few synchronous RPC call operations [2]
    in the compute API so exceptions from the compute service will leak
    back to the API like in this case and need to be handled to avoid a
    500 response to the user.

    The 403 response used here matches how PortLimitExceeded is handled in
    the server create API [3].

    [1] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/network/neutronv2/api.py#L565
    [2] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/compute/rpcapi.py#L489
    [3] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/api/openstack/compute/servers.py#L688

    Change-Id: I5687480a22542eea31d299442837bd64bf731285
    Closes-Bug: #1823203
    (cherry picked from commit 8ff0fae42bce9f98fd5897888c3c4c739b4b5f45)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.