PortLimitExceeded results in 500 error from POST /servers/{server_id}/os-interface

Bug #1823203 reported by Matt Riedemann on 2019-04-04
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Matt Riedemann
Pike
Medium
Unassigned
Queens
Medium
Unassigned
Rocky
Medium
Unassigned
Stein
Medium
Matt Riedemann

Bug Description

This is based on code inspection at the moment, but attaching an interface (port) to a server by providing a network and having nova-compute create the port on that network can result in an over-quota failure from neutron:

https://github.com/openstack/nova/blob/6ebb2c4cae65cb437e17a8c02fe5174a9825d8e0/nova/network/neutronv2/api.py#L565

Which is not handled in the API controller:

https://github.com/openstack/nova/blob/6ebb2c4cae65cb437e17a8c02fe5174a9825d8e0/nova/api/openstack/compute/attach_interfaces.py#L168

So it will result in a 500 response from the API when it should really be something like 403.

Matt Riedemann (mriedem) wrote :

It should be relatively easy to recreate this in devstack by setting [quotas]/quota_port=1 in neutron.conf, create a server and then try to attach a new port to the server via the same network.

Matt Riedemann (mriedem) wrote :
Download full text (4.7 KiB)

Yup I was able to easily recreate this in devstack:

g [None req-124eeb28-2f0f-4b47-b2c0-3889d65dd931 admin admin] 199.204.45.68 "POST /compute/v2.1/servers/8e90f1f0-e8dd-4783-8bb3-ec8d594e60f1/os-interface" status: 500 len: 1
9.204.45.68 () {64 vars in 1385 bytes} [Thu Apr 4 18:23:37 2019] POST /compute/v2.1/servers/8e90f1f0-e8dd-4783-8bb3-ec8d594e60f1/os-interface => generated 198 bytes in 2046
 ESCOD
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/manager.py", line 201, in decorated_function
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi return function(self, context, *args, **kwargs)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/manager.py", line 6081, in attach_interface
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi bind_host_id=bind_host_id, tag=tag)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1697, in allocate_port_for_in
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi bind_host_id=bind_host_id, attach=True)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1106, in allocate_for_instanc
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi security_group_ids)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1013, in _create_ports_for_in
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi neutron, instance, created_port_ids)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi self.force_reraise()
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi six.reraise(self.type_, self.value, self.tb)
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.api.openstack.wsgi
Apr 04 18:23:39 train <email address hidden>[12332]: ERROR nova.a...

Read more...

Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
importance: Undecided → Medium
status: New → Confirmed

Fix proposed to branch: master
Review: https://review.openstack.org/650097

Changed in nova:
status: Confirmed → In Progress

Reviewed: https://review.opendev.org/650097
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8ff0fae42bce9f98fd5897888c3c4c739b4b5f45
Submitter: Zuul
Branch: master

commit 8ff0fae42bce9f98fd5897888c3c4c739b4b5f45
Author: Matt Riedemann <email address hidden>
Date: Thu Apr 4 14:44:02 2019 -0400

    Handle PortLimitExceeded in POST /servers/{server_id}/os-interface

    When attaching an interface to a server, if an existing port is not
    specified, nova-compute will attempt to create a port on either the
    user-specified network or the network that is available to the tenant.

    If the tenant exceeds their port quota in neutron, a PortLimitExceeded
    exception is raised up from nova-compute [1] which is not being handled
    in the API controller code - which is fixed in this change.

    Note that this is one of the few synchronous RPC call operations [2]
    in the compute API so exceptions from the compute service will leak
    back to the API like in this case and need to be handled to avoid a
    500 response to the user.

    The 403 response used here matches how PortLimitExceeded is handled in
    the server create API [3].

    [1] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/network/neutronv2/api.py#L565
    [2] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/compute/rpcapi.py#L489
    [3] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/api/openstack/compute/servers.py#L688

    Change-Id: I5687480a22542eea31d299442837bd64bf731285
    Closes-Bug: #1823203

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/665513
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=43926ebca0f9a7f59be96e965e5be70b84501fa3
Submitter: Zuul
Branch: stable/stein

commit 43926ebca0f9a7f59be96e965e5be70b84501fa3
Author: Matt Riedemann <email address hidden>
Date: Thu Apr 4 14:44:02 2019 -0400

    Handle PortLimitExceeded in POST /servers/{server_id}/os-interface

    When attaching an interface to a server, if an existing port is not
    specified, nova-compute will attempt to create a port on either the
    user-specified network or the network that is available to the tenant.

    If the tenant exceeds their port quota in neutron, a PortLimitExceeded
    exception is raised up from nova-compute [1] which is not being handled
    in the API controller code - which is fixed in this change.

    Note that this is one of the few synchronous RPC call operations [2]
    in the compute API so exceptions from the compute service will leak
    back to the API like in this case and need to be handled to avoid a
    500 response to the user.

    The 403 response used here matches how PortLimitExceeded is handled in
    the server create API [3].

    [1] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/network/neutronv2/api.py#L565
    [2] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/compute/rpcapi.py#L489
    [3] https://github.com/openstack/nova/blob/6ebb2c4ca/nova/api/openstack/compute/servers.py#L688

    Change-Id: I5687480a22542eea31d299442837bd64bf731285
    Closes-Bug: #1823203
    (cherry picked from commit 8ff0fae42bce9f98fd5897888c3c4c739b4b5f45)

This issue was fixed in the openstack/nova 19.0.2 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers