allocation key is missing from the binding:profile of the neutron qos port when the server is created by a non-admin user

[Balazs Gibizer]

Description
===========
When a server is create by a non-admin tenant with a qos neutron port Nova does not add the allocation key to the binding:profile of the port.

Steps to reproduce
==================
1) Set up a devstack with bandwidth inventory
   * sudo ovs-vsctl add-br br-test
   * devstack local conf:
[[post-config|/etc/neutron/neutron.conf]]
[DEFAULT]
service_plugins = router, placement, qos

[[post-config|/etc/neutron/plugins/ml2/ml2_conf.ini]]
[ml2]
extension_drivers = port_security,qos
mechanism_drivers = openvswitch
tenant_network_types = vxlan

[ml2_type_vlan]
network_vlan_ranges = physnet0:1000:2000
[ovs]
bridge_mappings = public:br-ex,physnet0:br-test
resource_provider_bandwidths = br-test:5000:5000
[ovs_driver]
vnic_type_blacklist = direct

   * stack.sh

2) As admin user set up a network and a qos policy:

   * openstack network create net-demo --provider-network-type vlan --provider-physical-network physnet0 --provider-segment 101 --share
   * openstack subnet create subnet-demo --network net-demo --subnet-range 10.0.4.0/24
   * openstack network qos policy create qp-demo --share
   * openstack network qos rule create qp-demo --type minimum-bandwidth --min-kbps 1000 --egress
   * openstack network qos rule create qp-demo --type minimum-bandwidth --min-kbps 1000 --ingress

3) As a normal user (demo in devstack) create a port with the qos policy and create a server with the port

   * openstack port create port-normal-qos-demo --network net-demo --vnic-type normal --qos-policy qp-demo
   * openstack --os-compute-api-version 2.72 server create --image cirros-0.4.0-x86_64-disk --flavor c1 --nic port-id=port-normal-qos-demo vm-demo --wait

Expected result
===============

1) Server is reaching ACTIVE state
2) Bandwidth allocation is created in placement according to the qp-demo policy
3) The allocation key of the binding:profile of the port-normal-qos-demo port contains the UUID of the placement resource provider from where the bandwidth resource is allocated from.

Actual result
=============
1) and 2) are as expected but the binding:porfile of the neutron port does not have an allocation key.

Note that if the server is booted as admin user then both 1) 2) 3) are as expected.

Environment
===========
Devstack from master:

stack@aio:/opt/stack/nova$ git log --oneline | head -1
d3403e5294 Merge "Fix unit of hw_rng:rate_period"

stack@aio:/opt/stack/neutron$ git log --oneline | head -1
2ffaa40b43 Merge "ovsdb monitor: handle modified ports"

Triage
======
Looking at the port-normal-qos-demo port from the demo user. The resource_request filed of the port is None. While looking at the port from the admin user the resource_request field is properly filled according to the qos policy of the port.

As demo:

stack@aio:~$ openstack port show port-normal-qos-demo
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | None |
| binding_profile | None |
| binding_vif_details | None |
| binding_vif_type | None |
| binding_vnic_type | normal |
| created_at | 2019-10-24T11:05:27Z |
| data_plane_status | None |
| description | |
| device_id | |
| device_owner | |
| dns_assignment | None |
| dns_domain | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='10.0.4.213', subnet_id='736636a1-114f-4d9e-9e8f-82568593061d' |
| id | b1593c18-b088-4d5c-b3c6-bdd5348f3b52 |
| location | cloud='', project.domain_id='default', project.domain_name=, project.id='05c189206e0d4e3d8ea95f1a4067b420', project.name='demo', region_name='RegionOne', zone= |
| mac_address | fa:16:3e:8a:56:8a |
| name | port-normal-qos-demo |
| network_id | f32506ae-4cf4-414a-8349-09fc744f024c |
| port_security_enabled | True |
| project_id | 05c189206e0d4e3d8ea95f1a4067b420 |
| propagate_uplink_status | None |
| qos_policy_id | f774c8b2-d302-427b-bdef-bd0614a0fbaa |
| resource_request | None |
| revision_number | 1 |
| security_group_ids | fb5f417f-727b-48c9-881a-21571450ae06 |
| status | DOWN |
| tags | |
| trunk_details | None |
| updated_at | 2019-10-24T11:05:27Z |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+

As admin:

stack@aio:~$ openstack port show port-normal-qos-demo
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | aio |
| binding_profile | |
| binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' |
| binding_vif_type | ovs |
| binding_vnic_type | normal |
| created_at | 2019-10-24T11:05:27Z |
| data_plane_status | None |
| description | |
| device_id | f60856cc-9817-4fd9-a89f-c87c93fc729b |
| device_owner | compute:nova |
| dns_assignment | None |
| dns_domain | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='10.0.4.213', subnet_id='736636a1-114f-4d9e-9e8f-82568593061d' |
| id | b1593c18-b088-4d5c-b3c6-bdd5348f3b52 |
| location | cloud='', project.domain_id=, project.domain_name=, project.id='05c189206e0d4e3d8ea95f1a4067b420', project.name=, region_name='RegionOne', zone= |
| mac_address | fa:16:3e:8a:56:8a |
| name | port-normal-qos-demo |
| network_id | f32506ae-4cf4-414a-8349-09fc744f024c |
| port_security_enabled | True |
| project_id | 05c189206e0d4e3d8ea95f1a4067b420 |
| propagate_uplink_status | None |
| qos_policy_id | f774c8b2-d302-427b-bdef-bd0614a0fbaa |
| resource_request | {u'required': [u'CUSTOM_PHYSNET_PHYSNET0', u'CUSTOM_VNIC_TYPE_NORMAL'], u'resources': {u'NET_BW_EGR_KILOBIT_PER_SEC': 1000, u'NET_BW_IGR_KILOBIT_PER_SEC': 1000}} |
| revision_number | 4 |
| security_group_ids | fb5f417f-727b-48c9-881a-21571450ae06 |
| status | ACTIVE |
| tags | |
| trunk_details | None |
| updated_at | 2019-10-24T11:14:46Z |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+

The placement allocation is correct:

stack@aio:~$ openstack --os-placement-api-version 1.22 resource provider allocation show f60856cc-9817-4fd9-a89f-c87c93fc729b
+--------------------------------------+------------+----------------------------------------------------------------------------+----------------------------------+----------------------------------+
| resource_provider | generation | resources | project_id | user_id |
+--------------------------------------+------------+----------------------------------------------------------------------------+----------------------------------+----------------------------------+
| 1110cf59-cabf-526c-bacc-08baabbac692 | 9 | {u'NET_BW_EGR_KILOBIT_PER_SEC': 1000, u'NET_BW_IGR_KILOBIT_PER_SEC': 1000} | 05c189206e0d4e3d8ea95f1a4067b420 | 2d0cf2d4e46348fda2c6d47d0e619544 |
| 40f2860b-f0b4-4325-af91-011d374c8aba | 15 | {u'VCPU': 1, u'MEMORY_MB': 256, u'DISK_GB': 1} | 05c189206e0d4e3d8ea95f1a4067b420 | 2d0cf2d4e46348fda2c6d47d0e619544 |
+--------------------------------------+------------+----------------------------------------------------------------------------+----------------------------------+----------------------------------+

So Nova was able to gather the resource_request field from the Neutron port with admin (service) credentials before the scheduling. But Nova failed to include the allocation key to the same port. I assume that this happens because Nova only adds allocation key to the port if the port has resource_request [1] but Nova checks the port at [1] with the user credentials not with the service credentials.

[1] https://github.com/openstack/nova/blob/1bfa4626d13d0a73e63745cc4a864ae86d490daf/nova/network/neutronv2/api.py#L998

[Balazs Gibizer (balazs-gibizer)]

So during server create nova collects the ports' resource_request at [1] using and admin client. When nova updates the binding profile it it uses a non admin client [2] so here nova does not see that the port has resource request. This is the root cause of the bug.

[1] https://github.com/openstack/nova/blob/1bfa4626d13d0a73e63745cc4a864ae86d490daf/nova/network/neutronv2/api.py#L1991
[2] https://github.com/openstack/nova/blob/1bfa4626d13d0a73e63745cc4a864ae86d490daf/nova/network/neutronv2/api.py#L966

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/690999

[Balazs Gibizer (balazs-gibizer)]

This needs to be backported til Stein as we added the support for boot such server in Stein.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/693248

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/690999
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=aab4b7a0e2504c04e08389145bcb1414dea63631
Submitter: Zuul
Branch: master

commit aab4b7a0e2504c04e08389145bcb1414dea63631
Author: Balazs Gibizer <email address hidden>
Date: Thu Oct 24 17:01:02 2019 +0200

    Use admin neutron client to query ports for binding

    The compute service updates the binding:profile of the neutron port
    during server create. If the port has resource_request then the
    'allocation' key need to point to the resource provider the port is
    allocating resources. Unfortunately this code used a non admin client to
    query the port data and therefore if the original server create request
    was sent by a non admin user the returned port does not have its
    resource_request filled and as a consequence nova does not add the
    allocation key to the binding profile.

    This patch makes sure that the port is queried with an admin client.

    There is a tempest test change that reproduces the issue:
    https://review.opendev.org/#/c/690934

    Change-Id: Icc631cf2e81a5c78cb7fb1d0b625d19bd8f5a274
    Closes-Bug: #1849657

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/694013

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694013
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b6989836dde792dfbde2bbc8b4fe8d4cfb448f47
Submitter: Zuul
Branch: stable/train

commit b6989836dde792dfbde2bbc8b4fe8d4cfb448f47
Author: Balazs Gibizer <email address hidden>
Date: Thu Oct 24 17:01:02 2019 +0200

    Use admin neutron client to query ports for binding

    The compute service updates the binding:profile of the neutron port
    during server create. If the port has resource_request then the
    'allocation' key need to point to the resource provider the port is
    allocating resources. Unfortunately this code used a non admin client to
    query the port data and therefore if the original server create request
    was sent by a non admin user the returned port does not have its
    resource_request filled and as a consequence nova does not add the
    allocation key to the binding profile.

    This patch makes sure that the port is queried with an admin client.

    There is a tempest test change that reproduces the issue:
    https://review.opendev.org/#/c/690934

    Change-Id: Icc631cf2e81a5c78cb7fb1d0b625d19bd8f5a274
    Closes-Bug: #1849657
    (cherry picked from commit aab4b7a0e2504c04e08389145bcb1414dea63631)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/694665

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694665
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b1ae940ddb823fec6b8e1ab33c75ef94760f438d
Submitter: Zuul
Branch: stable/stein

commit b1ae940ddb823fec6b8e1ab33c75ef94760f438d
Author: Balazs Gibizer <email address hidden>
Date: Thu Oct 24 17:01:02 2019 +0200

    Use admin neutron client to query ports for binding

    The compute service updates the binding:profile of the neutron port
    during server create. If the port has resource_request then the
    'allocation' key need to point to the resource provider the port is
    allocating resources. Unfortunately this code used a non admin client to
    query the port data and therefore if the original server create request
    was sent by a non admin user the returned port does not have its
    resource_request filled and as a consequence nova does not add the
    allocation key to the binding profile.

    This patch makes sure that the port is queried with an admin client.

    There is a tempest test change that reproduces the issue:
    https://review.opendev.org/#/c/690934

    Conflicts:
          nova/tests/unit/network/test_neutronv2.py
    Conflicts due to mox removal patches merged in train. I basically needed
    to redo the change in test_neutronv2.py as the merge conflict was huge.

    Change-Id: Icc631cf2e81a5c78cb7fb1d0b625d19bd8f5a274
    Closes-Bug: #1849657
    (cherry picked from commit aab4b7a0e2504c04e08389145bcb1414dea63631)
    (cherry picked from commit b6989836dde792dfbde2bbc8b4fe8d4cfb448f47)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 20.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.1.0 release.