instance ingress bandwidth limiting doesn't works in ocata.

[Jorge Niedbalski]

[Environment]

Xenial-Ocata deployment

[Description]

The instance ingress bandwidth limit implementation was targeted for Ocata [0], but the full implementation ingress/egress was done during the pike [1] cycle.

However, isn't reported or explicit that ingress direction isn't supported in ocata, which causes the following exception when --ingress is specified.

It would be desirable for this feature to be available on Ocata for being able to
set ingress/egress bandwidth limits on the ports.

[Testing]

Without these patches, trying to set a ingress bandwidth-limit rule
the following exception will be raised.

$ openstack network qos rule create --type bandwidth-limit --max-kbps 300 --max-burst-kbits 300 --ingress bw-limiter
Failed to create Network QoS rule: BadRequestException: 400: Client Error for url: https://openstack:9696/v2.0/qos/policies/xxxx/bandwidth_limit_rules, Unrecognized attribute(s) 'direction'

A single policy set (without the --ingress parameter) as supported in Ocata will just create a limiter on the egress side.

1) Check the policy list

$ openstack network qos policy list
+--------------------------------------+------------+--------+---------+----------------------------------+
| ID | Name | Shared | Default | Project |
+--------------------------------------+------------+--------+---------+----------------------------------+
| 2c9c85e2-4b65-4146-b7bf-47895379c938 | bw-limiter | False | None | c45b1c0a681d4d9788f911e29166056d |
+--------------------------------------+------------+--------+---------+----------------------------------+

2) Check that the qoes rule is set to 300 kbps.

$ openstack network qos rule list 2c9c85e2-4b65-4146-b7bf-47895379c938

| 01eb228d-5803-4095-9e8e-f13d4312b2ef | 2c9c85e2-4b65-4146-b7bf-47895379c938 | bandwidth_limit | 300 | 300 | | | |

3) Set the Qos policy on any port.

$ openstack port set 9a74b3c8-9ed8-4670-ad1f-932febfcf059 --qos-policy 2c9c85e2-4b65-4146-b7bf-47895379c938

$ openstack port show 9a74b3c8-9ed8-4670-ad1f-932febfcf059 | grep qos
| qos_policy_id | 2c9c85e2-4b65-4146-b7bf-47895379c938 |

4) Check that the egress traffic rules have been applied

# iperf3 -c 192.168.21.9 -t 10
Connecting to host 192.168.21.9, port 5201
[ 4] local 192.168.21.3 port 34528 connected to 192.168.21.9 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 121 KBytes 988 Kbits/sec 23 2.44 KBytes
[ 4] 7.00-8.00 sec 40.2 KBytes 330 Kbits/sec 14 3.66 KBytes
[ 4] 8.00-9.00 sec 36.6 KBytes 299 Kbits/sec 15 2.44 KBytes
[ 4] 9.00-10.00 sec 39.0 KBytes 320 Kbits/sec 18 3.66 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 435 KBytes 356 Kbits/sec 159 sender
[ 4] 0.00-10.00 sec 384 KBytes 314 Kbits/sec receiver

iperf Done.

5) Check that no ingress traffic limit has been applied.

# iperf3 -c 192.168.21.9 -R -t 10
Connecting to host 192.168.21.9, port 5201
Reverse mode, remote host 192.168.21.9 is sending
[ 4] local 192.168.21.3 port 34524 connected to 192.168.21.9 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 38.1 MBytes 319 Mbits/sec
[ 4] 8.00-9.00 sec 74.6 MBytes 626 Mbits/sec
[ 4] 9.00-10.00 sec 73.2 MBytes 614 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.07 GBytes 918 Mbits/sec 1045 sender
[ 4] 0.00-10.00 sec 1.07 GBytes 916 Mbits/sec receiver

--->

6) With the patches applied from the PPA or proposed, run the migration steps on the neutron-api node, repeat the previous steps, but make sure to specify the traffic direction with --ingress as follows:

$ openstack network qos rule create --type bandwidth-limit --max-kbps 300 --ingress testing-policy
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| direction | ingress |
| id | 6d01cefa-0042-40cd-ae74-bcb723ca7ca4 |
| max_burst_kbps | 0 |
| max_kbps | 300 |
| name | None |
| project_id | |
+----------------+--------------------------------------+

7) Set the policy into any server port.

$ openstack port set 50b8f714-3ee4-4260-8359-820420471bdb --qos-policy fac2be5e-64e0-4308-b477-0f8c0096c0b8

8) Check that the policy has been applied

$ openstack port show 50b8f714-3ee4-4260-8359-820420471bdb | grep qos
| qos_policy_id | fac2be5e-64e0-4308-b477-0f8c0096c0b8 |

9) Check that the rules have the "Direction field"

$ openstack network qos rule list fac2be5e-64e0-4308-b477-0f8c0096c0b8
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+
| ID | QoS Policy ID | Type | Max Kbps | Max Burst Kbits | Min Kbps | DSCP mark | Direction |
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+
| 6d01cefa-0042-40cd-ae74-bcb723ca7ca4 | fac2be5e-64e0-4308-b477-0f8c0096c0b8 | bandwidth_limit | 300 | 0 | | | ingress |
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+

10) Validate that the hypervisor has a ovs queue and qos setup set, max-rate should be set to 300000.

$ sudo ovs-vsctl list qos
_uuid : 2862acaa-a79f-4a62-b387-f89bca4a6965
external_ids : {id="tap75d95386-01"}
other_config : {}
queues : {0=a15774f7-ae31-40e7-9c0f-937d68a7dc0c}
type : linux-htb
root@juju-cd6736-1841700-8:/home/ubuntu# sudo ovs-vsctl list queue
_uuid : a15774f7-ae31-40e7-9c0f-937d68a7dc0c
dscp : []
external_ids : {id="tap75d95386-01", queue_type="0"}
other_config : {burst="0", max-rate="300000"}

[0] https://blueprints.launchpad.net/neutron/+spec/instance-ingress-bw-limit
[1] https://bugs.launchpad.net/neutron/+bug/1560961

[Jorge Niedbalski (niedbalski)]

### Notes ####

The following is an example of how ingress/egress rules are applied
on > pike

130 ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ neutron qos-policy-create bw-limiter
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Created a new policy:
+-----------------+--------------------------------------+
| Field | Value |
+-----------------+--------------------------------------+
| created_at | 2019-08-23T17:10:52Z |
| description | |
| id | 9847ba6f-eca2-4d9f-8fdf-9a914b8cea12 |
| is_default | False |
| name | bw-limiter |
| project_id | 8280dd071ff74caf90a7a32e0d942da4 |
| revision_number | 0 |
| rules | |
| shared | False |
| tags | |
| tenant_id | 8280dd071ff74caf90a7a32e0d942da4 |
| updated_at | 2019-08-23T17:10:52Z |
+-----------------+--------------------------------------+

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ neutron qos-bandwidth-limit-rule-create bw-limiter --max-kbps 300 --max-burst-kbps 300
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Created a new bandwidth_limit_rule:
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| direction | egress |
| id | 749a6867-9cb9-421a-a53d-107abf57f914 |
| max_burst_kbps | 300 |
| max_kbps | 300 |
+----------------+--------------------------------------+

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ openstack network qos policy list
+--------------------------------------+------------+--------+---------+----------------------------------+
| ID | Name | Shared | Default | Project |
+--------------------------------------+------------+--------+---------+----------------------------------+
| 9847ba6f-eca2-4d9f-8fdf-9a914b8cea12 | bw-limiter | False | False | 8280dd071ff74caf90a7a32e0d942da4 |
+--------------------------------------+------------+--------+---------+----------------------------------+

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ openstack network qos rule list 9847ba6f-eca2-4d9f-8fdf-9a914b8cea12
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+
| ID | QoS Policy ID | Type | Max Kbps | Max Burst Kbits | Min Kbps | DSCP mark | Direction |
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+--...

[Jorge Niedbalski (niedbalski)]

### Notes ####

The following session shows how only egress rules are applied in ocata.

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ openstack network qos policy list
+--------------------------------------+------------+--------+---------+----------------------------------+
| ID | Name | Shared | Default | Project |
+--------------------------------------+------------+--------+---------+----------------------------------+
| 2c9c85e2-4b65-4146-b7bf-47895379c938 | bw-limiter | False | None | c45b1c0a681d4d9788f911e29166056d |
+--------------------------------------+------------+--------+---------+----------------------------------+

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ openstack network qos rule list 2c9c85e2-4b65-4146-b7bf-47895379c938
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+
| ID | QoS Policy ID | Type | Max Kbps | Max Burst Kbits | Min Kbps | DSCP mark | Direction |
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+
| 01eb228d-5803-4095-9e8e-f13d4312b2ef | 2c9c85e2-4b65-4146-b7bf-47895379c938 | bandwidth_limit | 300 | 300 | | | |
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ openstack port list
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+
| 89df124f-6198-4780-be7c-4b60f8825b66 | | fa:16:3e:5e:4d:9e | ip_address='192.168.21.1', subnet_id='f2b575c6-7def-46e6-8895-832975eaef87' | ACTIVE |
| 9a74b3c8-9ed8-4670-ad1f-932febfcf059 | | fa:16:3e:9a:a3:98 | ip_address='192.168.21.7', subnet_id='f2b575c6-7def-46e6-8895-832975eaef87' | ACTIVE |
| b39f6d04-dd4b-4a08-88af-0061541463de | | fa:16:3e:dc:39:fd | ip_address='10.5.150.9', subnet_id='41fe0291-44ae-48dd-9154-d0fb93011312' | ACTIVE |
| ca8c0063-7485-4a26-bd1b-68b22daa4534 | | fa:16:3e:6d:fc:d2 | ip_address='192.168.21.12', subnet_id='f2b575c6-7def-46e6-8895-832975eaef87' | ACTIVE |
| d93c186b-9792-4c6b-9455-fc01415d9393 | | fa:16:3e:fe:e8:9e | ip_address='192.168.21.2', subnet_id='f2b575c6-7def-46e6-8895-832975eaef87' | ACTIVE |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ openst...

[Jorge Niedbalski (niedbalski)]

### Notes ###

As further reference, I've built a PPA that addresses this bug, which includes
a backport of the following changes:

  * Backport of ingress bandwidth limits fix (LP: #1560961).

    - d/p/0001-Add-QoS-bandwidth-limit-for-instance-ingress-traffic.patch
    - d/p/0002-Change-duplicate-OVS-bridge-datapath-ids.patch
    - d/p/0003-Add-support-for-ingress-bandwidth-limit-rules-in-ovs.patch
    - d/p/0004-Ingress-bandwidth-limit-rule-in-Linuxbridge-agent.patch

PPA for testing can be found here: https://launchpad.net/~niedbalski/+archive/ubuntu/hf-00237904

[Lajos Katona (lajos-katona)]

Hi,
Thanks for the report. Feature backports are not allowed by the backport policy of Openstack (see: https://docs.openstack.org/murano/pike/contributor/stable_branches.html), and stable/ocata is near to end-of-lifecycle.

[Lajos Katona (lajos-katona)]

Due to the above, I state this bug as invalid. If you need further questions don't hesitate to ask on #openstack-neutron irc channel or on openstack-discuss mailing list.

[Jorge Niedbalski (niedbalski)]

Hello,

I've tested the PPA on https://launchpad.net/~niedbalski/+archive/ubuntu/hf-00237904/ with xenial-ocata deployed cloud.

Please note that it's a requirement to run the migrations steps up to heads.

ubuntu@niedbalski-bastion:~/octavia/openstack$ juju config neutron-api enable-qos=True

| 50b8f714-3ee4-4260-8359-820420471bdb | | fa:16:3e:88:c1:a0 | ip_address='10.5.150.6', subnet_id='7daca73f-31ab-4401-bc6e-e74dd38b7fc1' | N/A |

ubuntu@niedbalski-bastion:~/octavia/openstack$ openstack network qos policy create testing-policy --share
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| description | |
| id | fac2be5e-64e0-4308-b477-0f8c0096c0b8 |
| is_default | None |
| name | testing-policy |
| project_id | d215199a727a4384ad7d43825e86ff69 |
| rules | [] |
| shared | True |
| tags | [] |
+-------------+--------------------------------------+

ubuntu@niedbalski-bastion:~/octavia/openstack$ openstack network qos rule create --type bandwidth-limit --max-kbps 300 --ingress testing-policy
Failed to create Network QoS rule: BadRequestException: 400: Client Error for url: http://10.5.0.66:9696/v2.0/qos/policies/fac2be5e-64e0-4308-b477-0f8c0096c0b8/bandwidth_limit_rules, {"NeutronError": {"message": "Unrecognized attribute(s) 'direction'", "type": "HTTPBadRequest", "detail": ""}}

----> With the patch applied

neutron-api

root@juju-cd6736-1841700-4:/home/ubuntu# systemctl status neutron*
● neutron-server.service - OpenStack Neutron Server
   Loaded: loaded (/lib/systemd/system/neutron-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-09-09 15:47:00 UTC; 1min 39s ago
  Process: 13597 ExecStartPre=/bin/chown neutron:adm /var/log/neutron (code=exited, status=0/SUCCESS)

ubuntu@juju-cd6736-1841700-4:~$ sudo su
root@juju-cd6736-1841700-4:/home/ubuntu# dpkg -l |grep neutron
ii neutron-common 2:10.0.7-0ubuntu1~cloud1ubuntu1+hf1560961v20190826.13 all Neutron is a virtual network service for Openstack - common
ii neutron-plugin-ml2 2:10.0.7-0ubuntu1~cloud1ubuntu1+hf1560961v20190826.13 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:10.0.7-0ubuntu1~cloud1ubuntu1+hf1560961v20190826.13 all Neutron is a virtual network service for Openstack - server
ii python-neutron 2:10.0.7-0ubuntu1~cloud1ubuntu1+hf1560961v20190826.13 all Neutron is a virtual network service for Openstack - Python library

neutron-gateway

root@juju-cd6736-1841700-5:/home/ubuntu# dpkg -l | grep neutron
ii neutron-common 2:10.0.7-0ubuntu1~cloud1ubuntu1+hf1560961v20190826.13 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:10.0.7-0ubuntu1~cloud1ubuntu1+hf1560961v20190...

[Jorge Niedbalski (niedbalski)]

Waiting on OE approval.

* Testing cases comments have been updated, the xenial-ocata debdiff patch is on the previous comment.

[Jorge Niedbalski (niedbalski)]

Test results will be collected through https://docs.google.com/spreadsheets/d/1ajozctmyHAKEBlG2Aet-Zdv7v9uFcRlIRiW5KeOk778/edit#gid=0 , please feel free to add any missing test case on that document.

[Corey Bryant (corey.bryant)]

Hello Jorge, or anyone else affected,

Accepted neutron into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Jorge Niedbalski (niedbalski)]

I have tested the different upgrade scenarios outlayed on the spreadsheet
linked to this case.

I found an error with existing qos policies when upgrading from pike to ocata-patched
which might require a further change on the patchset.

I will mark this verification as failed and re-submit a new patch with the
required bits.

The results with all other combinations are shared next.

Bundle deployed: http://paste.ubuntu.com/p/tMNX577zCk/

Steps executed:

juju config neutron-api enable-qos=true

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ openstack network qos policy create bw-limit
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| description | |
| id | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 |
| is_default | None |
| name | bw-limit |
| project_id | 2c16c3a423444a43a39e11fcc768ad22 |
| rules | [] |
| shared | False |
+-------------+--------------------------------------+

openstack network qos rule create --type bandwidth-limit --max-kbps 300 --max-burst-kbits 300 --ingress bw-limit
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| direction | ingress |
| id | eda4481f-61d0-4f52-91f7-fe979f776705 |
| max_burst_kbps | 300 |
| max_kbps | 300 |
| name | None |
| project_id | |
+----------------+--------------------------------------+

openstack network qos rule list bw-limit
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+
| ID | QoS Policy ID | Type | Max Kbps | Max Burst Kbits | Min Kbps | DSCP mark | Direction |
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+-----------+-----------+
| eda4481f-61d0-4f52-91f7-fe979f776705 | 7e6d0652-6cd4-4d5c-b58e-b7664c1c4587 | bandwidth_limit | 300 | 300 | | | ingress |
+--------------------------------------+--------------------------------------+-----------------+----------+-----------------+----------+----------

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ openstack port list -f value | grep 10.5.150.5
7c010ca8-1e96-4419-a4e9-4c56da05c806 fa:16:3e:a1:38:50 ip_address='10.5.150.5', subnet_id='5bb6de3b-6c72-4ef5-a0c2-821dfafaa822' N/A

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ openstack port list -f value | grep 10.5.150.0
8c44909b-a4cf-4c7b-903f-f9f31f4a8045 fa:16:3e:4d:53:4e ip_address='10.5.150.0', subnet_id='5bb6de3b-6c72-4ef5-a0c2-82...

[Corey Bryant (corey.bryant)]

@jorge, can I revert the patches that are in ocata-proposed for this?

[Jorge Niedbalski (niedbalski)]

@corey.bryant, Yes corey, please go ahead. If this issue re-surfaces again, i'll provide you a new proposal that fixes the case mentioned before.

[Corey Bryant (corey.bryant)]

@jorge, thanks. I've uploaded neutron 2:10.0.7-0ubuntu1~cloud3 which will revert them.

[Corey Bryant (corey.bryant)]

Hello Jorge, or anyone else affected,

Accepted neutron into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Switching back to verification-ocata-failed since it did in fact fail. We'll leave it in that state.

[Corey Bryant (corey.bryant)]

The reverted package has passed regression testing:

======
Totals
======
Ran: 94 tests in 895.3082 sec.
 - Passed: 89
 - Skipped: 5
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 517.6046 sec.

I'm going to move this bug to Incomplete for now, since the fix has been reverted.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:10.0.7-0ubuntu1~cloud3
---------------

 neutron (2:10.0.7-0ubuntu1~cloud3) xenial-ocata; urgency=medium
 .
   * Revert backport of ingress bandwidth limits fix (LP: #1841700). The
     following patches are dropped.
     - d/p/0001-Change-duplicate-OVS-bridge-datapath-ids.patch
     - d/p/0002-Add-QoS-bandwidth-limit-for-instance-ingress-traffic.patch
     - d/p/0003-Add-support-for-ingress-bandwidth-limit-rules-in-ovs.patch
     - d/p/0004-Ingress-bandwidth-limit-rule-in-Linuxbridge-agent.patch
     - d/p/0005-Extend-QoS-L2-drivers-interface-to-handle-ingress-ru.patch
     - d/p/0006-add-missing-migration.patch

[Corey Bryant (corey.bryant)]

neutron 2:10.0.7-0ubuntu1~cloud3 has been promoted to ocata-updates