tc filters appear set but live migration does not use

[Wendy Mitchell]

Brief Description
-----------------
traffic controls filters appear set on the management interface by design (ie. #1:30 = migration class) but the filter is not used in live migration

Severity
--------
Major

Steps to Reproduce
------------------

1. both cluster and mgmt interfaces configured:
cluster
ens801f1.95
mgmt
ens801f1.96

Traffic shaping (by design now) is set on mgmt interface now by design
eg. in this lab ens801f1.96

compute-0:/home/sysadmin# tc qdisc show dev ens801f1.96
qdisc noqueue 0: root refcnt 2

compute-0:/home/sysadmin# tc qdisc show dev ens801f1.95
qdisc htb 1: root refcnt 2 r2q 10 default 40 direct_packets_stat 0 direct_qlen 1000
qdisc sfq 40: parent 1:40 limit 127p quantum 1514b depth 127 divisor 1024 perturb 10sec
qdisc sfq 30: parent 1:30 limit 127p quantum 1514b depth 127 divisor 1024 perturb 10sec
qdisc sfq 20: parent 1:20 limit 127p quantum 1514b depth 127 divisor 1024 perturb 10sec
qdisc sfq 10: parent 1:10 limit 127p quantum 1514b depth 127 divisor 1024 perturb 10sec

2. An instance exists on the host and is running memory stress (eg. stress-ng)
3. Validate/watch traffic control filtering ie. check output for each class during a live migration of the instance
$ watch -n 1 tc -s class show dev <interface>
Ensuring that the live migration uses class eg. 1:30 and borrows from other classes

Expected Behavior
------------------
live migration was expected to use migration class eg. 1:30 and "borrow" from other classes

Actual Behavior
----------------
Not seeing the live migration class 1:30 actually used (and not seeing it borrow either)

ie. class htb 1:30 on the mgmt. interface (for the live migration) appears to have 0 sent and 0 borrowed

For example
$ tc -s class show dev ens802f1.95
class htb 1:1 root rate 10Gbit ceil 10Gbit burst 13750b cburst 0b
Sent 601881737217 bytes 551996833 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
lended: 3709269 borrowed: 0 giants: 0
tokens: 183 ctokens: 11

class htb 1:10 parent 1:1 leaf 10: prio 3 rate 1Gbit ceil 2Gbit burst 15125b cburst 1250b
Sent 55226436 bytes 827028 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
lended: 827028 borrowed: 0 giants: 0
tokens: 1897 ctokens: 88

class htb 1:20 parent 1:1 leaf 20: prio 1 rate 5Gbit ceil 10Gbit burst 15000b cburst 0b
Sent 240533937129 bytes 228964871 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
lended: 109895640 borrowed: 455177 giants: 0
tokens: 373 ctokens: 14

class htb 1:30 parent 1:1 leaf 30: prio 2 rate 3Gbit ceil 10Gbit burst 15000b cburst 0b
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
lended: 0 borrowed: 0 giants: 0
tokens: 625 ctokens: 15

class htb 1:40 parent 1:1 leaf 40: prio 4 rate 1Gbit ceil 2Gbit burst 15125b cburst 1250b
Sent 253230717752 bytes 245057753 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
lended: 117233721 borrowed: 2947769 giants: 0
tokens: 1869 ctokens: 74

class htb 1:50 parent 1:1 leaf 50: prio 0 rate 8Gbit ceil 10Gbit burst 14000b cburst 0b
Sent 108061855900 bytes 77147181 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
lended: 12165474 borrowed: 306323 giants: 0
tokens: 232 ctokens: 14

Reproducibility
---------------
yes

System Configuration
--------------------
HW lab (PV-1)

Branch/Pull Time/Commit
-----------------------
"20190731T013000Z"

Last Pass
---------

Timestamp/Logs
--------------
see instance c99e6eb5-ea64-4d87-aa56-6bdb72292e8e

log-id = 936
event-id = instance-live-migrating
event-type = action-event
event-context = admin
importance = high
entity = tenant=cb61ced9-022c-4533-89b1-297daa98214a.instance=c99e6eb5-ea64-4d87-aa56-6bdb72292e8e
reason_text = Live-Migrate inprogress for instance test from host compute-0
additional_text =
timestamp = 2019-08-07 19:30:27.585069

Test Activity
-------------
Regression test

[Brent Rowsell (brent-rowsell)]

Just confirming
cluster-host is ens801f1.95 ?

[Brent Rowsell (brent-rowsell)]

This also does not align with the interfaces at the top ?

tc -s class show dev ens802f1.95

[Wendy Mitchell (wmitchellwr)]

Apologies..
that was a typo in the steps. It should have read

$ tc -s class show dev ens801f1.95
or alternatively
$watch tc -s class show dev ens801f1.95

[Wendy Mitchell (wmitchellwr)]

Correction in step 1

cluster interface has vlan 96 (ens801f1.96)
management interface has vlan 95 (ens801f1.95)

[Wendy Mitchell (wmitchellwr)]

see also new live migration and attached logs

log-id = 940
event-id = instance-live-migrating
event-type = action-event
event-context = admin
importance = high
entity = tenant=cb61ced9-022c-4533-89b1-297daa98214a.instance=c99e6eb5-ea64-4d87-aa56-6bdb72292e8e
reason_text = Live-Migrate inprogress for instance test from host compute-0
additional_text =
timestamp = 2019-08-08 13:49:48.478930

2019-08-08T13:51:18.000 compute-0 -sh: info HISTORY: PID=2165945 UID=42425 watch tc -s class show dev ens801f1.95

$ tc -s class show dev ens801f1.95

class htb 1:1 root rate 10Gbit ceil 10Gbit burst 13750b cburst 0b
 Sent 4770349146 bytes 16958800 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 lended: 4977 borrowed: 0 giants: 0
 tokens: 185 ctokens: 13

class htb 1:10 parent 1:1 leaf 10: prio 3 rate 1Gbit ceil 2Gbit burst 15125b cbu
 Sent 65951024 bytes 247822 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 lended: 225118 borrowed: 126 giants: 0
 tokens: 1893 ctokens: 86

class htb 1:20 parent 1:1 leaf 20: prio 1 rate 5Gbit ceil 10Gbit burst 15000b cb
 Sent 435923866 bytes 586656 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 lended: 339027 borrowed: 3478 giants: 0
 tokens: 371 ctokens: 13

class htb 1:30 parent 1:1 leaf 30: prio 2 rate 3Gbit ceil 10Gbit burst 15000b cb
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 lended: 0 borrowed: 0 giants: 0
 tokens: 625 ctokens: 15

class htb 1:40 parent 1:1 leaf 40: prio 4 rate 1Gbit ceil 2Gbit burst 15125b cbu
 Sent 4268474256 bytes 16124322 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 lended: 16031838 borrowed: 1373 giants: 0
 tokens: 1869 ctokens: 74

[Brent Rowsell (brent-rowsell)]

Wendy, can you run the following tc -s class show dev ens801f1.96

[Brent Rowsell (brent-rowsell)]

Nova is using the cluster network for live migration.
Traffic controls are not explicitly set up for the cluster network. It will work if the cluster network is multi-netted with the mgmt on the same interface but not when the cluster is on it's own interface.

[Brent Rowsell (brent-rowsell)]

Also any traffic controls on the compute/aio should only be configured if openstack is deployed on it.

[Frank Miller (sensfan22)]

Marking as medium priority as this is a performance issue as opposed to a functional issue.

[Ghada Khalil (gkhalil)]

As per agreement with the community, moving all unresolved medium priority bugs from stx.2.0 to stx.3.0

[Ghada Khalil (gkhalil)]

Steve, can you have a look at this and check if the TC scripts are doing any shaping on the cluster network? As per discussion with Brent, it shouldn't

[Ghada Khalil (gkhalil)]

As per Matt Peters, there are no traffic controls on the cluster network. They are still applied on the mgmt network. The only case where TC would be applied for cluster traffic is when the cluster network is shared with the mgmt network w/o having its own vlan.

[Ghada Khalil (gkhalil)]

Regarding live migration, the traffic is now going over the cluster network which explains why no traffic controls are applied.

[Matt Peters (mpeters-wrs)]

The recommendation is to remove all the obsolete TC entries and just fix the heartbeat service traffic to be properly prioritized. The long-term plan for K8s traffic shaping is to use K8s network policies to apply Pod level traffic shaping and scheduling.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/691414

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/691415

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691414
Committed: https://git.openstack.org/cgit/starlingx/utilities/commit/?id=fc33582e4fdaa4d2725f2d7c23395a3e4c19daba
Submitter: Zuul
Branch: master

commit fc33582e4fdaa4d2725f2d7c23395a3e4c19daba
Author: Steven Webster <email address hidden>
Date: Wed Oct 23 09:33:39 2019 -0500

    Traffic control fixes and refresh

    This commit fixes a couple of issues surrounding the main StarlingX
    traffic control script. The following has been changed:

    - The name of the script has been changed to remove the 'cgcs_'
      prefix

    - Obsolete code involving the (old) infrastructure network and
      related consolidation with the management interface has been
      removed.

    - There is no more class/qdisc for live-migration, since that
      traffic now runs over the cluster-host network. It should
      be prioritized via a kubernetes network policy instead.
      (not in this commit)

    - The code has been cleaned up to define the handles and qdisc
      parameters globally so they can be changed once instead of
      in each tc command.

    - SM heartbeats are treated as high priority traffic. Previously,
      we were only treating a tos of 0x10 (low delay) as high priority
      traffic, while SM uses a DSCP value of 0xc0.

    - The script now handles IPv6

    Closes-Bug: #1839386
    Change-Id: I32cd58842865d8c8efe9444b23b05a96f1df168a
    Signed-off-by: Steven Webster <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691415
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=9ba567119e7899394eda49ea189b712c748e7682
Submitter: Zuul
Branch: master

commit 9ba567119e7899394eda49ea189b712c748e7682
Author: Steven Webster <email address hidden>
Date: Wed Oct 23 09:42:45 2019 -0500

    Traffic control fixes and refresh

    This commit fixes a couple of issues surrounding the main StarlingX
    traffic control script. The following has been changed:

    - The name of the script has been changed to remove the 'cgcs_'
      prefix

    - It was discovered that traffic controls were not being set on a
      multi-netted interface shared by aliased management and
      cluster-host interfaces. This is because aliased interfaces
      do not support a 'post-up' command in their network-script
      configuration file.

    Depends-On: https://review.opendev.org/691414
    Partial-Bug: #1839386
    Change-Id: I27c699415d80ff10f5eac162ae3a6be17ec59c7a
    Signed-off-by: Steven Webster <email address hidden>

[Wendy Mitchell (wmitchellwr)]

Verified one part of the above
2019-11-22_20-00-00

1. Confirmed script name change to
/usr/local/bin/tc_setup.sh (ie. cgcs_ prefix removal)

2. Confirm tc_setup.sh script now has sm_filter that treats SM heartbeats as high priority traffic
# Treat system maintenance heartbeats as high priority traffic (where IPTOS_CLASS_CS6=0xc0)
compute-0:/usr/local/bin$ cat tc_setup.sh | grep IPTOS_CLASS_CS
local SM_TOS=${IPTOS_CLASS_CS6}

[Wendy Mitchell (wmitchellwr)]

Also confirmed no longer have class htb 1:30 on the mgmt. interface (tc class show dev ens802f1)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/utilities/+/880265