neutron-sriov-agent consumes much resources by redundand calls to ip link show

[Mikhail Chernik]

On a node with SR-IOV interfaces neutron-sriov-agent seems to call 'ip link show' every 2 second for each interface. This leads to CPU utilization up to 60+% (I've seen 32% for neutron-rootwrap and 32% for neutron-sriov-nic-agent)

Since output of `ip link show' is the same for all interfaces, it is enough to call it only once per 2 sec interval

root@node-7:~# grep '15:10:' /var/log/neutron/neutron-sriov-agent.log | grep "'ip', 'link', 'show'" | wc -l
5783

# all interfaces
root@node-7:~# ip link show | grep '^[0-9].*' | wc -l
274

# SR-IOV VF interfaces
root@node-7:~# ip link show | grep '^[0-9].*' | grep eth | wc -l
252

 9224 root 10 -10 5148300 144148 9940 S 299.9 0.1 8871:03 ovs-vswitchd
36425 root 20 0 199672 16764 7156 S 27.2 0.0 682:36.73 neutron-rootwra
36398 neutron 20 0 196936 60620 10416 S 20.3 0.0 522:57.02 neutron-sriov-n

[Mikhail Chernik (mchernik)]

snippets are taken on snapshot #688

[Kevin Benton (kevinbenton)]

This should be significantly improved by the following upstream patch: https://review.openstack.org/416400

[Kevin Benton (kevinbenton)]

Can someone cherry-pick this change and confirm the fix? https://review.openstack.org/#/c/416523/1

[Kevin Benton (kevinbenton)]

Grab the latest version from https://review.openstack.org/#/c/416523 , not the first one in the series.

[Kevin Benton (kevinbenton)]

Proposed cherry-pick here so we don't have to wait for upstream: https://review.fuel-infra.org/29676

[Alexander Ignatov (aignatov)]

Fix https://review.openstack.org/#/c/416524/ was merged in upstream for stable/mitaka. Waiting for regular sync.

[Alexander Ignatov (aignatov)]

stable/mitaka sync in downstream https://review.fuel-infra.org/#/c/29701/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29701
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 38f448751c553098539b787e755b6688dcdf59bf
Author: Jenkins <email address hidden>
Date: Tue Jan 10 12:12:06 2017

Merge the tip of origin/stable/mitaka into origin/9.0/mitaka

67e334b Reduce IP link show calls for SR-IOV scan loop

Closes-Bug: #1653131
Change-Id: I9c8ca8fb807836de944bf845ef9928135f82400a

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Alexander Ignatov <email address hidden> on branch: 9.0/mitaka
Review: https://review.fuel-infra.org/29676
Reason: Abandoning in favor of https://review.fuel-infra.org/#/c/29701/

[Alexander Ignatov (aignatov)]

Fixed by https://review.fuel-infra.org/#/c/28335/ in 10.0/newton

[Kristina Berezovskaia (kkuznetsova)]

Verified on 9.2 snap 822

root@node-3:~# grep '12:07:' /var/log/neutron/neutron-sriov-agent.log | grep "'ip', 'link', 'show'" | wc -l
30
root@node-3:~# vim /var/log/neutron/neutron-sriov-agent.log
root@node-3:~# grep '11:03:' /var/log/neutron/neutron-sriov-agent.log | grep "'ip', 'link', 'show'" | wc -l
15
root@node-3:~# ip link show | grep '^[0-9].*' | wc -l
25
root@node-3:~# ip link show | grep '^[0-9].*' | wc ^C
root@node-3:~# ip link show | grep '^[0-9].*' | grep eth | wc -l
8
root@node-3:~# grep '11:04:' /var/log/neutron/neutron-sriov-agent.log | grep "'ip', 'link', 'show'" | wc -l
30