Performance: L2 agent takes too much time to refresh sg rules

[Lan Qi song]

This issue is introducing a performance problem for the L2 agent including LinuxBridge and OVS agent in Compute node when there are lots of networks and instances in this Compute node (eg. 500 instances)

The performance problem reflect in two aspects:

1. When LinuxBridge agent service starts up(this seems only happened in LinuxBridge agent not for the OVS agent), I found there were two methods take too much time:

   1.1 get_interface_by_ip(), we should find the interface which was assigned with the "local ip" defined in configuration file, and to check whether this interface support "vxlan" or not. This method will iterate all the interface in this compute node and execute "ip link show [interface] to [local ip]" to judge the result. I think there should be a faster way.

   1.2 prepare_port_filter() , in this method , we should make sure the ipset are create correctly. But this method will execute too much "ipset" commands and take too much time.

2. When devices' sg rules are changed, L2 agent should refresh the firewalls.

    2.1 refresh_firewall() this method will call "modify_rules" to make the rules predicable, but this method also takes too much time.

It will be very benefit for the large scales of networks if this performance problem can be fix or optimize.

[Rossella Sblendido (rossella-o)]

there's a patch abandoned to improve _modify_rules() https://review.openstack.org/#/c/138793/ I think that would partially fix the issue.

[Brian Haley (brian-haley)]

What release is this on? There were a number of performance improvements made in the Liberty cycle to _modify_rules() that were even backported to stable/kilo, like a dictionary lookup of rules. See https://review.openstack.org/184318

There was even a change made to not call the ipset code as much by using sets, and to remove some duplicated logging messages. See https://review.openstack.org/193259 and https://review.openstack.org/191937

Can you please test with the latest code and see if the issue persists?

[Lan Qi song (lqslan)]

@Brian thanks for your reply, I used the stable/kilo branch, but seems not the latest code in this branch and the patches you mentioned look really helpful.

I will try the latest code and run the test again, thanks!

[Brad Behle (behle)]

Marking this bug as incomplete to wait for the submitter to try with the latest code to see if the problem still exists.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/335037

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/337064
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ca792b0d569ca9d55617d6fddfdb53a743c05661
Submitter: Jenkins
Branch: master

commit ca792b0d569ca9d55617d6fddfdb53a743c05661
Author: venkata anil <email address hidden>
Date: Fri Jul 8 18:49:45 2016 +0000

    Avoid duplicate ipset processing for security groups

    While applying firewall rules for ports, existing implementation
    iterates through each port and applies ipset for its security groups.
    With this, when ports share the security group, ipset for same security
    group is called again and again while iterating through ports.

    From the DB, we already get the list of security groups for which ipset
    members have to be updated. In the new approach, we apply ipset on these
    security groups(before firewall rules setup), instead of iterating
    through all ports(during settig up firewall rules)and parsing them for
    security groups and then applying ipset. With this we can avoid
    duplicate ipset processing for same security groups.

    Closes-bug: #1598734
    Partial-Bug: #1499177
    Change-Id: I3f16d1a3a847e706ff743a8e1a5e7598f9f4c6dd

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/347068

[Ihar Hrachyshka (ihar-hrachyshka)]

Not backporting to Liberty since it does not seem like a huge impact/critical bug.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/347068
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7238c8e1afe925b3d64ecb83199300191648a32a
Submitter: Jenkins
Branch: stable/mitaka

commit 7238c8e1afe925b3d64ecb83199300191648a32a
Author: venkata anil <email address hidden>
Date: Fri Jul 8 18:49:45 2016 +0000

    Avoid duplicate ipset processing for security groups

    While applying firewall rules for ports, existing implementation
    iterates through each port and applies ipset for its security groups.
    With this, when ports share the security group, ipset for same security
    group is called again and again while iterating through ports.

    From the DB, we already get the list of security groups for which ipset
    members have to be updated. In the new approach, we apply ipset on these
    security groups(before firewall rules setup), instead of iterating
    through all ports(during settig up firewall rules)and parsing them for
    security groups and then applying ipset. With this we can avoid
    duplicate ipset processing for same security groups.

    Closes-bug: #1598734
    Partial-Bug: #1499177
    Change-Id: I3f16d1a3a847e706ff743a8e1a5e7598f9f4c6dd
    (cherry picked from commit ca792b0d569ca9d55617d6fddfdb53a743c05661)

[OpenStack Infra (hudson-openstack)]

Change abandoned by venkata anil (<email address hidden>) on branch: master
Review: https://review.openstack.org/335037

[venkata anil (anil-venkata)]

https://review.openstack.org/#/c/351773/ proposed to fix this.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/351773
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Slawek Kaplonski (slaweq)]

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/351773
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Bug closed due to lack of activity, please feel free to reopen if needed.