Noopfirewall driver or security group disabled should avoid impose security group related calls to Neutron server

Bug #1365806 reported by zhu zhu
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
zhu zhu
Juno
Fix Released
Undecided
Unassigned

Bug Description

With openvswitch neutron agent, during the daemon loop, the phase for setup_port_filters will try to grab/call method 'security_group_rules_for_devices' to Neutron Server.

And this operation will be very time consuming and have big performance bottleneck as it include ports query, rules query, network query as well as reconstruct the huge Security groups Dict Message. This message size is very large and for processing it, it will occupy a lot of CPU of Neutron Server. In cases like VM/perhost arrive to 700, the Neutron server will be busy doing the message and couldn't to do other thing and this could lead to message queue connection timeout and make queue disconnect the consumers. As a result the Neutron server is crashed and not function either for deployments or for API calls.

For the Noopfirewall or security group disabled situation, this operation should be avoided. Because eventually these reply message would not be used by Noopfirewall driver. (There methods are pass).

 with self.firewall.defer_apply():
            for device in devices.values():
                LOG.debug(_("Update port filter for %s"), device['device'])
                self.firewall.update_port_filter(device)

zhu zhu (zhuzhubj)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/119313

Changed in neutron:
assignee: nobody → zhu zhu (zhuzhubj)
status: New → In Progress
Kyle Mestery (mestery)
Changed in neutron:
milestone: none → juno-rc1
importance: Undecided → High
tags: added: ovs sg-fw
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/121645

Kyle Mestery (mestery)
Changed in neutron:
milestone: juno-rc1 → kilo-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/121645
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9a6c073656a7e0b1a26b2bca0ba381489d04e322
Submitter: Jenkins
Branch: master

commit 9a6c073656a7e0b1a26b2bca0ba381489d04e322
Author: Eugene Nikanorov <email address hidden>
Date: Mon Sep 15 22:10:45 2014 +0400

    Add missing methods to NoopFirewallDriver

    The fix adds missing methods into generic Firewall class
    and in NoopFirewall driver class.

    Change-Id: I6402448075ed414434dc007f5c403fc85b6b1456
    Closes-Bug: #1369685
    Related-Bug: #1365806

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (proposed/juno)

Related fix proposed to branch: proposed/juno
Review: https://review.openstack.org/126902

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (proposed/juno)

Reviewed: https://review.openstack.org/126902
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0a64b61f8cbf41e1bf74961c235f03ac6cc6ead6
Submitter: Jenkins
Branch: proposed/juno

commit 0a64b61f8cbf41e1bf74961c235f03ac6cc6ead6
Author: Eugene Nikanorov <email address hidden>
Date: Mon Sep 15 22:10:45 2014 +0400

    Add missing methods to NoopFirewallDriver

    The fix adds missing methods into generic Firewall class
    and in NoopFirewall driver class.

    Change-Id: I6402448075ed414434dc007f5c403fc85b6b1456
    Closes-Bug: #1369685
    Related-Bug: #1365806
    (cherry picked from commit 9a6c073656a7e0b1a26b2bca0ba381489d04e322)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/128913

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/119313
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=524981cce05a9b365036c0a1e9810036936d3d5b
Submitter: Jenkins
Branch: master

commit 524981cce05a9b365036c0a1e9810036936d3d5b
Author: ZHU ZHU <email address hidden>
Date: Fri Sep 5 03:01:07 2014 -0500

    Reduce security group db calls to neutron server

    Within ovs agent daemon loop, prepare_devices_filter will impose heavy workloads
    to neutron server in order to retrieve the security groups message to apply
    firewall rules. If agent is configured to use Noopfirewall driver or security
    groups are disabled, there is no need for loading the rules from server and
    refreshing the firewalls. This will reduce the number of db calls and improve
    performance for neutron server in this case.

    Change-Id: Id244aab3cac37fc6ed3dc05cbee91cdf9e34d9cc
    Closes-Bug: #1365806

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
John Kasperski (jckasper) wrote :

Are there any plans to cherry pick this fix back into Juno ?

Yaguang Tang (heut2008)
tags: added: juno-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/130105

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/lbaasv2)

Fix proposed to branch: feature/lbaasv2
Review: https://review.openstack.org/130864

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/lbaasv2)
Download full text (72.6 KiB)

Reviewed: https://review.openstack.org/130864
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c089154a94e5872efc95eab33d3d0c9de8619fe4
Submitter: Jenkins
Branch: feature/lbaasv2

commit 62588957fbeccfb4f80eaa72bef2b86b6f08dcf8
Author: Kevin Benton <email address hidden>
Date: Wed Oct 22 13:04:03 2014 -0700

    Big Switch: Switch to TLSv1 in server manager

    Switch to TLSv1 for the connections to the backend
    controllers. The default SSLv3 is no longer considered
    secure.

    TLSv1 was chosen over .1 or .2 because the .1 and .2 weren't
    added until python 2.7.9 so TLSv1 is the only compatible option
    for py26.

    Closes-Bug: #1384487
    Change-Id: I68bd72fc4d90a102003d9ce48c47a4a6a3dd6e03

commit 17204e8f02fdad046dabdb8b31397289d72c877b
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Oct 22 06:20:15 2014 +0000

    Imported Translations from Transifex

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I58db0476c810aa901463b07c42182eef0adb5114

commit d712663b99520e6d26269b0ca193527603178742
Author: Carl Baldwin <email address hidden>
Date: Mon Oct 20 21:48:42 2014 +0000

    Move disabling of metadata and ipv6_ra to _destroy_router_namespace

    I noticed that disable_ipv6_ra is called from the wrong place and that
    in some cases it was called with a bogus router_id because the code
    made an incorrect assumption about the context. In other case, it was
    never called because _destroy_router_namespace was being called
    directly. This patch moves the disabling of metadata and ipv6_ra in
    to _destroy_router_namespace to ensure they get called correctly and
    avoid duplication.

    Change-Id: Ia76a5ff4200df072b60481f2ee49286b78ece6c4
    Closes-Bug: #1383495

commit f82a5117f6f484a649eadff4b0e6be9a5a4d18bb
Author: OpenStack Proposal Bot <email address hidden>
Date: Tue Oct 21 12:11:19 2014 +0000

    Updated from global requirements

    Change-Id: Idcbd730f5c781d21ea75e7bfb15959c8f517980f

commit be6bd82d43fbcb8d1512d8eb5b7a106332364c31
Author: Angus Lees <email address hidden>
Date: Mon Aug 25 12:14:29 2014 +1000

    Remove duplicate import of constants module

    .. and enable corresponding pylint check now the only offending instance
    is fixed.

    Change-Id: I35a12ace46c872446b8c87d0aacce45e94d71bae

commit 9902400039018d77aa3034147cfb24ca4b2353f6
Author: rajeev <email address hidden>
Date: Mon Oct 13 16:25:36 2014 -0400

    Fix race condition on processing DVR floating IPs

    Fip namespace and agent gateway port can be shared by multiple dvr routers.
    This change uses a set as the control variable for these shared resources
    and ensures that Test and Set operation on the control variable are
    performed atomically so that race conditions do not occur among
    multiple threads processing floating IPs.
    Limitation: The scope of this change is limited to addressing the race
    condition described in the bug report. It may not address other issues
    such as pre-existing issue wit...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)
Download full text (7.4 KiB)

Reviewed: https://review.openstack.org/128913
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=71df7c80b9efa84f2ef87a2299600066816870b4
Submitter: Jenkins
Branch: master

commit b28eda57223e492924edb731e24c2e4f64cc0de5
Author: Carl Baldwin <email address hidden>
Date: Wed Oct 8 03:22:49 2014 +0000

    Remove two sets that are not referenced

    The code no longer references the updated_routers and removed_routers
    sets. This should have been cleaned up before but was missed.

    Closes-bug: #1232525

    Change-Id: I0396e13d2f7c3789928e0c6a4c0a071b02d5ff17
    (cherry picked from commit edb26bfcddf9d9a0e95955a6590d11fa7245ea2b)

commit 9cce0bfdb713c2b975b289d90de6d57b68ca3854
Author: Mark McClain <email address hidden>
Date: Thu Oct 9 13:29:48 2014 +0000

    Add Juno release milestone

    Change-Id: Iea584b00329d9474c14847db958f8743d4058525
    Closes-Bug: #1378855
    (cherry picked from commit 4e8a5b7de71ba6f8c050c424613c025310498940)

commit 8e76cccb1ed9a248439b1188d1d805649169e46b
Author: Mark McClain <email address hidden>
Date: Wed Oct 8 18:49:20 2014 +0000

    Add database relationship between router and ports

    Add an explicit schema relationship between a router and its ports. This
    change ensures referential integrity among the entities and prevents orphaned
    ports.

    Change-Id: I09e8a694cdff7f64a642a39b45cbd12422132806
    Closes-Bug: #1378866
    (cherry picked from commit 93012915a3445a8ac8a0b30b702df30febbbb728)

commit 5610343d5aab876480cbe15c8d77631e67d6142f
Author: Henry Gessau <email address hidden>
Date: Tue Oct 7 20:38:38 2014 -0400

    Disable PUT for IPv6 subnet attributes

    In Juno we are not ready for allowing the IPv6 attributes on a subnet
    to be updated after the subnet is created, because:
    - The implementation for supporting updates is incomplete.
    - Perceived lack of usefulness, no good use cases known yet.
    - Allowing updates causes more complexity in the code.
    - Have not tested that radvd, dhcp, etc. behave OK after update.

    Therefore, for now, we set 'allow_put' to False for the two IPv6
    attributes, ipv6_ra_mode and ipv6_address_mode. This prevents the
    modes from being updated via the PUT:subnets API.

    Closes-bug: #1378952

    Change-Id: Id6ce894d223c91421b62f82d266cfc15fa63ed0e
    (cherry picked from commit 8a08a3cb47d0dd69d4aa2e8fa661d04054fe95ae)

commit 54be5a9e977ea344cc53addb87635ddba0cfd815
Author: Sean M. Collins <email address hidden>
Date: Mon Oct 6 15:47:24 2014 -0400

    Skip IPv6 Tests in the OpenContrail plugin

    Similar to the way we are skipping tests in the OneConvergence plugin,
    introduced by Kevin Benton in 9294de441e684a81f6e802ba0564083f1ad319d6.

    Partial-Bug: #1378952

    Change-Id: I1650b0708af73ce63e92c55bc842607bb69efe60
    (cherry picked from commit 67962943969bc737a3f680a0defc2fc9df03c429)

commit aefc12ec552afe32f0d1d6f7c8c588afac956988
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Aug 7 22:27:23 2014 +0200

    Removed kombu from requirements

    Since we've replaced oslo-incubator RPC layer with...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/juno)

Reviewed: https://review.openstack.org/130105
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f9f694f95f5338d62f75ce54677dd165dc87a8d5
Submitter: Jenkins
Branch: stable/juno

commit f9f694f95f5338d62f75ce54677dd165dc87a8d5
Author: ZHU ZHU <email address hidden>
Date: Fri Sep 5 03:01:07 2014 -0500

    Reduce security group db calls to neutron server

    Within ovs agent daemon loop, prepare_devices_filter will impose heavy workloads
    to neutron server in order to retrieve the security groups message to apply
    firewall rules. If agent is configured to use Noopfirewall driver or security
    groups are disabled, there is no need for loading the rules from server and
    refreshing the firewalls. This will reduce the number of db calls and improve
    performance for neutron server in this case.

    Change-Id: Id244aab3cac37fc6ed3dc05cbee91cdf9e34d9cc
    Closes-Bug: #1365806
    (cherry picked from commit 524981cce05a9b365036c0a1e9810036936d3d5b)

tags: added: in-stable-juno
Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: kilo-1 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.