Restarting OVS agent drops VMs traffic when using VLAN provider bridges

[Eugene Nikanorov]

(amuller) editing bug report based on comment 7:

Dropping flows on the the physical bridges causes networking to drop. It's one of the two places that still causes networking to drop when OVS agent is restarted. The other is that the patch port between br-int and br-tun is being deleted and rebuilt during startup.

Original bug description contained intent to set cookie on physical bridges for consistency purposes.
In other words, with absence of cookies, flows can't be removed properly, so for kilo/liberty version that lead to stale flows remaining in OVS and disrupting the network.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/242743

[Jian Wen (wenjianhn)]

After restarting an ovs agent, it still drops all existing flows of the physical bridges.
We need the timestamps to clean stale flows. Related to bug 1383674

Feel free to use my patch:
https://github.com/wenjianhn/neutron/tree/draft-graceful-restart-br-physical

[Ann Taraday (akamyshnikova)]

@wenjianhn

Yes, but dropping all flows on physical bridges does not led to vm's connectivity lost, so why is this need?

[Jian Wen (wenjianhn)]

Actually, it does.

In our production environment, hundreds of VM lost their network connection
because agents
 deleted all the flows even if the neutron servers were very busy.
We are still maintaining Neutron icehouse. The logical is the same with
HEAD.

We don't use tunnel yet. Only VLAN:
$ neutron net-show e8aba58a-6750-4e65-bf03-ab7c8875c310
....
| provider:network_type | vlan |
| provider:physical_network | physnet556 |
| provider:segmentation_id | 556 |

On Thu, Nov 26, 2015 at 10:01 PM, Ann Kamyshnikova <
<email address hidden>> wrote:

> @wenjianhn
>
> Yes, but dropping all flows on physical bridges does not led to vm's
> connectivity lost, so why is this need?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1514056
>
> Title:
> Set agent timestamp aka cookie to physical bridges
>
> Status in neutron:
> In Progress
>
> Bug description:
> Currently ovs agent only explicitly sets agent timestamp to br-int and
> br-tun bridges.
> Other physical bridges that are configured receive cookie=0x0 for their
> flows because the agent doesn't set timestamp for these bridges.
> Currently that doesn't lead to any malfunction, however it's better to
> provide a consistentsy over operations with bridges's flows.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/neutron/+bug/1514056/+subscriptions
>

--
Best,

Jian

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/257285

[SUZUKI, Kazuhiro (kaz-kaz)]

Excuse me, I wrote a wrong bug ID.
Please re-asign this bug.

[Clayton O'Neill (clayton-oneill)]

Dropping these flows on the the physical bridges definitely causes networking to drop. It's one of the two places that still causes networking to drop when OVS agent is restarted. The other is that the patch port between br-int and br-tun is being deleted and rebuilt during startup. As an operator this is something I'd definitely like to see fixed.

[Clayton O'Neill (clayton-oneill)]

Just a datapoint, with 50 instances, 50 FIPs, 50 routers on a single node, restarting OVS agent causes it to drop traffic for 10-15 seconds while the physical bridge flows are rebuilt. If you want to see this more clearly, put a 10 second sleep into root wrap and restart the OVS agent. In that scenario you can see the problem even with one instance.

[Clayton O'Neill (clayton-oneill)]

Here is more data comparing Kilo and Liberty OVS agent restart behavior:

https://docs.google.com/spreadsheets/d/1ZGra_MszBlL0fNsFqd4nOvh1PsgWu58-GxEeh1m1BPw/edit?usp=sharing

The situation is greatly improved for agents that only host compute, but for those hosting routers it's still quite bad. In my testing it looks like the problem is almost entirely because the br-ex bridge flows are being cleared.

I think the priority of this bug should be raised to high.

[Kevin Fox (kevpn)]

+1.

It might be good to amend the bug description to make this clear. As worded it sounds like a low pri thing rather then the high priority to ops feature (quick ovs agent restart) it is.

I'm also curious if the really bad behavior of the network node agent restart shows up on the compute nodes when dvr is enabled. We're going to be using that in production soon.

[Clayton O'Neill (clayton-oneill)]

This also occurs with the flat provider.

[Kevin Benton (kevinbenton)]

Tangent: We should have filed a new bug for this. The original bug wasn't about flows being lost at all.

[Hynek Mlnarik (hmlnarik-s)]

I believe the original bug was covered by Eugene Nikanorov's patch, and I'm proposing a patch for the rephrased part.

The issue seems to come out of the non-transactional nature of flow manipulation. The proper solution would be to have all commands modifying a flow issued and processed atomically while now every del-flows and add-flow is executed separately. Atomicity is not possible with OVS until 2.4 (using --bundle option) but even if older versions are used, there is space for optimizations. One - hopefully significant enough - is preparing all commands before they would be issued and squeezing all add-flow commands into a single execution. This way the time window when flows are manipulated is reduced. The patch works along these lines.

[Clayton O'Neill (clayton-oneill)]

Why can't the same technique that was used for br-tun be used for br-ex? There all the new flows are added with a cookie containing a unique UUID. After all the new flows are added, the old flows with the old cookie are removed. If that technique is used then it doesn't introduce a dependency on OVS 2.4. Is OVS 2.4 already required for Mitaka?

[Jian Wen (wenjianhn)]

For consistent reason we may need to use the cookie.
Hynek, could you check whether the following patch still works?

https://github.com/wenjianhn/neutron/commit/d8ebb562d752082a263cabeb074aea20b99c4de5

[Hynek Mlnarik (hmlnarik-s)]

@Jian Wen: The patches are conflicting in only a single line of code so it is straightforward to resolve it. It needs some more polishing though before applicable to current neutron code. I got the following issue:
 AttributeError: 'OVSNeutronAgent' object has no attribute 'root_helper'
After cleaning that up, another error appeared:
 AttributeError: 'OVSBridge' object has no attribute 'cleanup_flows'

[Hynek Mlnarik (hmlnarik-s)]

@Clayton O'Neill: The patch does not introduce dependency on OVS 2.4. It is ready for easy introduction of atomic transactions supported by OVS 2.4 - but only once that version would be made a requirement.

The same technique using agent UUIDs can - and indeed should - be used. It is actually Jian Wen's patch that starts the work on it. Yet approach of this technique is complementary to the one of the transaction patch.

There are two ways to approach flow reconstruction:
1) Existing flows are deleted, then new ones are gradually built
2) New flows are built with an agent cookie, then all flows that have a different cookie are removed

Both cases suffer from being non-atomic:
1) Connectivity is lost until flows are rebuilt
2) There is time window for inconsistency between "old" and "new" set of rules causing weird behaviour in connectivity until "new" ones take over

As there is no support for atomic manipulation of OF in OVS < 2.4, the only way to reduce the chance of hitting intermediate state is to reduce time of the action; whether it would be 1) or 2) does not really matter. That is done by this patch:
* time spent in the Python code between individual ovs-ofctl is effectively eliminated as all the commands are buffered and executed at commit time (this is the only similarity with DeferredOVSBridge)
* series of the same action (e.g. add-flow) with flows is squashed into a single ovs-ofctl call - reducing time and number of ovs-ofctl executions
* order of commands is preserved as no grouping/sorting is done

Proper solution to this issue would be to use transactions for their atomicity but that is not available at this moment. commit() function is ready to be straightforwardly adjusted to support these once OVS 2.4 would become a requirement.

[Clayton O'Neill (clayton-oneill)]

I agree that making the changes transactional is the better solution. However, I think most operators are ok with your option 2 in most cases. The most common case where this is an issue is when doing restarts of the agent to do upgrades, config changes or other maintenance. In that case your option 2 works well since the window for inconsistency is very small.

I think option 2 is better for the time being, although I'd gladly support optional transaction support also., since it is the better long term solution. I think the biggest issue with that approach is 2.4.0 is still fairly new and there is not yet an LTS version of it.

[Clayton O'Neill (clayton-oneill)]

Disregard my comments about OVS 2.4. I talked to Hynek and he cleared up my confusion regarding his current patch vs the previous one.

[Jian Wen (wenjianhn)]

I am not an OvS expert.

Most of the time adding the "new " rules only updating the cookie.
In another words, the old ones are replaced by the new.
Actually, cleanup_stale_flows() are not likely to clean any useful rules.

Does time window mentioned at #17 exist?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/284639

[Hynek Mlnarik (hmlnarik-s)]

To enable for faster fix of the most urgent part of this bug, I have submitted a patch that fixes the most severe part of the bug. It is based on Eugene and Jian's patches. The other part - preparation of transactional behaviour - will be submitted as a separate patch. Please review.

[Ihar Hrachyshka (ihar-hrachyshka)]

Seems high impact. Targeting for M.

[Joseph bajin (josephbajin)]

Will this be back-ported into Liberty?

I know that we won't be going to M for a while. It would be great to have this feature in Liberty as most production environments will be upgrading to that version before M is even thought about.

[Jian Wen (wenjianhn)]

Joseph, I think it's doable.

On Mon, Feb 29, 2016 at 11:17 PM, Joseph bajin <email address hidden>
wrote:

> Will this be back-ported into Liberty?
>
> I know that we won't be going to M for a while. It would be great to have
> this feature in Liberty as most production environments will be upgrading
> to that version before M is even thought about.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1514056
>
> Title:
> Restarting OVS agent drops VMs traffic when using VLAN provider
> bridges
>
> Status in neutron:
> In Progress
>
> Bug description:
> (amuller) editing bug report based on comment 7:
>
> Dropping flows on the the physical bridges causes networking to drop.
> It's one of the two places that still causes networking to drop when
> OVS agent is restarted. The other is that the patch port between br-
> int and br-tun is being deleted and rebuilt during startup.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/neutron/+bug/1514056/+subscriptions
>

--
Best,

Jian

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/297211

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/297244

[OpenStack Infra (hudson-openstack)]

Change abandoned by Hynek Mlnarik (<email address hidden>) on branch: master
Review: https://review.openstack.org/297244

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/297818

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/299243

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/299243

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/297211
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8dce6a5c873c2c18e5a9c6165bf3974aead02588
Submitter: Jenkins
Branch: master

commit 8dce6a5c873c2c18e5a9c6165bf3974aead02588
Author: Clayton O'Neill <email address hidden>
Date: Thu Mar 24 14:59:41 2016 +0000

    Don't delete br-int to br-tun patch on startup

    When starting up, we don't want to delete the patch port between br-int
    and br-tun unless we're also dropping the flows.. In liberty both of
    these bridges were switched to not dump flows on startup and to put the
    bridges in secure mode so that default flood flows are not installed
    when the bridge is created.

    Without this patch the patch port is torn down and not reinstalled until
    br-tun is setup again.

    Partial-Bug: #1514056
    Change-Id: Ia518a99a2de5d1bda467fde57892c43970f88bcd

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/299280

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/284639
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cacde308eef6f1d7005e555b4521332da95d3cf4
Submitter: Jenkins
Branch: master

commit cacde308eef6f1d7005e555b4521332da95d3cf4
Author: Hynek Mlnarik <email address hidden>
Date: Thu Feb 25 11:34:15 2016 +0100

    Cleanup stale OVS flows for physical bridges

    Perform deletion of the stale flows in physical bridges consistently with
    br-int and br-tun, respecting drop_flows_on_start configuration option.
    Added tests for auxiliary bridge and functional tests for the physical
    bridge using VLAN/flat external network. Fixes part of the bug 1514056;
    together with [1] and [2], the bug should be considered fixed.

    The commit also fixes inconsistency between netmask of allocated IP
    addresses assigned in _create_test_port_dict and ip_len in _plug_ports
    of base.py.

    [1] https://review.openstack.org/#/c/297211/
    [2] https://review.openstack.org/#/c/297818/

    Co-Authored-By: Jian Wen <email address hidden>
    Partial-Bug: 1514056
    Change-Id: I9801b76829021c9a0e6358982e1136637634a521

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/300022

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/300424

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/300430

[OpenStack Infra (hudson-openstack)]

Change abandoned by Clayton O'Neill (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/300430

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/297818
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a549f30fad93508bf9dfdcfb20cd522f7add27b0
Submitter: Jenkins
Branch: master

commit a549f30fad93508bf9dfdcfb20cd522f7add27b0
Author: Clayton O'Neill <email address hidden>
Date: Thu Mar 24 15:28:21 2016 +0000

    Don't disconnect br-int from phys br if connected

    When starting up, we don't want to delete the patch port between br-int
    and the physical bridges. In liberty the br-int bridge was changed to
    not tear down flows on startup, and change
    I9801b76829021c9a0e6358982e1136637634a521 will change the physical
    bridges to not tear down flows also.

    Without this patch the patch port is torn down and not reinstalled until
    after the initial flows are set back up.

    Partial-Bug: #1514056
    Change-Id: I05bf5105a6f3acf6a313ce6799648a095cf8ec96

[OpenStack Infra (hudson-openstack)]

Change abandoned by Hynek Mlnarik (<email address hidden>) on branch: master
Review: https://review.openstack.org/282636
Reason: Abandoning

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/299280
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b392ad1c66dfeb4d0954fae3c5170381ff1bdef0
Submitter: Jenkins
Branch: stable/liberty

commit b392ad1c66dfeb4d0954fae3c5170381ff1bdef0
Author: Clayton O'Neill <email address hidden>
Date: Thu Mar 24 14:59:41 2016 +0000

    Don't delete br-int to br-tun patch on startup

    When starting up, we don't want to delete the patch port between br-int
    and br-tun unless we're also dropping the flows.. In liberty both of
    these bridges were switched to not dump flows on startup and to put the
    bridges in secure mode so that default flood flows are not installed
    when the bridge is created.

    Without this patch the patch port is torn down and not reinstalled until
    br-tun is setup again.

    Partial-Bug: #1514056
    Change-Id: Ia518a99a2de5d1bda467fde57892c43970f88bcd
    (cherry picked from commit 8dce6a5c873c2c18e5a9c6165bf3974aead02588)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/299243
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=93d29d131c0234075ac547906f77900ce47cceec
Submitter: Jenkins
Branch: stable/mitaka

commit 93d29d131c0234075ac547906f77900ce47cceec
Author: Clayton O'Neill <email address hidden>
Date: Thu Mar 24 14:59:41 2016 +0000

    Don't delete br-int to br-tun patch on startup

    When starting up, we don't want to delete the patch port between br-int
    and br-tun unless we're also dropping the flows.. In liberty both of
    these bridges were switched to not dump flows on startup and to put the
    bridges in secure mode so that default flood flows are not installed
    when the bridge is created.

    Without this patch the patch port is torn down and not reinstalled until
    br-tun is setup again.

    Partial-Bug: #1514056
    Change-Id: Ia518a99a2de5d1bda467fde57892c43970f88bcd
    (cherry picked from commit 8dce6a5c873c2c18e5a9c6165bf3974aead02588)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/306033

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/306041

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/300022
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=07401352a964b92ef4bc09a09800554e4a84cc87
Submitter: Jenkins
Branch: stable/mitaka

commit 07401352a964b92ef4bc09a09800554e4a84cc87
Author: Hynek Mlnarik <email address hidden>
Date: Thu Feb 25 11:34:15 2016 +0100

    Cleanup stale OVS flows for physical bridges

    Perform deletion of the stale flows in physical bridges consistently with
    br-int and br-tun, respecting drop_flows_on_start configuration option.
    Added tests for auxiliary bridge and functional tests for the physical
    bridge using VLAN/flat external network. Fixes part of the bug 1514056;
    together with [1] and [2], the bug should be considered fixed.

    The commit also fixes inconsistency between netmask of allocated IP
    addresses assigned in _create_test_port_dict and ip_len in _plug_ports
    of base.py.

    [1] https://review.openstack.org/#/c/297211/
    [2] https://review.openstack.org/#/c/297818/

    Co-Authored-By: Jian Wen <email address hidden>
    Partial-Bug: 1514056
    Change-Id: I9801b76829021c9a0e6358982e1136637634a521
    (cherry picked from commit cacde308eef6f1d7005e555b4521332da95d3cf4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/306033
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bf66cc6f74133cfe6c1ab75287d39814ac44b068
Submitter: Jenkins
Branch: stable/mitaka

commit bf66cc6f74133cfe6c1ab75287d39814ac44b068
Author: Clayton O'Neill <email address hidden>
Date: Thu Mar 24 15:28:21 2016 +0000

    Don't disconnect br-int from phys br if connected

    When starting up, we don't want to delete the patch port between br-int
    and the physical bridges. In liberty the br-int bridge was changed to
    not tear down flows on startup, and change
    I9801b76829021c9a0e6358982e1136637634a521 will change the physical
    bridges to not tear down flows also.

    Without this patch the patch port is torn down and not reinstalled until
    after the initial flows are set back up.

    Partial-Bug: #1514056
    Change-Id: I05bf5105a6f3acf6a313ce6799648a095cf8ec96
    (cherry picked from commit a549f30fad93508bf9dfdcfb20cd522f7add27b0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/306041
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a46e00817a65ac7a55e47f4e4b0b89596d776f8d
Submitter: Jenkins
Branch: stable/liberty

commit a46e00817a65ac7a55e47f4e4b0b89596d776f8d
Author: Clayton O'Neill <email address hidden>
Date: Thu Mar 24 15:28:21 2016 +0000

    Don't disconnect br-int from phys br if connected

    When starting up, we don't want to delete the patch port between br-int
    and the physical bridges. In liberty the br-int bridge was changed to
    not tear down flows on startup, and change
    I9801b76829021c9a0e6358982e1136637634a521 will change the physical
    bridges to not tear down flows also.

    Without this patch the patch port is torn down and not reinstalled until
    after the initial flows are set back up.

    Partial-Bug: #1514056
    Change-Id: I05bf5105a6f3acf6a313ce6799648a095cf8ec96
    (cherry picked from commit a549f30fad93508bf9dfdcfb20cd522f7add27b0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/300424
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8d29f38356fc5b840fa8b5c31fcd9d76c0fdd336
Submitter: Jenkins
Branch: stable/liberty

commit 8d29f38356fc5b840fa8b5c31fcd9d76c0fdd336
Author: Hynek Mlnarik <email address hidden>
Date: Thu Feb 25 11:34:15 2016 +0100

    Cleanup stale OVS flows for physical bridges

    Perform deletion of the stale flows in physical bridges consistently with
    br-int and br-tun, respecting drop_flows_on_start configuration option.
    Added tests for auxiliary bridge and functional tests for the physical
    bridge using VLAN/flat external network. Fixes part of the bug 1514056;
    together with [1] and [2], the bug should be considered fixed.

    The commit also fixes inconsistency between netmask of allocated IP
    addresses assigned in _create_test_port_dict and ip_len in _plug_ports
    of base.py.

    Further, this commit sets agent UUID to physical bridges similarly to
    tun and int bridges. This is necessary for stale flows cleanup to work
    correctly. In upstream, it is treated using OVSBridgeCookieMixin.

    [1] https://review.openstack.org/#/c/297211/
    [2] https://review.openstack.org/#/c/297818/

    Conflicts:
     neutron/tests/functional/agent/l2/base.py
     neutron/tests/unit/plugins/ml2/drivers/openvswitch/agent/test_ovs_tunnel.py

    Co-Authored-By: Jian Wen <email address hidden>
    Co-Authored-By: Clayton O'Neill <email address hidden>
    Partial-Bug: 1514056
    Change-Id: I9801b76829021c9a0e6358982e1136637634a521
    (cherry picked from commit cacde308eef6f1d7005e555b4521332da95d3cf4)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/314250

[OpenStack Infra (hudson-openstack)]

Change abandoned by enikanorov (<email address hidden>) on branch: master
Review: https://review.openstack.org/242743
Reason: This has been superseeded

[Clayton O'Neill (clayton-oneill)]

I believe all aspects of this are now resolved and this can be closed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/314250
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3bf73801df169de40d365e6240e045266392ca63
Submitter: Jenkins
Branch: master

commit a323769143001d67fd1b3b4ba294e59accd09e0e
Author: Ryan Moats <email address hidden>
Date: Tue Oct 20 15:51:37 2015 +0000

    Revert "Improve performance of ensure_namespace"

    This reverts commit 81823e86328e62850a89aef9f0b609bfc0a6dacd.

    Unneeded optimization: this commit only improves execution
    time on the order of milliseconds, which is less than 1% of
    the total router update execution time at the network node.

    This also

    Closes-bug: #1574881

    Change-Id: Icbcdf4725ba7d2e743bb6761c9799ae436bd953b

commit 7fcf0253246832300f13b0aa4cea397215700572
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Apr 21 07:05:16 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I9e930750dde85a9beb0b6f85eeea8a0962d3e020

commit 643b4431606421b09d05eb0ccde130adbf88df64
Author: OpenStack Proposal Bot <email address hidden>
Date: Tue Apr 19 06:52:48 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I52d7460b3265b5460b9089e1cc58624640dc7230

commit 1ffea42ccdc14b7a6162c1895bd8f2aae48d5dae
Author: OpenStack Proposal Bot <email address hidden>
Date: Mon Apr 18 15:03:30 2016 +0000

    Updated from global requirements

    Change-Id: Icb27945b3f222af1d9ab2b62bf2169d82b6ae26c

commit b970ed5bdac60c0fa227f2fddaa9b842ba4f51a7
Author: Kevin Benton <email address hidden>
Date: Fri Apr 8 17:52:14 2016 -0700

    Clear DVR MAC on last agent deletion from host

    Once all agents are deleted from a host, the DVR MAC generated
    for that host should be deleted as well to prevent a buildup of
    pointless flows generated in the OVS agent for hosts that don't
    exist.

    Closes-Bug: #1568206
    Change-Id: I51e736aa0431980a595ecf810f148ca62d990d20
    (cherry picked from commit 92527c2de2afaf4862fddc101143e4d02858924d)

commit eee9e58ed258a48c69effef121f55fdaa5b68bd6
Author: Mike Bayer <email address hidden>
Date: Tue Feb 9 13:10:57 2016 -0500

    Add an option for WSGI pool size

    Neutron currently hardcodes the number of
    greenlets used to process requests in a process to 1000.
    As detailed in
    http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html

    this can cause requests to wait within one process
    for available database connection while other processes
    remain available.

    By adding a wsgi_default_pool_size option functionally
    identical to that of Nova, we can lower the number of
    greenlets per process to be more in line with a typical
    max database connection pool size.

    DocImpact: a previously unused configuration value
               wsgi_default_pool_size is now used to a...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/334926

[OpenStack Infra (hudson-openstack)]

Change abandoned by Hynek Mlnarik (<email address hidden>) on branch: master
Review: https://review.openstack.org/334926
Reason: Work will continue in https://review.openstack.org/#/c/348889/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/353373

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/353373
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=27aee4a9c53ad9e8916e074f5cb7675871964448
Submitter: Jenkins
Branch: master

commit 27aee4a9c53ad9e8916e074f5cb7675871964448
Author: Hynek Mlnarik <email address hidden>
Date: Wed Aug 10 10:07:42 2016 +0200

    Connectivity tests for OVS agent failures/restarts

    Adding two tests:

    * A test that for native ovs-ofctl interface verifies that stopping the
      ovs-neutron-agent does not disrupt network traffic. Stopping the agent
      means also stopping the OVS bridge controller, hence OVS can decide to
      take over management of OpenFlow rules, clear them up, and this way
      cause network traffic disruption.

    * A test that creates two ports in a single network, then starts
      pinging one from the other while restarting OVS agents. The test verifies
      that no packet is lost during OVS agent restarts.

    Change-Id: I2cd1195fc0622c8c8d614f00e9dd6884ad388d69
    Related-Bug: 1514056
    Related-Bug: 1607787

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/462998

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/462998
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=81bb6aa348c05b9cff39ce2acda2bd77e01167c7
Submitter: Jenkins
Branch: stable/ocata

commit 81bb6aa348c05b9cff39ce2acda2bd77e01167c7
Author: Hynek Mlnarik <email address hidden>
Date: Wed Aug 10 10:07:42 2016 +0200

    Connectivity tests for OVS agent failures/restarts

    Adding two tests:

    * A test that for native ovs-ofctl interface verifies that stopping the
      ovs-neutron-agent does not disrupt network traffic. Stopping the agent
      means also stopping the OVS bridge controller, hence OVS can decide to
      take over management of OpenFlow rules, clear them up, and this way
      cause network traffic disruption.

    * A test that creates two ports in a single network, then starts
      pinging one from the other while restarting OVS agents. The test verifies
      that no packet is lost during OVS agent restarts.

    Change-Id: I2cd1195fc0622c8c8d614f00e9dd6884ad388d69
    Related-Bug: 1514056
    Related-Bug: 1607787
    (cherry picked from commit 27aee4a9c53ad9e8916e074f5cb7675871964448)