l2 pop doesn't always provide the whole list of fdb entries on agent restart

[Oleg Bondarev]

The whole list of fdb entries is provided to the agent in case a port form new network appears, or when agent is restarted.
Currently agent restart is detected by agent_boot_time option, 180 sec by default.
In fact boot time differs depending on port count and on some loaded clusters may exceed 180 secs on gateway nodes easily. Changing boot time in config works, but honestly this is not an ideal solution.
There should be a smarter way for agent restart detection (like agent itself sending flag in state report).

[Swaminathan Vasudevan (swaminathan-vasudevan)]

I agree that there should be a better way to handle this other than the changing the config option.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/615246

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/634666

[LIU Yulong (dragon889)]

Another approach to fix this is here:
https://review.openstack.org/#/c/640797/

[Oleg Bondarev (obondarev)]

RPC change ^^ seems indeed the correct approach, but we also need a backportable fix

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.openstack.org/615246

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/645405

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/645406

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/645408

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/640797
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a5244d6d44d2b66de27dc77efa7830fa657260be
Submitter: Zuul
Branch: master

commit a5244d6d44d2b66de27dc77efa7830fa657260be
Author: LIU Yulong <email address hidden>
Date: Mon Mar 4 21:17:20 2019 +0800

    More accurate agent restart state transfer

    Ovs-agent can be very time-consuming in handling a large number
    of ports. At this point, the ovs-agent status report may have
    exceeded the set timeout value. Some flows updating operations
    will not be triggerred. This results in flows loss during agent
    restart, especially for hosts to hosts of vxlan tunnel flow.

    This fix will let the ovs-agent explicitly, in the first rpc loop,
    indicate that the status is restarted. Then l2pop will be required
    to update fdb entries.

    Closes-Bug: #1813703
    Closes-Bug: #1813714
    Closes-Bug: #1813715
    Closes-Bug: #1794991
    Closes-Bug: #1799178

    Change-Id: I8edc2deb509216add1fb21e1893f1c17dda80961

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/649729

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634666
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8f20963c5b9e6762b6322f686bce99871bec6be9
Submitter: Zuul
Branch: master

commit 8f20963c5b9e6762b6322f686bce99871bec6be9
Author: Oleg Bondarev <email address hidden>
Date: Mon Feb 4 14:58:27 2019 +0400

    OVS agent: always send start flag during initial sync

    In order to avoid inaccurate agent_boot_time setting,
    this patch suggests to consider agent as "started" only
    after completion of initial sync with server.

    Change-Id: Icba05288889219e8a606c3809efd88b2c234bef3
    Closes-Bug: #1799178

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.openstack.org/650398

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/650399

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/650400

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/650402

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/650403

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/645408
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=62fe7852bbd70a24174853997096c52ee015e269
Submitter: Zuul
Branch: stable/pike

commit 62fe7852bbd70a24174853997096c52ee015e269
Author: LIU Yulong <email address hidden>
Date: Mon Mar 4 21:17:20 2019 +0800

    More accurate agent restart state transfer

    Ovs-agent can be very time-consuming in handling a large number
    of ports. At this point, the ovs-agent status report may have
    exceeded the set timeout value. Some flows updating operations
    will not be triggerred. This results in flows loss during agent
    restart, especially for hosts to hosts of vxlan tunnel flow.

    This fix will let the ovs-agent explicitly, in the first rpc loop,
    indicate that the status is restarted. Then l2pop will be required
    to update fdb entries.

    Conflicts:
     neutron/plugins/ml2/rpc.py

    Conflicts:
     neutron/plugins/ml2/drivers/l2pop/mech_driver.py

    Closes-Bug: #1813703
    Closes-Bug: #1813714
    Closes-Bug: #1813715
    Closes-Bug: #1794991
    Closes-Bug: #1799178

    Change-Id: I8edc2deb509216add1fb21e1893f1c17dda80961
    (cherry picked from commit a5244d6d44d2b66de27dc77efa7830fa657260be)
    (cherry picked from commit cc49ab550179bdc76d79f48be67886681cb43d4e)
    (cherry picked from commit 5ffca4966877454c605442e9e429aa83ea7d7348)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/645405
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cc49ab550179bdc76d79f48be67886681cb43d4e
Submitter: Zuul
Branch: stable/rocky

commit cc49ab550179bdc76d79f48be67886681cb43d4e
Author: LIU Yulong <email address hidden>
Date: Mon Mar 4 21:17:20 2019 +0800

    More accurate agent restart state transfer

    Ovs-agent can be very time-consuming in handling a large number
    of ports. At this point, the ovs-agent status report may have
    exceeded the set timeout value. Some flows updating operations
    will not be triggerred. This results in flows loss during agent
    restart, especially for hosts to hosts of vxlan tunnel flow.

    This fix will let the ovs-agent explicitly, in the first rpc loop,
    indicate that the status is restarted. Then l2pop will be required
    to update fdb entries.

    Conflicts:
     neutron/plugins/ml2/rpc.py

    Closes-Bug: #1813703
    Closes-Bug: #1813714
    Closes-Bug: #1813715
    Closes-Bug: #1794991
    Closes-Bug: #1799178

    Change-Id: I8edc2deb509216add1fb21e1893f1c17dda80961
    (cherry picked from commit a5244d6d44d2b66de27dc77efa7830fa657260be)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650398
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=382431c944e21ef6aab06f3746f23a84f7425cb2
Submitter: Zuul
Branch: stable/stein

commit 382431c944e21ef6aab06f3746f23a84f7425cb2
Author: Oleg Bondarev <email address hidden>
Date: Mon Feb 4 14:58:27 2019 +0400

    OVS agent: always send start flag during initial sync

    In order to avoid inaccurate agent_boot_time setting,
    this patch suggests to consider agent as "started" only
    after completion of initial sync with server.

    Change-Id: Icba05288889219e8a606c3809efd88b2c234bef3
    Closes-Bug: #1799178
    (cherry picked from commit 8f20963c5b9e6762b6322f686bce99871bec6be9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650399
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6b9d8bf3087a8fea2a991d392b314706039191a3
Submitter: Zuul
Branch: stable/rocky

commit 6b9d8bf3087a8fea2a991d392b314706039191a3
Author: Oleg Bondarev <email address hidden>
Date: Mon Feb 4 14:58:27 2019 +0400

    OVS agent: always send start flag during initial sync

    In order to avoid inaccurate agent_boot_time setting,
    this patch suggests to consider agent as "started" only
    after completion of initial sync with server.

    Change-Id: Icba05288889219e8a606c3809efd88b2c234bef3
    Closes-Bug: #1799178
    (cherry picked from commit 8f20963c5b9e6762b6322f686bce99871bec6be9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650400
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=55fa2d7ed42d255a89c23cb89df287727bf1ed03
Submitter: Zuul
Branch: stable/queens

commit 55fa2d7ed42d255a89c23cb89df287727bf1ed03
Author: Oleg Bondarev <email address hidden>
Date: Mon Feb 4 14:58:27 2019 +0400

    OVS agent: always send start flag during initial sync

    In order to avoid inaccurate agent_boot_time setting,
    this patch suggests to consider agent as "started" only
    after completion of initial sync with server.

    Change-Id: Icba05288889219e8a606c3809efd88b2c234bef3
    Closes-Bug: #1799178
    (cherry picked from commit 8f20963c5b9e6762b6322f686bce99871bec6be9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650402
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8865466e9fc39ff8e58576875f401f9fb8301beb
Submitter: Zuul
Branch: stable/pike

commit 8865466e9fc39ff8e58576875f401f9fb8301beb
Author: Oleg Bondarev <email address hidden>
Date: Mon Feb 4 14:58:27 2019 +0400

    OVS agent: always send start flag during initial sync

    In order to avoid inaccurate agent_boot_time setting,
    this patch suggests to consider agent as "started" only
    after completion of initial sync with server.

    Change-Id: Icba05288889219e8a606c3809efd88b2c234bef3
    Closes-Bug: #1799178
    (cherry picked from commit 8f20963c5b9e6762b6322f686bce99871bec6be9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650403
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=64e98ff9597684868fc07979c1e861d2dd506688
Submitter: Zuul
Branch: stable/ocata

commit 64e98ff9597684868fc07979c1e861d2dd506688
Author: Oleg Bondarev <email address hidden>
Date: Mon Feb 4 14:58:27 2019 +0400

    OVS agent: always send start flag during initial sync

    In order to avoid inaccurate agent_boot_time setting,
    this patch suggests to consider agent as "started" only
    after completion of initial sync with server.

    Change-Id: Icba05288889219e8a606c3809efd88b2c234bef3
    Closes-Bug: #1799178
    (cherry picked from commit 8f20963c5b9e6762b6322f686bce99871bec6be9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.1 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/649729
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9583dc0549da2b4529a59b5862ba42aebc5ae15f
Submitter: Zuul
Branch: stable/ocata

commit 9583dc0549da2b4529a59b5862ba42aebc5ae15f
Author: LIU Yulong <email address hidden>
Date: Mon Mar 4 21:17:20 2019 +0800

    More accurate agent restart state transfer

    Ovs-agent can be very time-consuming in handling a large number
    of ports. At this point, the ovs-agent status report may have
    exceeded the set timeout value. Some flows updating operations
    will not be triggerred. This results in flows loss during agent
    restart, especially for hosts to hosts of vxlan tunnel flow.

    This fix will let the ovs-agent explicitly, in the first rpc loop,
    indicate that the status is restarted. Then l2pop will be required
    to update fdb entries.

    Conflicts:
     neutron/plugins/ml2/rpc.py

    Conflicts:
     neutron/plugins/ml2/drivers/l2pop/mech_driver.py

    Closes-Bug: #1813703
    Closes-Bug: #1813714
    Closes-Bug: #1813715
    Closes-Bug: #1794991
    Closes-Bug: #1799178

    Change-Id: I8edc2deb509216add1fb21e1893f1c17dda80961
    (cherry picked from commit a5244d6d44d2b66de27dc77efa7830fa657260be)
    (cherry picked from commit cc49ab550179bdc76d79f48be67886681cb43d4e)
    (cherry picked from commit 5ffca4966877454c605442e9e429aa83ea7d7348)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ocata-eol release.