OVN: bump the ovn-remote-probe-interval for ovn-controller

Bug #1836604 reported by Kamil Sambor
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Undecided
Kamil Sambor

Bug Description

For some deployments the default ovn-remove-probe-interval of 5000ms (5s) [0] is not enough.

If the ovn-controller takes more time recomputing the flows than the probe interval, chances are that it will try to reconnected to the SB OVSDB causing a full re-sync of the DB with it's in-memory replica (costy).

In order to change this default one can run the following command where the ovn-controller is running:

$ ovs-vsctl set open . external_ids:ovn-remote-probe-interval=<TIME IN MS>

We need to make it either configurable via Tripleo-Heat-Templates.

Perhaps even thing about a "saner" default to accommodate for bigger deployments which requires ovn-controller to process a lot of flows (e.g having a lot of SG and SG rules would result in a lot of flows being created for ACLs, specially in OSP 13 where we do not have Port Groups).

[0] https://github.com/openvswitch/ovs/blob/88d2ac50aa4e3383e185b698a1b3a44a6f7b4f80/ovn/controller/ovn-controller.c#L75

Kamil Sambor (ksambor)
Changed in tripleo:
assignee: nobody → Kamil Sambor (ksambor)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/670864

Changed in tripleo:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/670864
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=65a132505a65de3cc91c52bc07a768fbfe139824
Submitter: Zuul
Branch: master

commit 65a132505a65de3cc91c52bc07a768fbfe139824
Author: Kamil Sambor <email address hidden>
Date: Mon Jul 15 17:23:59 2019 +0200

    Add the ability to configure ovn-remote-probe-interval

    This parameter sets inactive probe interval of the JSON
    session from ovn-controller to the OVN SB database.
    By default this it is 5s which not be sufficient in
    loaded systems or during high control-plane activity spikes,
    leading to unnecessary reconnections to OVSDB server.
    Now it is extended by default to 1 min and it
    is configurable.

    Depends-On: https://review.opendev.org/#/c/670861/
    Change-Id: Ie7cb761ad3b4a180990de2916d6210d15ec0bf50
    Closes-Bug: #1836604

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/673310

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/673310
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6d72398122707f3176e142fb581fdff0f9e60efc
Submitter: Zuul
Branch: stable/stein

commit 6d72398122707f3176e142fb581fdff0f9e60efc
Author: Kamil Sambor <email address hidden>
Date: Mon Jul 15 17:23:59 2019 +0200

    Add the ability to configure ovn-remote-probe-interval

    This parameter sets inactive probe interval of the JSON
    session from ovn-controller to the OVN SB database.
    By default this it is 5s which not be sufficient in
    loaded systems or during high control-plane activity spikes,
    leading to unnecessary reconnections to OVSDB server.
    Now it is extended by default to 1 min and it
    is configurable.

    Depends-On: https://review.opendev.org/#/c/673308/
    Change-Id: Ie7cb761ad3b4a180990de2916d6210d15ec0bf50
    Closes-Bug: #1836604
    (cherry picked from commit 65a132505a65de3cc91c52bc07a768fbfe139824)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/675924

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/675927

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.opendev.org/675924
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=091cdd4369f613e99a0e63513a6b91cec23a1ad0
Submitter: Zuul
Branch: stable/rocky

commit 091cdd4369f613e99a0e63513a6b91cec23a1ad0
Author: Kamil Sambor <email address hidden>
Date: Mon Aug 12 16:53:48 2019 +0200

    Add the ability to configure ovn-remote-probe-interval

    This parameter sets inactive probe interval of the JSON
    session from ovn-controller to the OVN SB database.
    By default this it is 5s which not be sufficient in
    loaded systems or during high control-plane activity spikes,
    leading to unnecessary reconnections to OVSDB server.
    Now it is extended by default to 1 min and it
    is configurable.

    Depends-On: https://review.opendev.org/#/c/675918
    Change-Id: Ie7cb761ad3b4a180990de2916d6210d15ec0bf50
    Closes-Bug: #1836604
    (cherry picked from commit 942dd1367eb1e379528569e719e5b9df45684857)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/675927
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8ec885d04443ee2e73813ee65f0a9ffcda2e34b6
Submitter: Zuul
Branch: stable/queens

commit 8ec885d04443ee2e73813ee65f0a9ffcda2e34b6
Author: Kamil Sambor <email address hidden>
Date: Mon Aug 12 16:53:48 2019 +0200

    Add the ability to configure ovn-remote-probe-interval

    This parameter sets inactive probe interval of the JSON
    session from ovn-controller to the OVN SB database.
    By default this it is 5s which not be sufficient in
    loaded systems or during high control-plane activity spikes,
    leading to unnecessary reconnections to OVSDB server.
    Now it is extended by default to 1 min and it
    is configurable.

    Depends-On: https://review.opendev.org/#/c/675921
    Change-Id: Ie7cb761ad3b4a180990de2916d6210d15ec0bf50
    Closes-Bug: #1836604
    (cherry picked from commit 942dd1367eb1e379528569e719e5b9df45684857)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.6.1

This issue was fixed in the openstack/tripleo-heat-templates 10.6.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.4.1

This issue was fixed in the openstack/tripleo-heat-templates 9.4.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.4.1

This issue was fixed in the openstack/tripleo-heat-templates 8.4.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.2.0

This issue was fixed in the openstack/tripleo-heat-templates 11.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.