tripleo

With OVN deployments, ovn db servers are flooding the logs

Bug #1781585 reported by Numan Siddique
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Numan Siddique
tripleo rocky-3

Bug Description

In the controller nodes below error messages are seen in ovn db server logs

*****
/var/log/containers/openvswitch/ovsdb-server-nb.log
>
> 2018-07-12T05:16:35.383Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server-nb.log
> 2018-07-12T05:16:35.387Z|00002|socket_util|ERR|6641:172.17.1.18: bind: Cannot assign requested address
> 2018-07-12T05:16:35.387Z|00003|ovsdb_jsonrpc_server|ERR|ptcp:6641:172.17.1.18: listen failed: Cannot assign requested address
> 2018-07-12T05:16:35.390Z|00004|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.9.0
> 2018-07-12T05:16:35.390Z|00005|socket_util|ERR|6641:172.17.1.18: bind: Cannot assign requested address
> 2018-07-12T05:16:35.390Z|00006|ovsdb_jsonrpc_server|ERR|ptcp:6641:172.17.1.18: listen failed: Cannot assign requested address
> 2018-07-12T05:16:35.390Z|00007|reconnect|INFO|tcp:192.0.2.254:6641: connecting...
> 2018-07-12T05:16:35.390Z|00008|socket_util|ERR|6641:172.17.1.18: bind: Cannot assign requested address
> 2018-07-12T05:16:35.390Z|00009|ovsdb_jsonrpc_server|ERR|ptcp:6641:172.17.1.18: listen failed: Cannot assign requested address
> 2018-07-12T05:16:35.399Z|00010|socket_util|ERR|6641:172.17.1.18: bind: Cannot assign requested address
> 2018-07-12T05:16:35.399Z|00011|ovsdb_jsonrpc_server|ERR|ptcp:6641:172.17.1.18: listen failed: Cannot assign requested address
> 2018-07-12T05:16:35.400Z|00012|socket_util|ERR|6641:172.17.1.18: bind: Cannot assign requested address
> 2018-07-12T05:16:35.400Z|00013|ovsdb_jsonrpc_server|ERR|ptcp:6641:172.17.1.18: listen failed: Cannot assign requested address
> 2018-07-12T05:16:35.401Z|00014|socket_util|ERR|6641:172.17.1.18: bind: Cannot assign requested address
> 2018-07-12T05:16:35.402Z|00015|socket_util|ERR|6641:172.17.1.18: bind: Cannot assign requested address
> 2018-07-12T05:16:35.402Z|00016|socket_util|ERR|6641:172.17.1.18: bind: Cannot assign requested address
> 2018-07-12T05:16:35.498Z|00017|socket_util|ERR|6641:172.17.1.18: bind: Cannot assign requested address
> 2018-07-12T05:16:35.499Z|00018|socket_util|ERR|6641:172.17.1.18: bind: Cannot assign requested address
>
*****

Numan Siddique (numansiddique)
Changed in tripleo:
assignee: nobody → Numan Siddique (numansiddique)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/582520

Emilien Macchi (emilienm)
Changed in tripleo:
milestone: none → rocky-3
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/583335

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/582520
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5ed628f3534718dd61b66d4438fd8b33fa1362f8
Submitter: Zuul
Branch: master

commit 5ed628f3534718dd61b66d4438fd8b33fa1362f8
Author: Numan Siddique <email address hidden>
Date: Tue Jul 17 23:47:17 2018 +0530

    OVN: Set sysctl value 'net.ipv4.ip_nonlocal_bind' to 1

    When OVN DBs pacemaker bundle is started on the controller nodes,
    all the ovsdb-servers are configured to listen on the virtual ip
    (on ports 6641/6642). But only master node is configured with virtual ip.
    On the other nodes, we see the below error messages

    ovsdb_jsonrpc_server|ERR|ptcp:6641:172.17.1.18: listen failed: Cannot assign
    requested address.

    The commit [1] addressed this issue in puppet-tripleo by setting the
    sysctl value 'net.ipv4.ip_nonlocal_bind' to 1 using puppet's sysctl::value.
    But the ovn-dbs.yaml service file is missing this puppet tag when the
    container 'ovn_dbs_init_bundle' is started.

    We can fix the issue by defining this tag, but this also requires setting
    'privileged=true' when starting the docker container.

    Instead, this patch makes use of the t-h-t param 'KernelIpNonLocalBind'
    which was introduced in this commit [2].

    [1] - I6f762534350a3f96696c87ccd2d14545dccc8a0b
    [2] - Idd3d432b8f7eb573d94cd56be8e05614510ebddf

    Change-Id: I5ae8cd368bcd58810b18e172ee685fdbf0e48d98
    Closes-bug: #1781585

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to puppet-tripleo (master)

Reviewed: https://review.openstack.org/582523
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=cac7047169979c34f4967bd5901ff27659a6c680
Submitter: Zuul
Branch: master

commit cac7047169979c34f4967bd5901ff27659a6c680
Author: Numan Siddique <email address hidden>
Date: Fri Jul 13 17:56:12 2018 +0530

    Remove unused code in ovn dbs bundle to set the sysctl value

    The docker/services/pacemaker/ovn-dbs.yaml service file doesn't
    define the puppet tag 'sysctl::value'. So this patch removes
    the puppet code to set the sysctl - net.ipv4.ip_nonlocalbind' to 1.
    Instead of adding this tag, the service file sets the above sysctl
    value using the host_prep_tasks section in this patch - [1]

    [1] - https://review.openstack.org/#/c/582520/
    Change-Id: I1076c88051cc48a263996024fcdfe8d917537d20
    Related-bug: #1781585

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Numan Siddique (<email address hidden>) on branch: master
Review: https://review.openstack.org/583335

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.0.0.0b4

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b4 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/586533

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.openstack.org/586533
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e0adcc6e0e31bf27e582e12a4ca1b95e7a4b9b17
Submitter: Zuul
Branch: stable/queens

commit e0adcc6e0e31bf27e582e12a4ca1b95e7a4b9b17
Author: Numan Siddique <email address hidden>
Date: Tue Jul 17 23:47:17 2018 +0530

    OVN: Set sysctl value 'net.ipv4.ip_nonlocal_bind' to 1

    When OVN DBs pacemaker bundle is started on the controller nodes,
    all the ovsdb-servers are configured to listen on the virtual ip
    (on ports 6641/6642). But only master node is configured with virtual ip.
    On the other nodes, we see the below error messages

    ovsdb_jsonrpc_server|ERR|ptcp:6641:172.17.1.18: listen failed: Cannot assign
    requested address.

    The commit [1] addressed this issue in puppet-tripleo by setting the
    sysctl value 'net.ipv4.ip_nonlocal_bind' to 1 using puppet's sysctl::value.
    But the ovn-dbs.yaml service file is missing this puppet tag when the
    container 'ovn_dbs_init_bundle' is started.

    We can fix the issue by defining this tag, but this also requires setting
    'privileged=true' when starting the docker container.

    Instead, this patch makes use of the t-h-t param 'KernelIpNonLocalBind'
    which was introduced in this commit [2].

    [1] - I6f762534350a3f96696c87ccd2d14545dccc8a0b
    [2] - Idd3d432b8f7eb573d94cd56be8e05614510ebddf

    Change-Id: I5ae8cd368bcd58810b18e172ee685fdbf0e48d98
    Closes-bug: #1781585
    (cherry picked from 5ed628f3534718dd61b66d4438fd8b33fa1362f8)
    Conflicts:
        environments/neutron-ml2-ovn-dvr-ha.yaml
        environments/neutron-ml2-ovn-ha.yaml

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.0.6

This issue was fixed in the openstack/tripleo-heat-templates 8.0.6 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.