[focal][ussuri] "connection dropped (Protocol error)" in ovsdb-server-nb.log and ovsdb-server-sb.log

[TWENTY |20]

Ubuntu version: Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-67-generic x86_64)
OpenStack version: Openstack Ussuri
Charm version: #5

After fresh openstack deployment (with vault and automatically generate a self-signed root CA) I get the following recurring errors in ovn-central logs:

==> /var/log/ovn/ovsdb-server-nb.log <==
2021-03-22T13:12:17.750Z|00038|reconnect|WARN|ssl:127.0.0.1:44454: connection dropped (Protocol error)
2021-03-22T13:18:16.506Z|00039|stream_ssl|WARN|SSL_accept: system error (Success)
2021-03-22T13:18:16.506Z|00040|jsonrpc|WARN|ssl:127.0.0.1:44302: receive error: Protocol error
2021-03-22T13:18:16.506Z|00041|reconnect|WARN|ssl:127.0.0.1:44302: connection dropped (Protocol error)
2021-03-22T13:23:09.124Z|00042|stream_ssl|WARN|SSL_accept: system error (Success)
2021-03-22T13:23:09.124Z|00043|jsonrpc|WARN|ssl:127.0.0.1:39020: receive error: Protocol error
2021-03-22T13:23:09.124Z|00044|reconnect|WARN|ssl:127.0.0.1:39020: connection dropped (Protocol error)
2021-03-22T13:28:18.537Z|00045|stream_ssl|WARN|SSL_accept: system error (Success)
2021-03-22T13:28:18.537Z|00046|jsonrpc|WARN|ssl:127.0.0.1:35000: receive error: Protocol error
2021-03-22T13:28:18.537Z|00047|reconnect|WARN|ssl:127.0.0.1:35000: connection dropped (Protocol error)

==> /var/log/ovn/ovsdb-server-sb.log <==
2021-03-22T13:12:17.752Z|00034|reconnect|WARN|ssl:127.0.0.1:44692: connection dropped (Protocol error)
2021-03-22T13:18:16.507Z|00035|stream_ssl|WARN|SSL_accept: system error (Success)
2021-03-22T13:18:16.507Z|00036|jsonrpc|WARN|ssl:127.0.0.1:44540: receive error: Protocol error
2021-03-22T13:18:16.507Z|00037|reconnect|WARN|ssl:127.0.0.1:44540: connection dropped (Protocol error)
2021-03-22T13:23:09.125Z|00038|stream_ssl|WARN|SSL_accept: system error (Success)
2021-03-22T13:23:09.125Z|00039|jsonrpc|WARN|ssl:127.0.0.1:39258: receive error: Protocol error
2021-03-22T13:23:09.126Z|00040|reconnect|WARN|ssl:127.0.0.1:39258: connection dropped (Protocol error)
2021-03-22T13:28:18.539Z|00041|stream_ssl|WARN|SSL_accept: system error (Success)
2021-03-22T13:28:18.539Z|00042|jsonrpc|WARN|ssl:127.0.0.1:35238: receive error: Protocol error
2021-03-22T13:28:18.539Z|00043|reconnect|WARN|ssl:127.0.0.1:35238: connection dropped (Protocol error)

I could not see any problems in Openstack / neutron-api. Everything seems to be working normally.
The ports in the errors always seem to change. This errors came up in all 3 ovn-central units.

[Aurelien Lourot (aurelien-lourot)]

I'm seeing this as well when deploying a focal-xena OVN bundle [1] on s390x. The system is working well, i.e. it's possible to create instances, give them a floating IP and SSH to them. But the nb and sb logs are full of these exact same SSL warnings.

[1] https://github.com/ubuntu-openstack/zopenstack/blob/master/bundles/lpar/focal-xena-ovn-next.yaml

[Shunde Zhang (shunde-zhang)]

This is caused by ovn-central's update status hook, which uses nc to check ovn's listening ports.
To reproduce this, running 'nc -z 0.0.0.0 6641' on a ovn-central unit will cause such error messages to print out in /var/log/ovn/ovsdb-server-nb.log

[Edward Hope-Morley (hopem)]

Correct, reactive openstack/ovn charms check service status on every update-status hook run which is every 5 mins by default (which matches with the timestamps in the logs above) and calls [1] as @shunde-zhang points out. Because that behaviour is not specific to the ovn-central charm it might be hard to change but at least those warnings are benign and maybe we can alter loglevels to make them go away.

[1] https://github.com/juju/charm-helpers/blob/b78107dc750644b1d868ff4a61748086783e02bd/charmhelpers/contrib/network/ip.py#L530

[Edward Hope-Morley (hopem)]

See also https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1979070

[Edward Hope-Morley (hopem)]

So basically the issue is that netcat doesn't send a close_notify. There are other commands that we could use to test the connection like openssl and this would also have the added benefit of testing the certs e.g.

echo "Q"|openssl s_client -connect 127.0.0.1:6641 -key /etc/ovn/key_host -cert /etc/ovn/cert_host

will do the same thing as the charm is currently doing.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/x/charm-ovn-central/+/911091

[Edward Hope-Morley (hopem)]

charm-helpers patch: https://github.com/juju/charm-helpers/pull/881

[macchese (max-liccardo)]

same iusse on:

openstack: bundle-jammy-2023.2.yaml

App Version Status Scale Charm Channel Rev Exposed Message
ovn-central 22.09.1 active 3 ovn-central 23.09/stable 158

tail -f ovsdb-server-nb.log
2024-04-12T07:14:07.103Z|03574|reconnect|WARN|ssl:127.0.0.1:39078: connection dropped (Protocol error)
2024-04-12T07:18:30.637Z|03575|stream_ssl|WARN|SSL_accept: error:0A000126:SSL routines::unexpected eof while reading

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-central/+/911091
Committed: https://opendev.org/x/charm-ovn-central/commit/fafa6d418afe04d9b79326a621091d9d6c5f74f4
Submitter: "Zuul (22348)"
Branch: master

commit fafa6d418afe04d9b79326a621091d9d6c5f74f4
Author: Edward Hope-Morley <email address hidden>
Date: Tue Mar 5 11:47:26 2024 +0000

    Use SSL compatible service status check

    Override the layer_ovn method used to check if services
    are running so that we can use SSL connections instead
    of netcat since the latter causes problems with
    ovsdb-server due to connections not being closed
    properly.

    Closes-Bug: #1920770
    Change-Id: I6b2c09c643c6fddfb48dc15d6272e58303fae265

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/23.09
Review: https://review.opendev.org/c/x/charm-ovn-central/+/916471

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/23.03
Review: https://review.opendev.org/c/x/charm-ovn-central/+/916473

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/22.09
Review: https://review.opendev.org/c/x/charm-ovn-central/+/916475

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/22.03
Review: https://review.opendev.org/c/x/charm-ovn-central/+/916476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-central/+/916471
Committed: https://opendev.org/x/charm-ovn-central/commit/05722e0fbc9474c98c373a589c4f86116cdd9b23
Submitter: "Zuul (22348)"
Branch: stable/23.09

commit 05722e0fbc9474c98c373a589c4f86116cdd9b23
Author: Edward Hope-Morley <email address hidden>
Date: Tue Mar 5 11:47:26 2024 +0000

    Use SSL compatible service status check

    Override the layer_ovn method used to check if services
    are running so that we can use SSL connections instead
    of netcat since the latter causes problems with
    ovsdb-server due to connections not being closed
    properly.

    Closes-Bug: #1920770
    Change-Id: I6b2c09c643c6fddfb48dc15d6272e58303fae265
    (cherry picked from commit fafa6d418afe04d9b79326a621091d9d6c5f74f4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-central/+/916473
Committed: https://opendev.org/x/charm-ovn-central/commit/0a7904fba22b196a9f2484f4d55bc6f98b5a71fb
Submitter: "Zuul (22348)"
Branch: stable/23.03

commit 0a7904fba22b196a9f2484f4d55bc6f98b5a71fb
Author: Edward Hope-Morley <email address hidden>
Date: Tue Mar 5 11:47:26 2024 +0000

    Use SSL compatible service status check

    Override the layer_ovn method used to check if services
    are running so that we can use SSL connections instead
    of netcat since the latter causes problems with
    ovsdb-server due to connections not being closed
    properly.

    Closes-Bug: #1920770
    Change-Id: I6b2c09c643c6fddfb48dc15d6272e58303fae265
    (cherry picked from commit fafa6d418afe04d9b79326a621091d9d6c5f74f4)
    (cherry picked from commit 05722e0fbc9474c98c373a589c4f86116cdd9b23)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-central/+/916475
Committed: https://opendev.org/x/charm-ovn-central/commit/202684c9f70373bb3531653e134fcfababebb9fb
Submitter: "Zuul (22348)"
Branch: stable/22.09

commit 202684c9f70373bb3531653e134fcfababebb9fb
Author: Edward Hope-Morley <email address hidden>
Date: Tue Mar 5 11:47:26 2024 +0000

    Use SSL compatible service status check

    Override the layer_ovn method used to check if services
    are running so that we can use SSL connections instead
    of netcat since the latter causes problems with
    ovsdb-server due to connections not being closed
    properly.

    Closes-Bug: #1920770
    Change-Id: I6b2c09c643c6fddfb48dc15d6272e58303fae265
    (cherry picked from commit fafa6d418afe04d9b79326a621091d9d6c5f74f4)
    (cherry picked from commit 05722e0fbc9474c98c373a589c4f86116cdd9b23)
    (cherry picked from commit 0a7904fba22b196a9f2484f4d55bc6f98b5a71fb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-central/+/916476
Committed: https://opendev.org/x/charm-ovn-central/commit/633d78bdbdd6bdbdbcece631fd8da4cad7b5ee8d
Submitter: "Zuul (22348)"
Branch: stable/22.03

commit 633d78bdbdd6bdbdbcece631fd8da4cad7b5ee8d
Author: Edward Hope-Morley <email address hidden>
Date: Tue Mar 5 11:47:26 2024 +0000

    Use SSL compatible service status check

    Override the layer_ovn method used to check if services
    are running so that we can use SSL connections instead
    of netcat since the latter causes problems with
    ovsdb-server due to connections not being closed
    properly.

    Closes-Bug: #1920770
    Change-Id: I6b2c09c643c6fddfb48dc15d6272e58303fae265
    (cherry picked from commit fafa6d418afe04d9b79326a621091d9d6c5f74f4)
    (cherry picked from commit 05722e0fbc9474c98c373a589c4f86116cdd9b23)
    (cherry picked from commit 0a7904fba22b196a9f2484f4d55bc6f98b5a71fb)
    (cherry picked from commit 202684c9f70373bb3531653e134fcfababebb9fb)