[groovy-victoria] Can't SSH to instance after OVS to OVN migration

[Aurelien Lourot]

Seen on OSCI:
https://review.opendev.org/c/openstack/charm-neutron-openvswitch/+/761576
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-openvswitch/761576/4/8018/index.html

Attempting to ssh to instance-1(172.17.110.206)
ERROR
======================================================================
ERROR: test_instances_have_networking (zaza.openstack.charm_tests.neutron.tests.NeutronNetworkingTest)
Validate North/South and East/West networking.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.COfrMItW8d/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/neutron/tests.py", line 905, in test_instances_have_networking
    self.check_connectivity(instance_1, instance_2)
  File "/tmp/tmp.COfrMItW8d/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/neutron/tests.py", line 818, in check_connectivity
    instance_1, instance_2, verify, mtu_min)
  File "/tmp/tmp.COfrMItW8d/func/lib/python3.5/site-packages/tenacity/__init__.py", line 333, in wrapped_f
    return self(f, *args, **kw)
  File "/tmp/tmp.COfrMItW8d/func/lib/python3.5/site-packages/tenacity/__init__.py", line 423, in __call__
    do = self.iter(retry_state=retry_state)
  File "/tmp/tmp.COfrMItW8d/func/lib/python3.5/site-packages/tenacity/__init__.py", line 372, in iter
    raise retry_exc.reraise()
  File "/tmp/tmp.COfrMItW8d/func/lib/python3.5/site-packages/tenacity/__init__.py", line 189, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/tmp/tmp.COfrMItW8d/func/lib/python3.5/site-packages/tenacity/__init__.py", line 426, in __call__
    result = fn(*args, **kwargs)
  File "/tmp/tmp.COfrMItW8d/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/neutron/tests.py", line 679, in validate_instance_can_reach_other
    password=password, privkey=privkey, verify=verify)
  File "/tmp/tmp.COfrMItW8d/func/lib/python3.5/site-packages/zaza/openstack/utilities/openstack.py", line 2647, in ssh_command
    ssh.connect(ip, username=username, password=None, pkey=key)
  File "/tmp/tmp.COfrMItW8d/func/lib/python3.5/site-packages/paramiko/client.py", line 368, in connect
    raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.17.110.206

Still happening after lp:1912471 's fix. Also ovn-controller.log shows a lot of "connection dropped" as in lp:1903028

[Frode Nordahl (fnordahl)]

This is a smoking gun, from /var/log/ovn/ovn-controller.log:
2021-01-24T17:04:46.334Z|00052|vconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: version negotiation failed (we support version 0x06, peer supports versions 0x01, 0x04, 0x05)

May be we need to extend this list depending on which OVN release is in use:
https://github.com/openstack-charmers/charm-layer-ovn/blob/b5a86ead9e130880a8a9a8c6c5cc861aeef3e9eb/lib/charms/ovn_charm.py#L652

[Frode Nordahl (fnordahl)]

Interestingly if I deploy a Groovy chassis it has no problems talking to the bridge:
2021-01-27T08:55:15.222Z|00008|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch
2021-01-27T08:55:15.222Z|00009|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...
2021-01-27T08:55:15.222Z|00010|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected

# ovs-vsctl list bridge
...
protocols : [OpenFlow13, OpenFlow15]

So perhaps this is an Open vSwitch bug?

[Frode Nordahl (fnordahl)]

Reminds me of bug 1852221

[Frode Nordahl (fnordahl)]

Indeed:
root@brave-horse:/home/ubuntu# tail -f /var/log/ovn/ovn-controller.log
2021-01-27T10:27:24.331Z|00027|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2021-01-27T10:27:32.333Z|00028|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2021-01-27T10:27:40.336Z|00029|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2021-01-27T10:27:48.340Z|00030|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2021-01-27T10:27:56.343Z|00031|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2021-01-27T10:28:04.344Z|00032|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2021-01-27T10:28:12.349Z|00033|vconn|WARN|Dropped 5 log messages in last 40 seconds (most recently, 8 seconds ago) due to excessive rate
2021-01-27T10:28:12.349Z|00034|vconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: version negotiation failed (we support version 0x06, peer supports versions 0x01, 0x04, 0x05)
2021-01-27T10:28:12.349Z|00035|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2021-01-27T10:28:20.350Z|00036|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2021-01-27T10:28:28.352Z|00037|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Protocol error)
^C
root@brave-horse:/home/ubuntu# systemctl restart ovs-vswitchd
root@brave-horse:/home/ubuntu# tail -f /var/log/ovn/ovn-controller.log
2021-01-27T10:28:12.349Z|00033|vconn|WARN|Dropped 5 log messages in last 40 seconds (most recently, 8 seconds ago) due to excessive rate
2021-01-27T10:28:12.349Z|00034|vconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: version negotiation failed (we support version 0x06, peer supports versions 0x01, 0x04, 0x05)
2021-01-27T10:28:12.349Z|00035|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2021-01-27T10:28:20.350Z|00036|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2021-01-27T10:28:28.352Z|00037|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Protocol error)
2021-01-27T10:28:36.356Z|00038|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2021-01-27T10:28:38.844Z|00039|jsonrpc|WARN|unix:/var/run/openvswitch/db.sock: receive error: Connection reset by peer
2021-01-27T10:28:38.844Z|00040|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection dropped (Connection reset by peer)
2021-01-27T10:28:39.845Z|00041|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2021-01-27T10:28:39.845Z|00042|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2021-01-27T10:28:44.356Z|00043|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected
2021-01-27T10:28:44.358Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch
2021-01-27T10:28:44.358Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...
2021-01-27T10:28:44.399Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected
^C
root@brave-ho...