Unable to open stream to ssl / Could not retrieve schema - wrong port?

[Dominik Bender]

Unable to open stream to ssl:10.6.68.8:16642 to retrieve schema: Connection refused

Exception: Could not retrieve schema from ssl:10.6.68.8:16642,ssl:10.6.68.40:16642,ssl:10.6.68.20:16642

Connection to port 16642 but OVN Southbound listen on 6644. Whats going wrong?

neutron log
-------------
2021-07-05 05:29:12.060 99687 ERROR ovsdbapp.backend.ovs_idl.idlutils [req-205e4a2a-b4cc-4809-b8ca-719503086ccd - - - - -] Unable to open stream to ssl:10.6.68.8:16642 to retrieve schema: Connection refused
2021-07-05 05:29:12.061 99687 ERROR ovsdbapp.backend.ovs_idl.idlutils [req-205e4a2a-b4cc-4809-b8ca-719503086ccd - - - - -] Unable to open stream to ssl:10.6.68.40:16642 to retrieve schema: Connection refused
2021-07-05 05:29:12.062 99687 ERROR ovsdbapp.backend.ovs_idl.idlutils [req-205e4a2a-b4cc-4809-b8ca-719503086ccd - - - - -] Unable to open stream to ssl:10.6.68.20:16642 to retrieve schema: Connection refused
2021-07-05 05:29:12.062 99687 ERROR neutron.service [req-205e4a2a-b4cc-4809-b8ca-719503086ccd - - - - -] Unrecoverable error: please check log for details.: Exception: Could not retrieve schema from ssl:10.6.68.8:16642,ssl:10.6.68.40:16642,ssl:10.6.68.20:16642
------------

UnitId: ovn-central/3
---------------------
- Stdout: |
    3560
    Name: OVN_Southbound
    Cluster ID: 12eb (12eb0ace-1165-4cf7-8bd4-7ba97a16a541)
    Server ID: 3560 (3560f236-fa8b-4d3e-870b-1c7a34827973)
    Address: ssl:10.6.68.20:6644
    Status: cluster member
    Role: follower
    Term: 1
    Leader: 716b
    Vote: unknown

    Election timer: 4000
    Log: [2, 15]
    Entries not yet committed: 0
    Entries not yet applied: 0
    Connections: ->0000 ->0000 <-716b <-cc6b
    Disconnections: 0
    Servers:
        3560 (3560 at ssl:10.6.68.20:6644) (self)
        716b (716b at ssl:10.6.68.8:6644) last msg 1012 ms ago
        cc6b (cc6b at ssl:10.6.68.33:6644) last msg 25859939 ms ago
---------

charms config
---------------------
  neutron-api-plugin-ovn:
    charm: cs:neutron-api-plugin-ovn-6
  neutron-api:
    charm: cs:neutron-api-294
    num_units: 3
    bindings:
      "": *internal-space
      public: *public-space
      internal: *internal-space
      shared-db: *internal-space
    options:
      neutron-security-groups: true
      flat-network-providers: physnet1
      worker-multiplier: *worker-multiplier
      openstack-origin: *openstack-origin
      vip: 10.6.66.2 10.6.162.2
      vip_cidr: 19
      use-internal-endpoints: True
      enable-sriov: True
      enable-ml2-port-security: True
      enable-vlan-trunking: True
      default-tenant-network-type: vlan
      vlan-ranges: physnet1:2200:3500
      vni-ranges: 1001:10000
      global-physnet-mtu: 1500
      quota-floatingip: -1
      quota-network: -1
      quota-port: -1
      quota-router: -1
      quota-subnet: -1
      quota-vip: -1
      region: de1
      nagios_servicegroups: network
ovn-central:
    charm: cs:ovn-central-7
    num_units: 3
    bindings:
      "": *internal-space
      ovsdb: *internal-space
    options:
      source: *openstack-origin
      nagios_servicegroups: network
    to:
    - lxd:0
    - lxd:1
    - lxd:2
ovn-chassis:
    charm: cs:ovn-chassis-14
    bindings:
      "": *internal-space
      ovsdb: *internal-space
      data: *overlay-space
    options:
      ovn-bridge-mappings: physnet1:br-ex
      bridge-interface-mappings: *data-port
--------

System-Release: Ubuntu 20.04 LTS

[Frode Nordahl (fnordahl)]

Hello Dominik, thank you for your bug report.

As noted in the charm documentation [0] the OVN databases are configured to listen on several ports.

The 16642 port is opened for administrative access to the Southbound database by consumers of the ovsdb-peer and ovsdb-cms relations [1]. The 6644 port is used for cluster-internal communication between the OVSDB instances and is not for external consumption.

Are the neutron-api units and ovn-central units connected to the same L2 broadcast domain? If not could there be an external firewall in the way somewhere? Could there be missing bindings for the neutron-api-plugin-ovn relations?

Would you be able to provide me with the following:
- Output of `ufw status` from the ovn-central units
- IP addresses of neutron-api units
- Packet capture of a connection attempt from the neutron-api unit which shows source/destination IPs

0: https://jaas.ai/ovn-central
1: https://opendev.org/x/charm-ovn-central/src/commit/8e305de633f701cd52775685286c02cd1a09e7ed/src/reactive/ovn_central_handlers.py#L116-L127

[Dominik Bender (ephermeral)]

Thanks for your quick response. I changed the maas settings from multiple gateways to only one (only the internal-space have an gateway configured) and added new space bindings because of new general oam-space and octavia binding (see below). After I redeployed the entire bundle now it works.

ssl:10.6.68.20:16642: connected

I think the additional space bindings should have normally no effect because the neutron-api-plugin-ovn is a subordinate charm and the neutron-load-balancer binding must not be explicit configured, right?

Mutliple gateways should also no problem. I readed it should handled by maas/juju.

---
  neutron-api-plugin-ovn:
    charm: cs:neutron-api-plugin-ovn-6
    bindings:
      "": *internal-space
  neutron-api:
    charm: cs:neutron-api-294
    num_units: 3
    bindings:
      "": *internal-space
      public: *public-space
      admin: *oam-space
      internal: *internal-space
      shared-db: *internal-space
      neutron-load-balancer: *internal-space

[Launchpad Janitor (janitor)]

[Expired for OpenStack neutron-api charm because there has been no activity for 60 days.]