Instances are not accessible from a compute node

[Peter Matulis]

In multi-node mode instances are not reachable from a compute node. OpenStack security group rules have been modified to allow TCP 22 to travel to the instances.

$ ssh -i ~/cloud-keys/admin-key cirros@10.20.20.192
ssh: connect to host 10.20.20.192 port 22: No route to host

See attachement for details.

[Dmitrii Shcherbakov (dmitriis)]

This is expected due to the lack of external ports on br-ex connected to the same L2 (ARP failures, no L2 connectivity to the external gateway interface).

Also, microstack uses the L3HA approach by default and distributed floating IPs are not enabled but external ports are needed for both approaches.

https://opendev.org/x/microstack/src/commit/0ef39f2865c7251264c10014a7ed2e211fe9342f/snap-overlay/templates/neutron-snap.conf.j2#L24-L25

https://docs.openstack.org/neutron/latest/admin/ovn/routing.html#l3ha-support

This has to do with the lack of L2 external interfaces by default: MicroStack may be installed on a laptop which does not have any additional available or on a server which might (but then microstack needs to be told which ones to add to br-ex).

Even if there is IP connectivity between MicroStack nodes, in order for the distributed floating IP approach to work, nodes in a cluster have to have L2 connectivity between them via L2 interfaces attached to an external bridge (in MicroStack, it's configured to be br-ex).

Due to the lack of a default L2 interface to plug into br-ex MicroStack relied on this approach from the beginning:

br-int -> patch port -> br-ex -> <no-l2-port-here>

br-ex contains an IP address and there a masquerade iptables rule in order to NAT all traffic leaving via br-ex from the subnet configured by default for a test provider network (10.20.20.0/24).

 sudo iptables-save | grep 10.20
-A POSTROUTING -s 10.20.20.0/24 ! -d 10.20.20.0/24 -j MASQUERADE

ip -4 -o a s br-ex
115: br-ex inet 10.20.20.1/24 scope global br-ex\ valid_lft forever preferred_lft forever

sudo microstack.ovs-vsctl show
dd47f44b-84a6-4ca9-838a-a0ac56c77edc
    Bridge br-ex
        datapath_type: system
        Port patch-provnet-9f21d8cf-85bf-4155-82bb-b5c06ac460ee-to-br-int
            Interface patch-provnet-9f21d8cf-85bf-4155-82bb-b5c06ac460ee-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-9f21d8cf-85bf-4155-82bb-b5c06ac460ee}
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port ovn-ef40ff-0
            Interface ovn-ef40ff-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.246.114.17"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
        Port patch-br-int-to-provnet-9f21d8cf-85bf-4155-82bb-b5c06ac460ee
            Interface patch-br-int-to-provnet-9f21d8cf-85bf-4155-82bb-b5c06ac460ee
                type: patch
                options: {peer=patch-provnet-9f21d8cf-85bf-4155-82bb-b5c06ac460ee-to-br-int}
    ovs_version: "2.14.0"

Checking that all traffic goes via an active gateway chassis can be done by validating the external_mac column is empty (the dnat_and_snat entry is for a floating IP of an instance):

sudo microstack.ovn-nbctl lr-nat-list neutron-854e22b3-ebb1-40e3-b791-e81a91decb80
TYPE EXTERNAL_IP LOGICAL_IP EXTERNAL_MAC LOGICAL...

[Dmitrii Shcherbakov (dmitriis)]

In order to address this, MicroStack needs to gain an ability to configure bridge ports used for provider network connectivity and there will be some input and preparation required from a user to set this up for a particular environment.