[bionic][train->ussuri] neutron-gateway ovs ports missing from br-int after OpenStack upgrade

[Alex Kavanagh]

This may not be actually the fault of the charm, but reporting here to be able to track it and work gather information about the bug.

Essentially, after a mojo CI run on bionic, train -> ussuri, there are ports missing from the br-int. This means that the network loses connectivity; e.g. you can't ping hosts, VIPs also won't attach, etc.

Also, launching NEW instances also fails until the neutron-gateway is rebooted. Restarting the various neutron-* and ovs services doesn't appear to help. Note that the ovs db and "ovs-vsctl show" *do* seem to be consistent; this might point to a neutron issue, rather than a charm issue, but it could be due to configuration. There doesn't appear to be anything in the logs, but I'm now going to try two further tests:

  1. Deploy a ussuri using mojo (but obviously with config option for legacy support on neutron).
  2. Re-try the upgrade but switch as much logging on as possible to see when the ports get deleted.

[Alex Kavanagh (ajkavanagh)]

I've deployed ussuri directly using a bashed up copy of the bionic-train mojo openstack upgrade scripts (but obviously, without trying to do an upgrade) and the networking works fine.

Thus, it definitely seems to be something to do with the upgrade action. I wonder if the packages do anything?

[Alex Kavanagh (ajkavanagh)]

This bug has transformed into https://bugs.launchpad.net/cloud-archive/+bug/1885264 -- i.e. it's a packaging bug that exposes a weakness/bug in one of the neutron agents that doesn't reconstruct the ovs ports properly if they've inadvertently been deleted.