Package versions before we start: $ juju run --application ovn-central 'dpkg -l |grep ovn' - Stdout: | ii ovn-central 20.03.2-0ubuntu0.20.04.4 amd64 OVN central components ii ovn-common 20.03.2-0ubuntu0.20.04.4 amd64 OVN common components UnitId: ovn-central/0 - Stdout: | ii ovn-central 20.03.2-0ubuntu0.20.04.4 amd64 OVN central components ii ovn-common 20.03.2-0ubuntu0.20.04.4 amd64 OVN common components UnitId: ovn-central/1 - Stdout: | ii ovn-central 20.03.2-0ubuntu0.20.04.4 amd64 OVN central components ii ovn-common 20.03.2-0ubuntu0.20.04.4 amd64 OVN common components UnitId: ovn-central/2 $ juju run --application ovn-chassis 'dpkg -l |grep ovn' - Stdout: | ii neutron-ovn-metadata-agent 2:16.4.2-0ubuntu4 all Neutron is a virtual network service for Openstack - OVN metadata agent ii ovn-common 20.03.2-0ubuntu0.20.04.4 amd64 OVN common components ii ovn-host 20.03.2-0ubuntu0.20.04.4 amd64 OVN host components UnitId: ovn-chassis/0 - Stdout: | ii neutron-ovn-metadata-agent 2:16.4.2-0ubuntu4 all Neutron is a virtual network service for Openstack - OVN metadata agent ii ovn-common 20.03.2-0ubuntu0.20.04.4 amd64 OVN common components ii ovn-host 20.03.2-0ubuntu0.20.04.4 amd64 OVN host components UnitId: ovn-chassis/1 Ping running instances: $ ping 10.78.95.55 PING 10.78.95.55 (10.78.95.55) 56(84) bytes of data. 64 bytes from 10.78.95.55: icmp_seq=1 ttl=63 time=1.80 ms 64 bytes from 10.78.95.55: icmp_seq=2 ttl=63 time=1.22 ms 64 bytes from 10.78.95.55: icmp_seq=3 ttl=63 time=1.06 ms ... $ ping 10.78.95.162 PING 10.78.95.162 (10.78.95.162) 56(84) bytes of data. 64 bytes from 10.78.95.162: icmp_seq=1 ttl=63 time=1.08 ms 64 bytes from 10.78.95.162: icmp_seq=2 ttl=63 time=0.545 ms 64 bytes from 10.78.95.162: icmp_seq=3 ttl=63 time=0.516 ms ... Ensure OVN DNS interception/resolution is enabled and working: ubuntu@zaza-neutrontests-ins-1:~$ dig zaza-neutrontests-ins-2 @10.78.95.1 ... ;; ADDITIONAL SECTION: zaza-neutrontests-ins-2. 3600 IN A 192.168.0.180 Ensure ovn-controllers picks up version mismatch prior to upgrade: Note that the backported version mismatch handling code does not have the additional version mismatch check in the incremental processing engine that later versions have, this means that we need to ensure the main loop version mismatch check has run prior to allowing northd to fill database tables after an upgrade. We will just have to deal with this in the charms and/or as part of upgrade documentation. Force the version mismatch to happen before we actually perform the upgrade to ensure the ovn-controller does not make any mistakes. $ juju run --application ovn-central 'systemctl stop ovn-northd; systemctl mask ovn-northd' $ juju run --unit ovn-central/0 'ovn-sbctl set sb-global . options:northd_internal_version="20.03.2-2.7.0-42.1"' $ juju run --application ovn-chassis 'tail -1 /var/log/ovn/ovn-controller.log' - Stdout: | 2022-11-06T11:04:26.305Z|00017|main|WARN|controller version - 20.03.2-2.7.0-42.0 mismatch with northd version - 20.03.2-2.7.0-42.1 UnitId: ovn-chassis/0 - Stdout: | 2022-11-06T11:04:26.272Z|00021|main|WARN|controller version - 20.03.2-2.7.0-42.0 mismatch with northd version - 20.03.2-2.7.0-42.1 UnitId: ovn-chassis/1 Confirm OVN DNS interception/resolution is still working: ubuntu@zaza-neutrontests-ins-1:~$ dig zaza-neutrontests-ins-2 @10.78.95.1 ... ;; ADDITIONAL SECTION: zaza-neutrontests-ins-2. 3600 IN A 192.168.0.180 Upgrade packages on central units: $ juju config ovn-central ovn-source=cloud:focal-ovn-22.03 $ juju run --application ovn-central 'systemctl unmask ovn-northd; systemctl restart ovn-northd' Confirm instances are still responding: 64 bytes from 10.78.95.55: icmp_seq=1167 ttl=63 time=1.31 ms 64 bytes from 10.78.95.55: icmp_seq=1168 ttl=63 time=1.06 ms 64 bytes from 10.78.95.55: icmp_seq=1169 ttl=63 time=1.07 ms ... 64 bytes from 10.78.95.162: icmp_seq=1145 ttl=63 time=0.569 ms 64 bytes from 10.78.95.162: icmp_seq=1146 ttl=63 time=0.564 ms 64 bytes from 10.78.95.162: icmp_seq=1147 ttl=63 time=0.937 ms ... Confirm OVN DNS interception/resolution is still working: ubuntu@zaza-neutrontests-ins-1:~$ dig zaza-neutrontests-ins-2 @10.78.95.1 ... ;; ADDITIONAL SECTION: zaza-neutrontests-ins-2. 3600 IN A 192.168.0.180 Upgrade packages on chassis units: $ juju config ovn-chassis ovn-source=cloud:focal-ovn-22.03 Collect instance ping statistics after completing the upgrade: 64 bytes from 10.78.95.55: icmp_seq=1299 ttl=63 time=1.02 ms 64 bytes from 10.78.95.55: icmp_seq=1300 ttl=63 time=1.04 ms 64 bytes from 10.78.95.55: icmp_seq=1301 ttl=63 time=1.01 ms ^C --- 10.78.95.55 ping statistics --- 1301 packets transmitted, 1293 received, 0.614912% packet loss, time 1301821ms rtt min/avg/max/mdev = 0.871/1.296/28.365/1.310 ms 64 bytes from 10.78.95.162: icmp_seq=1264 ttl=63 time=0.642 ms 64 bytes from 10.78.95.162: icmp_seq=1265 ttl=63 time=0.711 ms 64 bytes from 10.78.95.162: icmp_seq=1266 ttl=63 time=0.564 ms ^C --- 10.78.95.162 ping statistics --- 1266 packets transmitted, 1261 received, 0.394945% packet loss, time 1292577ms rtt min/avg/max/mdev = 0.434/0.677/28.644/0.921 ms