Comment 10 for bug 2059721

Revision history for this message
Vadim Kuznetsov (vakuznet) wrote :

Here is my explanation:
1. According to ovn docs https://docs.ovn.org/en/latest/intro/install/ovn-upgrades.html
"The only step required after upgrading the packages is to restart ovn-northd, which automatically restarts the databases and upgrades the database schema, as well.
if you’re using a Linux distribution with systemd:
$ sudo systemctl restart ovn-northd

which is not true in Ubuntu case
```
ExecStart=/usr/share/ovn/scripts/ovn-ctl start_northd --ovn-manage-ovsdb=no --no-monitor $OVN_CTL_OPTS
```
ovn-manage-ovsdb=no, so it will not restart the DBs.

2. DB restarts by ovn-ovsdb-server-nb.service and ovn-ovsdb-server-sb.service
run_nb_ovsdb and run_sb_ovsdb options in ovn-ctl
```
ExecStart=/usr/share/ovn/scripts/ovn-ctl run_nb_ovsdb $OVN_CTL_OPTS
```

3. This commit fixed Cluster OVN database upgrade https://github.com/ovn-org/ovn/commit/67e2f386cc838d0b0f9b4b5da7fe611e1113b70c
in this bug https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1907081

This is the code that is responsible for the upgrade
https://github.com/ovn-org/ovn/blob/5c2d311b8b7b4d5c3a619de72be6a433aa4c44db/utilities/ovn-ctl#L293
```
if test X$detach = Xno && test $mode = cluster && test -z "$cluster_remote_addr" ; then
```
First two checks are true.
Third check means upgrade_cluster (L313) will run on leader node, i.e --db-nb-cluster-remote-addr= and --db-sb-cluster-remote-addr= is not set.

4. During rolling upgrades of controller nodes, the leader will likely move.
OVS_CTL_OPTS generated from this template
https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/2023.1/templates/ovn-northd-opts.j2
but only when new node joins the cluster https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/2023.1/tasks/providers/ovn_cluster_setup.yml#L105
so after upgrade all 3 nodes will have --db-nb-cluster-remote-addr and --db-sb-cluster-remote-addr set to the some value that was leader node at that time.

What we need is --db-nb-cluster-remote-addr= not set on leader node, so during ovn-central restart on leader node upgrade_cluster will run.