Rocky -> Stein overcloud upgrade prepare: Unable to switch from openvswitch to ovn neutron mechanism drivers on upgrade

[Anton Antonov]

Description:

  I'm trying to upgrade RDO from Rocky to Stein following this documentation:
https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/post_deployment/upgrade/major_upgrade.html

  But, the "openstack overcloud upgrade prepare" step fails with "Unable to switch from openvswitch to ovn neutron mechanism drivers on upgrade. Please consult the documentation." error.

  I haven't wanted to switch from openswitch to OVN and I have no idea what documentation I should consult accordingly to the error message.

Step to reproduce:

  I'm running "openstack overcloud update prepare" with the same parameters I used for "openstack overcloud deploy" command when my Rocky RDO was deployed and with the new container image parameter (containers-prepare-parameter.yaml) and Yum repository switch parameter (init-repo.yaml):

```
openstack overcloud update prepare --templates \
-e ~/vxrdo/templates/node-info.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
-n ~/vxrdo/templates/network_data.yaml \
-e ~/vxrdo/templates/network-isolation.yaml \
-e ~/vxrdo/templates/scheduler_hints_env.yaml \
-e ~/vxrdo/templates/network-environment.yaml \
-e ~/vxrdo/templates/ips-from-pool-all.yaml \
-e ~/vxrdo/templates/ceph-ansible.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
-e ~/vxrdo/templates/firstboot/firstboot.yaml \
--ntp-server 10.35.10.2 \
-e ~/vxrdo/templates/init-repo.yaml \
-e ~/vxrdo/templates/containers-prepare-parameter.yaml \
 2>&1 | tee prepare.log
```

Expected results:

 Upgrade prepare operation has successfully completed

Actual results:

```
2019-09-06 16:45:59.099 80663 INFO osc_lib.shell [-] command: overcloud update prepare -> tripleoclient.v1.overcloud_update.UpdatePrepare (auth=True)
2019-09-06 16:45:59.100 80663 INFO osc_lib.clientmanager [-] Using auth plugin: password
2019-09-06 16:45:59.101 80663 DEBUG osc_lib.clientmanager [-] Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'https://10.35.5.2:13000', 'password': '***', 'project_domain_name': 'Default'} setup_auth /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:157
2019-09-06 16:45:59.105 80663 DEBUG osc_lib.clientmanager [-] Get auth_ref auth_ref /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:201
2019-09-06 16:46:03.480 80663 INFO tripleoclient.v1.overcloud_update.MinorUpdatePrepare [-] Stack found, will be doing a stack update
Removing the current plan files
Uploading new plan files
Temporary Swift GET/PUT URL parameters have successfully been updated.
Temporary Swift GET/PUT URL parameters have successfully been updated.
The backup of the ceph-ansible fetch directory did not need to be renamed
Plan updated.
Processing templates in the directory /tmp/tripleoclient-1K3nE7/tripleo-heat-templates
2019-09-06 16:49:09.533 80663 ERROR openstack [-] Unable to switch from openvswitch to ovn neutron mechanism drivers on upgrade. Please consult the documentation.: CommandError: Unable to switch from openvswitch to ovn neutron mechanism drivers on upgrade. Please consult the documentation.
2019-09-06 16:49:09.544 80663 INFO osc_lib.shell [-] END return value: 1
```

Environment:

```
[stack@undercloud ~]$ rpm -qa|grep tripleo
openstack-tripleo-puppet-elements-10.3.2-0.20190820214417.5453b89.el7.noarch
python2-tripleo-repos-0.0.1-0.20190724014728.1cf6e0b.el7.noarch
ansible-tripleo-ipsec-9.1.1-0.20190513182453.ffe104c.el7.noarch
python2-tripleo-common-10.8.1-0.20190831025359.300785c.el7.noarch
ansible-role-tripleo-modify-image-1.1.1-0.20190711162806.eabaed0.el7.noarch
openstack-tripleo-common-containers-10.8.1-0.20190831025359.300785c.el7.noarch
python2-tripleoclient-heat-installer-11.5.1-0.20190903220813.85188eb.el7.noarch
puppet-tripleo-10.5.1-0.20190826210937.c689405.el7.noarch
openstack-tripleo-image-elements-10.4.2-0.20190829202743.1ebd7af.el7.noarch
openstack-tripleo-validations-10.5.1-0.20190830050759.585c119.el7.noarch
openstack-tripleo-common-10.8.1-0.20190831025359.300785c.el7.noarch
python2-tripleoclient-11.5.1-0.20190903220813.85188eb.el7.noarch
openstack-tripleo-heat-templates-10.6.1-0.20190905013450.b33b839.el7.noarch
```

[Alex Schultz (alex-schultz)]

So you would need to specify the non-ovn environment files for the upgrade. Currently ml2 -> ovn requires some sort of migration which is not performed automatically. This error pops up because the default for Stein is OVN and you did not specify the ml2 environment file It looks like it's THT/updates/update-from-ml2-ovs-from-rocky.yaml

[Anton Antonov (anta-nok)]

  Thank you Alex for pointing out the updates file. It's not documented anywhere. It's not even include into readme.md in THT/updates/ directory.

  Before seeing your comment I had a look through tripleo code and added

parameter_defaults:
  NeutronMechanismDrivers: ['openvswitch']

  into my network-environment.yaml template.

  But, either with my change or with the file you suggested I now have the issue described in
https://bugs.launchpad.net/tripleo/+bug/1819672 when I run "openstack overcloud update prepare"

  Is there a proper fix/workaround? I can't do "openstack overcloud plan delete overcloud" as I already have overcloud stack deployed and I have no intentions to destroy it.

[Jose Luis Franco (jfrancoa)]

Hello Anton,

Thanks for the bugzilla, Alex answer is right, if you have an ml2-ovs environment and you want to perform the upgrade you need to pass the -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml environment file in the prepare step. However, I'm sorry to say that the upstream workflow for upgrading from Rocky to Stein is not supported. Due to the complications and divergences between the upstream and dowstream upgrades procedures (OSP14 to OSP15 implies a Operating System major upgrade from RHEL7 to RHEL8 which complicated things) the upgrade upstream from Rocky to Stein is not supported. We need to add a note into upstream documentation to avoid this type of issues again.
However, if you would like to test the upgrade from OSP14 to OSP15 you can follow the steps documented in https://gitlab.cee.redhat.com/osp15/osp-upgrade-el8/blob/master/README.md.

[Jose Luis Franco (jfrancoa)]

The link shared ni the last comment is for Red Hat only access, sorry for the confusion. We'll update the upstream docs accordingly.

[Jiří Stránský (jistr)]

While we wait for CentOS 8, there is a chance that one could upgrade from Rocky to Stein while staying on CentOS 7. The CI tests Stein *deployment* on CentOS 7 so the upgrade could be made to work somehow. However, due to CI run time limitations we don't have a full CI upgrade job from R to S, and i don't know anyone who tested a full extent of such upgrade. So this would be a task up for grabs i think, opportunity to contribute to the docs.

[Anton Antonov (anta-nok)]

Hello Jose,

  Thank you for you comment. You mentioned THT/environments/services/neutron-ovs.yaml file and ALex mentioned THT/updates/update-from-ml2-ovs-from-rocky.yaml. Existence of the latter means someone worked on upgrade from Rocky and probably tested it :)

  I've upgraded my undercloud to Stein and it stayed on Centos7, so I expected that overcloud nodes can stay on Centos7 as well.
[stack@undercloud vxrdo]$ cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)

  The problem is that I cannot continue testing the upgrade because of the overcloud plan "KeyError: 'passwords'" issue similar to https://bugs.launchpad.net/tripleo/+bug/1819672
  My problem is that I get this issue on the "openstack overcloud update prepare" step and I can't follow RedHat recommendations from here: https://access.redhat.com/solutions/3714651

  Do you know if my overcloud plan can be fixed somehow or point my to the right direction?

  Thank you.