[update] Openstack cluster update failed: /usr/bin/murano-db-manage --config-file=/etc/murano/murano.conf upgrade returned 1 instead of one of [0]

[Artem Panchenko]

This issue was caught after upgrade from 5.0 to 5.1 using custom upgrade tarball with the following patches:

https://review.openstack.org/#/c/120127/
https://review.openstack.org/#/c/119105/

Fuel version info:

http://paste.openstack.org/show/109537/

Steps to reproduce:

1. Create new cluster (CentOS, HA, NeutronGRE, Ceph for Volumes, Sahara, Murano)
2. Add 3 controller+ceph and 2 compute+cinder nodes. Deploy changes
3. Upgarde master node to 5.1. Run environment update to 5.0.2

Expected result:

- cluster successfully updated

Actual:

- update failed

Update failed on first controller (node-1), here is the part of its puppet logs:

http://paste.openstack.org/show/109538/

And output of failed command (executed manually):

http://paste.openstack.org/show/109536/

[Dmitry Teselkin (teselkin-d)]

Different commands used for db-sync during installation / upgare, see http://paste.openstack.org/show/110059/

[Dmitry Teselkin (teselkin-d)]

Murano in FUEL 5.0.1 and 5.0.2 uses different database upgrade tools (sqlalchemy in 5.0.1 and alembic in 5.0.2).
Because of this there is no way to perform upgrade from 5.0.1 to 5.0.2 or downgrade from 5.0.2 to 5.0.1.
The command that performs database migration will fail in both cases.

The only workaround is to drop entire Murano database before upgrade / downgrade and re-create it again.

The user must be warned that they will lost all their Murano-related data and settings.

Steps to perform 5.0.1 -> 5.0.2 upgrade:

1. Stop active murano-api services
2. Drop database murano
3. Create empty database murano
4. Run database migrations
5. Start murano-api services

Steps to perform 5.0.2 -> 5.0.1 downgrade:

1. Stop active murano-api services
2. Drop database murano
3. Create empty database murano
4. Run database migrations
5. Start murano-api services

[Serg Melikyan (smelikyan)]

This issue should be assigned to Dmitry Ilyin

[Dmitry Ilyin (idv1985)]

I've made hook that recreates the murano database on update and will add it to the pre-update Astute hooks.

[Serg Melikyan (smelikyan)]

Dmitry Ilyin, can you provide change-request for that?

[Dmitry Ilyin (idv1985)]

I have included this to deployment tasks https://review.openstack.org/#/c/127284/5/deployment/tasks/astute/lib/migration.rb,cm
But looks like the new tasks system is not getting into 6.0 so we'll have to backport it to the Astute hooks.

[Matthew Mosesohn (raytrac3r)]

Dmitry do you have a status update? Can we get this before release?

[Dmitry Ilyin (idv1985)]

I'll try to backport this fix https://review.openstack.org/#/c/127284/5/deployment/tasks/astute/lib/migration.r to Astute's pre-update hook.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/134926

[Stanislaw Bogatkin (sbogatkin)]

We do not support such type of patching anymore. Closed.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Igor Shishkin (<email address hidden>) on branch: master
Review: https://review.openstack.org/134926
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.