Kilo to Liberty: migrate flavor data impossible

[axel vanzaghi]

NOTE: (in comment #9) there is a proposed solution to fix this about dropping foreign keys. This will result in a broken data model that will prevent upgrade to Newton. Please use the solution in comment #8.

Hi everyone,

I'm currently upgrading my openstack installation from kilo to liberty on Ubuntu 14.04 LTS.

I have upgraded all packets fine and tried to update database:
nova-manage db sync

which fails complaining about the flavor not been migrated and so that I should run nova-manage db migrate_flavor_data before.

I runned this command but then nova-manage complains that migrate_data_flavor is not a valid command.

Is this a real bug ? Am I missing something ?

For the upgrade, I removed the kilo sources definition in /etc/apt/sources.list.d and then I add the apt repository using the command provided in the install documentation, after what I did an update, upgrade and dist-upgrade, forcing dpkg to keep old config.

If you need more informations, feel free to ask.

I'll post more detailed informations by tomorrow.

thanks in advance for your responses and sorry for not following the guidelines.

[axel vanzaghi (axellinkgm)]

Hi,

some informations :

# dpkg -l | grep nova
ii nova-api 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - API frontend
ii nova-cert 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - certificate management
ii nova-common 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-conductor 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - conductor service
ii nova-consoleauth 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - Console Authenticator
ii nova-scheduler 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:2.30.1-1~cloud0 all client library for OpenStack Compute API

# nova-mange db sync
2015-10-30 09:06:46.364 4920 INFO migrate.versioning.api [-] 290 -> 291...
Command failed, please check log for more info
2015-10-30 09:06:46.585 4920 CRITICAL nova [-] ValidationError: There are still 2 unmigrated flavor records. Migration cannot continue until all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.
2015-10-30 09:06:46.585 4920 ERROR nova Traceback (most recent call last):
2015-10-30 09:06:46.585 4920 ERROR nova File "/usr/bin/nova-manage", line 10, in <module>
2015-10-30 09:06:46.585 4920 ERROR nova sys.exit(main())
2015-10-30 09:06:46.585 4920 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1443, in main
2015-10-30 09:06:46.585 4920 ERROR nova ret = fn(*fn_args, **fn_kwargs)
2015-10-30 09:06:46.585 4920 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 910, in sync
2015-10-30 09:06:46.585 4920 ERROR nova return migration.db_sync(version)
2015-10-30 09:06:46.585 4920 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 26, in db_sync
2015-10-30 09:06:46.585 4920 ERROR nova return IMPL.db_sync(version=version, database=database)
2015-10-30 09:06:46.585 4920 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 106, in db_sync
2015-10-30 09:06:46.585 4920 ERROR nova version)
2015-10-30 09:06:46.585 4920 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
2015-10-30 09:06:46.585 4920 ERROR nova return _migrate(url, repository, version, upgrade=True, err=err, **opts)
2015-10-30 09:06:46.585 4920 ERROR nova File "<string>", line 2, in _migrate
2015-10-30 09:06:46.585 4920 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
2015-10-30 09:06:46.585 4920 ERROR nova return f(*a, **kw)
2015-10-30 09:06:46.585 4920 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 366, in _migrate
2015-10-30 09:06:46.585 4920 ERROR nova sc...

[Davanum Srinivas (DIMS) (dims-v)]

Axel,

reading the following:
http://openstack.markmail.org/search/?q=migrate_flavor_data+order%3Adate-backward
https://review.openstack.org/#/c/196911/
https://review.openstack.org/#/c/198060/

I believe, you should first get the controller to latest stable/kilo, run "nova db migrate_flavor_data" and THEN upgrade the code to Liberty.

[axel vanzaghi (axellinkgm)]

I did it, before going to upgrade I runned the migrate_flavor_data and made sure there was no instance left to upgrade. But as you can see in the error, there is still two instances that have to be migrated after update.

The solution for that was to delete every instances with the deleted state after the migrate_flavor_update and before going to liberty. Looks like two of the deleted VMs were not ignored by the db sync or completely ignored by migrate_flavor_data, I don't know how it is supposed to work.

[Florian Ermisch (0xf10e)]

I see the same error:

          ID: nova-manage db sync
    Function: cmd.run
        Name: nova-manage db sync; sleep 15
      Result: True
     Comment: Command "nova-manage db sync; sleep 15" run
     Started: 18:07:09.511897
    Duration: 17747.739 ms
     Changes:
              ----------
              pid:
                  18000
              retcode:
                  0
              stderr:
                  No handlers could be found for logger "oslo_config.cfg"
                  2016-01-26 18:07:12.092 18001 DEBUG migrate.versioning.repository [-] Loading repository /usr/lib/python2.7/dist-packages/nova/db/sql
alchemy/migrate_repo... __init__ /usr/lib/python2.7/dist-packages/migrate/versioning/repository.py:76
                  2016-01-26 18:07:12.093 18001 DEBUG migrate.versioning.script.base [-] Loading script /usr/lib/python2.7/dist-packages/nova/db/sqlalc
hemy/migrate_repo/versions/216_havana.py... __init__ /usr/lib/python2.7/dist-packages/migrate/versioning/script/base.py:27
[...]
[...]
[...]
                  2016-01-26 18:07:12.157 18001 INFO migrate.versioning.api [-] 290 -> 291...
                  2016-01-26 18:07:12.167 18001 CRITICAL nova [-] ValidationError: There are still 3 unmigrated flavor records. Migration cannot conti$
ue until all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.
                  2016-01-26 18:07:12.167 18001 ERROR nova Traceback (most recent call last):
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/bin/nova-manage", line 10, in <module>
                  2016-01-26 18:07:12.167 18001 ERROR nova sys.exit(main())
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1443, in main
                  2016-01-26 18:07:12.167 18001 ERROR nova ret = fn(*fn_args, **fn_kwargs)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 910, in sync
                  2016-01-26 18:07:12.167 18001 ERROR nova return migration.db_sync(version)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 26, in db_sync
                  2016-01-26 18:07:12.167 18001 ERROR nova return IMPL.db_sync(version=version, database=database)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 106, in db_$
ync
                  2016-01-26 18:07:12.167 18001 ERROR nova version)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
                  2016-01-26 18:07:12.167 18001 ERROR nova return _migrate(url, repository, version, upgrade=True, err=err, **opts)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "<string>", line 2, in _migrate
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 160, in
with_engine
  ...

[Florian Ermisch (0xf10e)]

By the way: There is no `nova-manage db migrate_flavor_data`:

root@controller:~# nova-manage db migrate_flavor_data
No handlers could be found for logger "oslo_config.cfg"
usage: nova-manage db [-h]
                      {archive_deleted_rows,contract,expand,migrate,null_instance_uuid_scan,sync,version}
                      ...
nova-manage db: error: argument action: invalid choice: 'migrate_flavor_data' (choose from 'archive_deleted_rows', 'contract', 'expand', 'migrate', 'null_instance_uuid_scan', 'sync', 'version')
root@controller:~#

So I tried :
root@controller:~# nova-manage db migrate
[...]
2016-01-26 19:03:43.055 19522 ERROR nova DatabaseMigrationError: Database migration failed: expand phase still has operations that need to be executed
2016-01-26 19:03:43.055 19522 ERROR nova
root@controller:~# echo $?
1

Sooo:
root@controller:~# nova-manage db expand
[...]
2016-01-26 19:05:16.959 19572 ERROR nova DatabaseMigrationError: Database migration failed: Cannot run 'db sync' until 'db contract' is run
2016-01-26 19:05:16.959 19572 ERROR nova
Command failed, please check log for more info
root@controller:~#

BUT:
root@controller:~# nova-manage db contract
No handlers could be found for logger "oslo_config.cfg"
The "contract" command is experimental and potentially dangerous. As such, it is disabled by default. Enable using "--force-experimental".

root@controller:~# nova-manage db contract --force-experimental
[...]
2016-01-26 19:06:14.357 19618 INFO alembic.runtime.migration [-] Context impl MySQLImpl.
2016-01-26 19:06:14.357 19618 INFO alembic.runtime.migration [-] Will assume non-transactional DDL.
Command failed, please check log for more info
2016-01-26 19:06:15.449 19618 CRITICAL nova [-] TypeError: expected string or buffer
2016-01-26 19:06:15.449 19618 ERROR nova Traceback (most recent call last):
2016-01-26 19:06:15.449 19618 ERROR nova File "/usr/bin/nova-manage", line 10, in <module>
2016-01-26 19:06:15.449 19618 ERROR nova sys.exit(main())
2016-01-26 19:06:15.449 19618 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1443, in main
2016-01-26 19:06:15.449 19618 ERROR nova ret = fn(*fn_args, **fn_kwargs)
2016-01-26 19:06:15.449 19618 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 932, in contract
2016-01-26 19:06:15.449 19618 ERROR nova return migration.db_contract(dry_run)
2016-01-26 19:06:15.449 19618 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 41, in db_contract
2016-01-26 19:06:15.449 19618 ERROR nova return IMPL.db_contract(dryrun=dryrun, database=database)
2016-01-26 19:06:15.449 19618 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 1069, in db_contract
2016-01-26 19:06:15.449 19618 ERROR nova _set_db_sync_version(repository, repository.latest)
2016-01-26 19:06:15.449 19618 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 87, in _set_db_sync_version
2016-01-26 19:06:15.449 19618 ERROR nova values(version=version).execute()
2016-01-26 19:06:15.449 19618 ERROR nova File "/usr/lib/python2.7/dist-packages/sqlalche...

[gadLinux (gad-aguilardelgado)]

This is a curious situation:

We are upgrading so server is not 'bootable' but we should be done:

nova db migrate_flavor_data

@Florian is not nova-manage one. I suppose he's referring to nova db command. But I'm not sure.

But the migration tool is not able to handle the migration. M

There are still 12 unmigrated flavor records. Migration cannot continue until all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.

The message is for this command:
nova-manage db migrate_flavor_data

usage: nova-manage db [-h]
                      {archive_deleted_rows,null_instance_uuid_scan,online_data_migrations,sync,version}
                      ...
nova-manage db: error: argument action: invalid choice: 'migrate_flavor_data' (choose from 'archive_deleted_rows', 'null_instance_uuid_scan', 'online_data_migrations', 'sync', 'version')

[gadLinux (gad-aguilardelgado)]

Hi I found this:
http://www.danplanet.com/blog/2015/06/26/upgrading-nova-to-kilo-with-minimal-downtime/

In fact it should be without the db but is not working either...

[gadLinux (gad-aguilardelgado)]

Hi again,

This is what I did. I cloned nova kilo from source repository.

I copied the directory nova and all its contents to the controller node machine.
I moved the original nova libs from /usr/lib/python2.7/dist-packages/nova to /usr/lib/python2.7/dist-packages/nova.mitaka
And linked /home/ubuntu/nova /usr/lib/python2.7/dist-packages/nova

then I ran the command and voilà:

root@jb4ja:/usr/lib/python2.7/dist-packages# su -s /bin/sh -c "nova-manage db migrate_flavor_data" nova
/usr/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py:572: SAWarning: Can't resolve label reference 'deleted'; converting to text() (this warning may be suppressed after 10 occurrences)
  util.ellipses_string(element.element))
/usr/lib/python2.7/dist-packages/sqlalchemy/sql/compiler.py:572: SAWarning: Can't resolve label reference 'id'; converting to text() (this warning may be suppressed after 10 occurrences)
  util.ellipses_string(element.element))
12 instances matched query, 12 completed

Then I reverted the change from nova.mitaka to nova.

It seems that sync can continue now...

[Saverio Proto (zioproto)]

The solution from Axel Vanzaghi worked also for me.

However to delete the rows I had to disable FOREIGN_KEY_CHECKS

SET FOREIGN_KEY_CHECKS=0;
delete from instances where vm_state='deleted';
SET FOREIGN_KEY_CHECKS=1;

If anyone has a more clean solution for this, please share :)

Saverio

[Mathieu Gagné (mgagne)]

For those that will find this bug and wonder why you should delete instances in 'deleted' state, I found that our issue was related to rows/instances that were manually deleted in the database by setting the deleted field to 0 instead of the instance id.

The sanity check assumes that instances are deleted if instances.deleted=instances.id and fails if it finds occurrences that doesn't match this expectation.

You can find those occurrences with this query:

SELECT instances.uuid,
       instances.id,
       instances.deleted
FROM instances,
     instance_system_metadata
WHERE instances.uuid = instance_system_metadata.instance_uuid
  AND instance_system_metadata.key = 'instance_type_id'
  AND instance_system_metadata.deleted != instance_system_metadata.id
  AND instances.deleted != instances.id
  AND instances.deleted != 0;

All you need to do is update the affected rows (only) accordingly by setting instances.deleted=instances.id

Friendly reminder, make sure you run the flavor migration (nova-manage db migrate_flavor_data) BEFORE updating to OpenStack Liberty or later as the command no longer exist in OpenStack Liberty and later, despite the error message suggesting to run it.

[Mathieu Gagné (mgagne)]

In fact, you should make sure all instances.deleted==instances.id if instances.deleted != 0, not just the ones affected by the flavor migration sanity check.

[Dan Smith (danms)]

For anyone who applied the change in comment #9 to their database, you have broken the data model severely and will in all likelihood set yourself up for bug 1684861, among others. If you haven't yet done it DO NOT DO IT.

[Sean Dague (sdague)]

Closing this out because it's a Liberty issue, which we can't fix any more.