migrate_flavor_data doesn't flavor migrate meta data of VMs spawned during upgrade

[Suresh Vinapamula]

Hi,

I am upgrading from Juno to Kilo and from that to Liberty.

I understand I need to nova-manage db migrate_flavor_data before upgrading from Kilo to Liberty to let VMs that were spawned while the system was in Juno to flavor migrate to Kilo.

Depending on the number of computes, complete upgrade can potentially be spanned for longer duration, days if not months.

While migrate_flavor_data seem to flavor migrate meta data of the VMs that were spawned before upgrade procedure, it doesn't seem to flavor migrate for the VMs that were spawned during the upgrade procedure more specifically after openstack controller upgrade and before compute upgrade. Am I missing something here or is it by intention?

Since, the compute upgrade procedure could last for days, would it be practical to block spawning work load VMs for that long duration? Otherwise, next upgrade will fail right?

thanks

>> While migrate_flavor_data seem to flavor migrate meta data of the VMs
>> that were spawned before upgrade procedure, it doesn't seem to flavor
>> migrate for the VMs that were spawned during the upgrade procedure more
>> specifically after openstack controller upgrade and before compute
>> upgrade. Am I missing something here or is it by intention?

>You can run the flavor migration as often as you need, and can certainly
>run it after your last compute is upgraded before you start to move into
>liberty.
>
>--Dan

Thanks Dan for your response. While I do run that before I start my move to liberty, what I see is that it doesn't seem to flavor migrate meta data for the VMs that are spawned after controller upgrade from juno to kilo and before all computes upgraded from juno to kilo. The current work around is to delete those VMs that are spawned after controller upgrade and before all computes upgrade, and then initiate liberty upgrade. Then it works fine.

Dan Smith <email address hidden>

Aug 31

to me, openstack-dev
> Thanks Dan for your response. While I do run that before I start my
> move to liberty, what I see is that it doesn't seem to flavor migrate
> meta data for the VMs that are spawned after controller upgrade from
> juno to kilo and before all computes upgraded from juno to kilo. The
> current work around is to delete those VMs that are spawned after
> controller upgrade and before all computes upgrade, and then initiate
> liberty upgrade. Then it works fine.

I can't think of any reason why that would be, or why it would be a
problem. Instances created after the controllers are upgraded should not
have old-style flavor info, so they need not be touched by the migration
code.

Maybe filing a bug is in order describing what you see?

Hi,

I did some investigation last week, why this was happening before I file the bug. Here are my observations and I would like to work on it. Any guidance is appreciated.

Upgrade procedure:
In my setup I have openstack controller on one node, neutron on another node and rest of the nodes are computes. Assume they are running juno. As the first step,
a) Bringup up a new openstack node in kilo
b) Migrate database from juno to kilo.
c) Run xxx manage db sync commands.
d) Set upgrade levels to juno
e) Migrate neutron and computes to the newer controller.

second step.
a) Upgrade neutron, and then start compute upgrades one at a time.

third step,
a) Finalize the upgrade by clearing upgrade levels and restarting services.
b) Issue migrate flavor data command.

This procedure is openstack side by side upgraded and neutron and computes inplace upgraded.

Issue:
VMs spawned after step1 and during step2 are not flavor migrated. So, migration to liberty was failing.

Investigation:
flavor migrate command filters instance uuids in nova.instance_extra table whose 'flavor' column is NULL and fill in with the necessary flavor information and delete those respective rows from the nova.instance_system_metadata associated with that VM.
For the VMs spawned after the first step and before step2, this flavor information in nova.instance_extra and also respective flavor information in nova.instance_system_metadata table.
Since the filter looks for 'flavor ' column as NULL, these instances are not caught to delete entries in nova.instance_system_metadata. And presence of this data is failing nova db sync while migrating to liberty.

Possible Solutions:
I feel we should broaden the filter so that instances spawned after step1 and during step 2 are also accounted for data translation. As a quick verification, I tried this by having no filer and it worked. I know filter give efficiency, but would it matter in this context or should we broaden the filter?

Please let me know if I am going in the right direction.

thanks

[Dan Smith (danms)]

This is for some really old code at this point, but as I said, I'm not sure how this could happen. This bug really needs more detail like what fails exactly, as well as maybe some dumps of the affected instances from the database, etc.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]