nova-manage db online_data_migrations hangs on instances with no host set

[Jiri Suchomel]

When there are some deleted instances present before upgrading,
"nova-manage db online_data_migrations" will not be able to finish.

I think this is because populate_missing_availability_zones does not check for deleted instances (or for non-existent AZ) - so it is run over and over again because it cannot set AZ to a deleted instance.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/594050

[Surya Seetharaman (tssurya)]

While it is true that populate_missing_availability_zone does not check for deleted instances; I don't see how it can go into infinite loop unless there is no CONF.default_availability_zone set (https://github.com/openstack/nova/blob/722d5b477219f0a2435a9f4ad4d54c61b83219f1/nova/availability_zones.py#L99) which is the last resort; meaning CONF.default_availability_zone is set to None, which by default is "nova".

Probably its the situation where the instance was deleted before being scheduled to a compute node in which case it could hit https://github.com/openstack/nova/blob/722d5b477219f0a2435a9f4ad4d54c61b83219f1/nova/availability_zones.py#L168. Could you confirm if it is indeed this i.e instance.host was NULL?

[Jiri Suchomel (jsuchome)]

That's quite possible (deleted before being scheduling). I'm gonna test it

[Jiri Suchomel (jsuchome)]

Indeed that's the case - instance.host is NULL and get_instance_availability_zone returned NULL AZ.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/594178

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/594184

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/594185

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/594050
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=690f91b5c7f7e84a4e6d351b27c05818d947cce1
Submitter: Zuul
Branch: master

commit 690f91b5c7f7e84a4e6d351b27c05818d947cce1
Author: Jiří Suchomel <email address hidden>
Date: Tue Aug 21 09:10:07 2018 +0200

    Filter out instances without a host when populating AZ

    It could happen that instance does not have a host set, e.g.
    when its creation failed before it was scheduled.
    During online_migration, populate_missing_availability_zones tries to
    add missing AZs to all instances. However for instances without a host
    there's no reasonable value for AZ (we can't use a logic that bases
    the value on a host) so let's skip this kind of instances completely.

    Change-Id: Ic6060beaa08af5ea70e5e54fffb94eea58aa7bbf
    Closes-Bug: #1788115

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/594178
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ad14e428f82f02d71e0b33ec6d20e7810a978e3b
Submitter: Zuul
Branch: stable/rocky

commit ad14e428f82f02d71e0b33ec6d20e7810a978e3b
Author: Jiří Suchomel <email address hidden>
Date: Tue Aug 21 09:10:07 2018 +0200

    Filter out instances without a host when populating AZ

    It could happen that instance does not have a host set, e.g.
    when its creation failed before it was scheduled.
    During online_migration, populate_missing_availability_zones tries to
    add missing AZs to all instances. However for instances without a host
    there's no reasonable value for AZ (we can't use a logic that bases
    the value on a host) so let's skip this kind of instances completely.

    Change-Id: Ic6060beaa08af5ea70e5e54fffb94eea58aa7bbf
    Closes-Bug: #1788115
    (cherry picked from commit 690f91b5c7f7e84a4e6d351b27c05818d947cce1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.0.0rc3 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/594185
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=059c2d4a8a9967403755c40beda667c9463848a8
Submitter: Zuul
Branch: stable/queens

commit 059c2d4a8a9967403755c40beda667c9463848a8
Author: Jiří Suchomel <email address hidden>
Date: Tue Aug 21 09:10:07 2018 +0200

    Filter out instances without a host when populating AZ

    It could happen that instance does not have a host set, e.g.
    when its creation failed before it was scheduled.
    During online_migration, populate_missing_availability_zones tries to
    add missing AZs to all instances. However for instances without a host
    there's no reasonable value for AZ (we can't use a logic that bases
    the value on a host) so let's skip this kind of instances completely.

    Change-Id: Ic6060beaa08af5ea70e5e54fffb94eea58aa7bbf
    Closes-Bug: #1788115
    (cherry picked from commit 690f91b5c7f7e84a4e6d351b27c05818d947cce1)
    (cherry picked from commit ad14e428f82f02d71e0b33ec6d20e7810a978e3b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/594184
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=62464883e1c2ba98e1fddb1284171ca5bef4d8e1
Submitter: Zuul
Branch: stable/pike

commit 62464883e1c2ba98e1fddb1284171ca5bef4d8e1
Author: Jiří Suchomel <email address hidden>
Date: Tue Aug 21 09:10:07 2018 +0200

    Filter out instances without a host when populating AZ

    It could happen that instance does not have a host set, e.g.
    when its creation failed before it was scheduled.
    During online_migration, populate_missing_availability_zones tries to
    add missing AZs to all instances. However for instances without a host
    there's no reasonable value for AZ (we can't use a logic that bases
    the value on a host) so let's skip this kind of instances completely.

    Change-Id: Ic6060beaa08af5ea70e5e54fffb94eea58aa7bbf
    Closes-Bug: #1788115
    (cherry picked from commit 690f91b5c7f7e84a4e6d351b27c05818d947cce1)
    (cherry picked from commit ad14e428f82f02d71e0b33ec6d20e7810a978e3b)
    (cherry picked from commit 059c2d4a8a9967403755c40beda667c9463848a8)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.