nova-manage db online_data_migrations hangs on instances with no host set

Bug #1788115 reported by Jiri Suchomel on 2018-08-21
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Jiri Suchomel
Pike
High
Matt Riedemann
Queens
High
Matt Riedemann
Rocky
High
Matt Riedemann

Bug Description

When there are some deleted instances present before upgrading,
"nova-manage db online_data_migrations" will not be able to finish.

I think this is because populate_missing_availability_zones does not check for deleted instances (or for non-existent AZ) - so it is run over and over again because it cannot set AZ to a deleted instance.

Fix proposed to branch: master
Review: https://review.openstack.org/594050

Changed in nova:
assignee: nobody → Jiri Suchomel (jsuchome)
status: New → In Progress
tags: added: nova-manage
Changed in nova:
importance: Undecided → Medium
Changed in nova:
importance: Medium → Undecided

While it is true that populate_missing_availability_zone does not check for deleted instances; I don't see how it can go into infinite loop unless there is no CONF.default_availability_zone set (https://github.com/openstack/nova/blob/722d5b477219f0a2435a9f4ad4d54c61b83219f1/nova/availability_zones.py#L99) which is the last resort; meaning CONF.default_availability_zone is set to None, which by default is "nova".

Probably its the situation where the instance was deleted before being scheduled to a compute node in which case it could hit https://github.com/openstack/nova/blob/722d5b477219f0a2435a9f4ad4d54c61b83219f1/nova/availability_zones.py#L168. Could you confirm if it is indeed this i.e instance.host was NULL?

Jiri Suchomel (jsuchome) wrote :

That's quite possible (deleted before being scheduling). I'm gonna test it

Jiri Suchomel (jsuchome) wrote :

Indeed that's the case - instance.host is NULL and get_instance_availability_zone returned NULL AZ.

Matt Riedemann (mriedem) on 2018-08-21
summary: - nova-manage db online_data_migrations hangs with deleted instances
+ nova-manage db online_data_migrations hangs with instances with no host
+ set
summary: - nova-manage db online_data_migrations hangs with instances with no host
+ nova-manage db online_data_migrations hangs on instances with no host
set
Changed in nova:
importance: Undecided → Medium
tags: added: rocky-rc-potential
Changed in nova:
importance: Medium → High
Changed in nova:
assignee: Jiri Suchomel (jsuchome) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem) on 2018-08-21
Changed in nova:
assignee: Matt Riedemann (mriedem) → Jiri Suchomel (jsuchome)

Reviewed: https://review.openstack.org/594050
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=690f91b5c7f7e84a4e6d351b27c05818d947cce1
Submitter: Zuul
Branch: master

commit 690f91b5c7f7e84a4e6d351b27c05818d947cce1
Author: Jiří Suchomel <email address hidden>
Date: Tue Aug 21 09:10:07 2018 +0200

    Filter out instances without a host when populating AZ

    It could happen that instance does not have a host set, e.g.
    when its creation failed before it was scheduled.
    During online_migration, populate_missing_availability_zones tries to
    add missing AZs to all instances. However for instances without a host
    there's no reasonable value for AZ (we can't use a logic that bases
    the value on a host) so let's skip this kind of instances completely.

    Change-Id: Ic6060beaa08af5ea70e5e54fffb94eea58aa7bbf
    Closes-Bug: #1788115

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/594178
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ad14e428f82f02d71e0b33ec6d20e7810a978e3b
Submitter: Zuul
Branch: stable/rocky

commit ad14e428f82f02d71e0b33ec6d20e7810a978e3b
Author: Jiří Suchomel <email address hidden>
Date: Tue Aug 21 09:10:07 2018 +0200

    Filter out instances without a host when populating AZ

    It could happen that instance does not have a host set, e.g.
    when its creation failed before it was scheduled.
    During online_migration, populate_missing_availability_zones tries to
    add missing AZs to all instances. However for instances without a host
    there's no reasonable value for AZ (we can't use a logic that bases
    the value on a host) so let's skip this kind of instances completely.

    Change-Id: Ic6060beaa08af5ea70e5e54fffb94eea58aa7bbf
    Closes-Bug: #1788115
    (cherry picked from commit 690f91b5c7f7e84a4e6d351b27c05818d947cce1)

This issue was fixed in the openstack/nova 18.0.0.0rc3 release candidate.

Reviewed: https://review.openstack.org/594185
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=059c2d4a8a9967403755c40beda667c9463848a8
Submitter: Zuul
Branch: stable/queens

commit 059c2d4a8a9967403755c40beda667c9463848a8
Author: Jiří Suchomel <email address hidden>
Date: Tue Aug 21 09:10:07 2018 +0200

    Filter out instances without a host when populating AZ

    It could happen that instance does not have a host set, e.g.
    when its creation failed before it was scheduled.
    During online_migration, populate_missing_availability_zones tries to
    add missing AZs to all instances. However for instances without a host
    there's no reasonable value for AZ (we can't use a logic that bases
    the value on a host) so let's skip this kind of instances completely.

    Change-Id: Ic6060beaa08af5ea70e5e54fffb94eea58aa7bbf
    Closes-Bug: #1788115
    (cherry picked from commit 690f91b5c7f7e84a4e6d351b27c05818d947cce1)
    (cherry picked from commit ad14e428f82f02d71e0b33ec6d20e7810a978e3b)

Reviewed: https://review.openstack.org/594184
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=62464883e1c2ba98e1fddb1284171ca5bef4d8e1
Submitter: Zuul
Branch: stable/pike

commit 62464883e1c2ba98e1fddb1284171ca5bef4d8e1
Author: Jiří Suchomel <email address hidden>
Date: Tue Aug 21 09:10:07 2018 +0200

    Filter out instances without a host when populating AZ

    It could happen that instance does not have a host set, e.g.
    when its creation failed before it was scheduled.
    During online_migration, populate_missing_availability_zones tries to
    add missing AZs to all instances. However for instances without a host
    there's no reasonable value for AZ (we can't use a logic that bases
    the value on a host) so let's skip this kind of instances completely.

    Change-Id: Ic6060beaa08af5ea70e5e54fffb94eea58aa7bbf
    Closes-Bug: #1788115
    (cherry picked from commit 690f91b5c7f7e84a4e6d351b27c05818d947cce1)
    (cherry picked from commit ad14e428f82f02d71e0b33ec6d20e7810a978e3b)
    (cherry picked from commit 059c2d4a8a9967403755c40beda667c9463848a8)

This issue was fixed in the openstack/nova 16.1.5 release.

This issue was fixed in the openstack/nova 17.0.6 release.

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers