N -> O upgrade: instances live migration gets stuck before starting compute upgrade

Bug #1675739 reported by Sofer Athlan-Guyot on 2017-03-24
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Sofer Athlan-Guyot

Bug Description

Originally reported there https://bugzilla.redhat.com/show_bug.cgi?id=1435609

1. Deploy newton with 3 controllers, 2 compute nodes, 3 ceph storage nodes
2. Launch Nova instances on overcloud
3. Start ocata upgrade procedure
4. Before upgrading 1st compute node live migrate all instances that are hosted by the node

Instances gets stuck with MIGRATING status:

+--------------------------------------+-------------------------------------------------------+-----------+------------+-------------+---------------------------------------------------------------------------
-------------------------+ │
| ID | Name | Status | Task State | Power State | Networks │
                         | │
+--------------------------------------+-------------------------------------------------------+-----------+------------+-------------+---------------------------------------------------------------------------
-------------------------+ │
| 65a9bb75-b90b-4e36-b175-941403a02aab | st--ba-instance-hdad4xr725fv-my_instance-a5ggs37nh4bu | MIGRATING | migrating | Running | stack-ba-tenant_net_ext_tagged-b6ttm4vqbjes-private_network-4h5khjqmzhma=1│

It looks that we are doing the nova placement configuration on the
compute nodes before the compute nodes have been upgraded:
https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/major_upgrade_steps.j2.yaml#L60-L68

Changed in tripleo:
assignee: nobody → Sofer Athlan-Guyot (sofer-athlan-guyot)
status: New → In Progress
Changed in tripleo:
milestone: ongoing → pike-1

Reviewed: https://review.openstack.org/449572
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bfd485406d8f3847b1969579ebbdaa912c592a4a
Submitter: Jenkins
Branch: master

commit bfd485406d8f3847b1969579ebbdaa912c592a4a
Author: Sofer Athlan-Guyot <email address hidden>
Date: Fri Mar 24 12:11:33 2017 +0100

    N->O Upgrade, make sure all nova placement parameter properly set.

    The restart of openstack-nova-compute takes place before crudini set
    the password, user_domain and project_name get set.

    Change-Id: I57b54d5f59d5803d7ad4e399d598f699785a5825
    Closes-Bug: #1675739
    Co-Authored-By: Oliver Walsh <email address hidden>

Changed in tripleo:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/450142
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c077b204a57bb98cdc4066f3cbd608b23313a420
Submitter: Jenkins
Branch: stable/ocata

commit c077b204a57bb98cdc4066f3cbd608b23313a420
Author: Sofer Athlan-Guyot <email address hidden>
Date: Fri Mar 24 12:11:33 2017 +0100

    N->O Upgrade, make sure all nova placement parameter properly set.

    The restart of openstack-nova-compute takes place before crudini set
    the password, user_domain and project_name get set.

    Change-Id: I57b54d5f59d5803d7ad4e399d598f699785a5825
    Closes-Bug: #1675739
    Co-Authored-By: Oliver Walsh <email address hidden>
    (cherry picked from commit bfd485406d8f3847b1969579ebbdaa912c592a4a)

tags: added: in-stable-ocata

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b1 development milestone.

This issue was fixed in the openstack/tripleo-heat-templates 6.1.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers