N -> O upgrade: instances live migration gets stuck before starting compute upgrade

Bug #1675739 reported by Sofer Athlan-Guyot
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Sofer Athlan-Guyot

Bug Description

Originally reported there https://bugzilla.redhat.com/show_bug.cgi?id=1435609

1. Deploy newton with 3 controllers, 2 compute nodes, 3 ceph storage nodes
2. Launch Nova instances on overcloud
3. Start ocata upgrade procedure
4. Before upgrading 1st compute node live migrate all instances that are hosted by the node

Instances gets stuck with MIGRATING status:

+--------------------------------------+-------------------------------------------------------+-----------+------------+-------------+---------------------------------------------------------------------------
-------------------------+ │
| ID | Name | Status | Task State | Power State | Networks │
                         | │
+--------------------------------------+-------------------------------------------------------+-----------+------------+-------------+---------------------------------------------------------------------------
-------------------------+ │
| 65a9bb75-b90b-4e36-b175-941403a02aab | st--ba-instance-hdad4xr725fv-my_instance-a5ggs37nh4bu | MIGRATING | migrating | Running | stack-ba-tenant_net_ext_tagged-b6ttm4vqbjes-private_network-4h5khjqmzhma=1│

It looks that we are doing the nova placement configuration on the
compute nodes before the compute nodes have been upgraded:
https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/major_upgrade_steps.j2.yaml#L60-L68

Changed in tripleo:
assignee: nobody → Sofer Athlan-Guyot (sofer-athlan-guyot)
status: New → In Progress
Changed in tripleo:
milestone: ongoing → pike-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/450142

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/449572
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bfd485406d8f3847b1969579ebbdaa912c592a4a
Submitter: Jenkins
Branch: master

commit bfd485406d8f3847b1969579ebbdaa912c592a4a
Author: Sofer Athlan-Guyot <email address hidden>
Date: Fri Mar 24 12:11:33 2017 +0100

    N->O Upgrade, make sure all nova placement parameter properly set.

    The restart of openstack-nova-compute takes place before crudini set
    the password, user_domain and project_name get set.

    Change-Id: I57b54d5f59d5803d7ad4e399d598f699785a5825
    Closes-Bug: #1675739
    Co-Authored-By: Oliver Walsh <email address hidden>

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/ocata)

Reviewed: https://review.openstack.org/450142
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c077b204a57bb98cdc4066f3cbd608b23313a420
Submitter: Jenkins
Branch: stable/ocata

commit c077b204a57bb98cdc4066f3cbd608b23313a420
Author: Sofer Athlan-Guyot <email address hidden>
Date: Fri Mar 24 12:11:33 2017 +0100

    N->O Upgrade, make sure all nova placement parameter properly set.

    The restart of openstack-nova-compute takes place before crudini set
    the password, user_domain and project_name get set.

    Change-Id: I57b54d5f59d5803d7ad4e399d598f699785a5825
    Closes-Bug: #1675739
    Co-Authored-By: Oliver Walsh <email address hidden>
    (cherry picked from commit bfd485406d8f3847b1969579ebbdaa912c592a4a)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 7.0.0.0b1

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 6.1.0

This issue was fixed in the openstack/tripleo-heat-templates 6.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.