overcloud upgrade hangs with heat throwing "ERROR: Timed out waiting for a reply to message ID 84a44ca3ed724eda991ba689cc364852"

[Marios Andreou]

after upgrading your undercloud and starting the overcloud upgrade by updating the overcloud stack using the major-upgrade-pacemaker-init.yaml environment file the deployment ultimately times out "ERROR: Timed out waiting for a reply to message ID 84a44ca3ed724eda991ba689cc364852"

This is originally discussed at https://bugzilla.redhat.com/show_bug.cgi?id=1315467

The fix for now seems to be restarting (undercloud) openstack-nova-api before proceeding with the overcloud upgrade (fixup incoming to the tripleoclient)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/293960

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/293960
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=a51c7202ef36bf4931b1bf752982ec33567d970d
Submitter: Jenkins
Branch: master

commit a51c7202ef36bf4931b1bf752982ec33567d970d
Author: marios <email address hidden>
Date: Thu Mar 17 13:16:24 2016 +0200

    Add a restart of openstack-nova-api after upgrading undercloud

    During upgrades testing and as discussed in bug 1558495 the
    initial overcloud upgrade step (major-upgrade-pacemaker-init.yaml)
    times out. The fix for now is to restart openstack-nova-api after
    the undercloud upgrade.

    Change-Id: Ifd9c2f4bdbca701d655c741ab21da2a571c1dcb5
    Closes-Bug: 1558495

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/296797

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/296797
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=ea4f253a1fb7eb2bcf711e57a1e7b649f15aa6eb
Submitter: Jenkins
Branch: stable/liberty

commit ea4f253a1fb7eb2bcf711e57a1e7b649f15aa6eb
Author: marios <email address hidden>
Date: Thu Mar 17 13:16:24 2016 +0200

    Add a restart of openstack-nova-api after upgrading undercloud

    During upgrades testing and as discussed in bug 1558495 the
    initial overcloud upgrade step (major-upgrade-pacemaker-init.yaml)
    times out. The fix for now is to restart openstack-nova-api after
    the undercloud upgrade.

    Change-Id: Ifd9c2f4bdbca701d655c741ab21da2a571c1dcb5
    Closes-Bug: 1558495
    (cherry picked from commit a51c7202ef36bf4931b1bf752982ec33567d970d)