Interactive mode of overcloud update command racing with Heat

[Jiří Stránský]

The issue has been hit on a Mitaka deployment, but it's likely that master would have the same problem.

When performing minor update in interactive mode, the command exits immediately:

[stack@instack ~]$ openstack overcloud update stack overcloud --templates -i
starting package update on stack overcloud
COMPLETE
update finished with status COMPLETE

But in fact the minor update has been triggered and there are Heat hooks waiting for clearing:

[stack@instack ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+--------------------+---------------------+---------------------+
| id | stack_name | stack_status | creation_time | updated_time |
+--------------------------------------+------------+--------------------+---------------------+---------------------+
| 7eb70f4b-3b59-4095-947c-1ca65e7874ae | overcloud | UPDATE_IN_PROGRESS | 2016-06-07T09:37:40 | 2016-06-07T12:36:51 |
+--------------------------------------+------------+--------------------+---------------------+---------------------+

[stack@instack ~]$ heat hook-poll -n3 overcloud
WARNING (shell) "heat hook-poll" is deprecated, please use "openstack stack hook poll" instead
+------------------+--------------------------------------+------------------------------------------------+-----------------+---------------------+--------------------------------------------------+
| resource_name | id | resource_status_reason | resource_status | event_time | stack_name |
+------------------+--------------------------------------+------------------------------------------------+-----------------+---------------------+--------------------------------------------------+
| UpdateDeployment | 3c165653-b25f-49bb-ad78-b863eb16bde5 | UPDATE paused until Hook pre-update is cleared | CREATE_COMPLETE | 2016-06-07T12:37:41 | overcloud-Controller-pqtwkj4my7fm-1-ntsjy2bttdko |
| UpdateDeployment | 180cbdfc-fd79-4358-a8f3-739500a006fd | UPDATE paused until Hook pre-update is cleared | CREATE_COMPLETE | 2016-06-07T12:37:32 | overcloud-Compute-ycfixlrurix2-0-hyumzgiv2udb |
| UpdateDeployment | c07af6d7-11f7-4114-a051-c71207c92765 | UPDATE paused until Hook pre-update is cleared | CREATE_COMPLETE | 2016-06-07T12:37:38 | overcloud-Controller-pqtwkj4my7fm-2-xqs5ok6yieir |
| UpdateDeployment | 44e37b83-ceb6-4802-b0ce-89447c64692f | UPDATE paused until Hook pre-update is cleared | CREATE_COMPLETE | 2016-06-07T12:37:40 | overcloud-Controller-pqtwkj4my7fm-0-bs7wv5s2gqy4 |
+------------------+--------------------------------------+------------------------------------------------+-----------------+---------------------+--------------------------------------------------+

[Jeffrey Guan (double12gzh)]

I also met the same error as what you mentioned above:

root@test-orch-vm:~# heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead

[Jiří Stránský (jistr)]

The error seems to be intermittent, after a couple of tries for a minor update CLI went to the interactive phase fine. It's likely some kind of a race condition.

[Jiří Stránský (jistr)]

Fix posted https://review.openstack.org/#/c/328260/

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/tripleo-common 5.0.0.0b2 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/tripleo-common 2.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 2.1.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/405585

[OpenStack Infra (hudson-openstack)]

Change abandoned by Joshua Hesketh (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/405585
Reason: This branch (stable/liberty) is at End Of Life