Contrail 2.21-14: heat stack stuck in UPDATE_IN_PROGRESS for more than 4 hours

[Deepak Jeyaraman]

Build:
======

Contrail 2.21-14

Log location:
============

jdeepak@ttsv-shell04 /volume/vmcores/sagrawal/jdeepak_temp/NFV/contrail_bug/1539228
$ ls
total 775784
-rwxrwxrwx 1 jdeepak software 94468 Jan 28 11:39 heat-api-cfn.log
-rwxrwxrwx 1 jdeepak software 149383700 Jan 28 11:40 heat-api.log
-rwxrwxrwx 1 jdeepak software 641726037 Jan 28 11:41 heat-engine.log
-rwxrwxrwx 1 jdeepak software 54544 Jan 28 11:43 heat-manage.log

Setup:
======

3 contrail control/config/db nodes
11 compute nodes

Problem:
========

Total heat stacks - 150

Of these, one heat stack is stuck in the UPDATE_IN_PROGRESS state (the stack is updated with policy attached to the left network)

the policy is also not attached , there are no failures.

Will be uploading the logs to the above location

root@ccra-15:~# heat stack-show jsm-5dff1f92-5f5c-404b-896f-7e737b2f4121
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| capabilities | [] |
| creation_time | 2016-01-28T19:01:18Z |
| description | HOT template to create network service |
| disable_rollback | True |
| id | fbf2e383-cc80-4fc4-886f-495c47d80346 |
| links | http://10.102.28.148:8004/v1/7459eae2bd9449dd928ee70f0c98dd06/stacks/jsm-5dff1f92-5f5c-404b-896f-7e737b2f4121/fbf2e383-cc80-4fc4-886f-495c47d80346 |
| notification_topics | [] |
| parameters | { |
| | "right-net-id": "edb8569a-a3da-4d80-b1da-003679024738", |
| | "OS::stack_name": "jsm-5dff1f92-5f5c-404b-896f-7e737b2f4121", |
| | "nsid": "5dff1f92-5f5c-404b-896f-7e737b2f4121", |
| | "mgmt-net-id": "05d94019-145e-46ab-a01c-9e40005fbdd7", |
| | "OS::stack_id": "fbf2e383-cc80-4fc4-886f-495c47d80346", |
| | "left-net-rt": "[u'64512:67132']", |
| | "left-subnet-cidr": "67.19.132.0/24" |
| | } |
| stack_name | jsm-5dff1f92-5f5c-404b-896f-7e737b2f4121 |
| stack_status | UPDATE_IN_PROGRESS |
| stack_status_reason | Stack UPDATE started |
| template_description | HOT template to create network service |
| timeout_mins | None |
| updated_time | None |
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+

root@ccra-15:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 2.21.1-14 14
contrail-config 2.21.1-14 14
contrail-config-openstack 2.21.1-14 14
contrail-control 2.21.1-14 14
contrail-dns 2.21.1-14 14
contrail-f5 2.21.1-14 14
contrail-fabric-utils 2.21.1-14 14
contrail-heat 2.21.1-14 14
contrail-install-packages 2.21.1-14~icehouse 14
contrail-lib 2.21.1-14 14
contrail-nodemgr 2.21.1-14 14
contrail-nova-networkapi 2.21.1-14 14