Contrail 2.21-14: Heat stack-delete failed because of Gateway Time-out

Bug #1527712 reported by Deepak Jeyaraman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
New
Undecided
Unassigned

Bug Description

We tracked heat stack-deletes with bug - https://bugs.launchpad.net/juniperopenstack/+bug/1524079

Now with the fix for above, we see different scenario where heat stack-deletes fail.

Contrail-collector is spinning at 1600%

root@ccra-06:~# heat stack-list | grep DELETE
| 6f60223d-7b19-40dc-8123-de8263c2b4cc | jsm-36317684-5150-449a-81c6-d983b5c104d2 | DELETE_FAILED | 2015-12-18T06:27:47Z |
| 45c7ebe0-45f2-4dd6-b964-a59e4ced32de | jsm-a5b45f54-049c-4e2f-9792-d42876594cc3 | DELETE_FAILED | 2015-12-18T06:27:54Z |
root@ccra-06:~# heat stack-show 6f60223d-7b19-40dc-8123-de8263c2b4cc
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| capabilities | [] |
| creation_time | 2015-12-18T06:27:47Z |
| description | HOT template to create network service |
| disable_rollback | True |
| id | 6f60223d-7b19-40dc-8123-de8263c2b4cc |
| links | http://10.102.28.59:8004/v1/9e20cb419c194e2283b364e14bbc4f2d/stacks/jsm-36317684-5150-449a-81c6-d983b5c104d2/6f60223d-7b19-40dc-8123-de8263c2b4cc |
| notification_topics | [] |
| parameters | { |
| | "right-net-id": "641bec47-9e5b-462c-a458-872508071245", |
| | "OS::stack_name": "jsm-36317684-5150-449a-81c6-d983b5c104d2", |
| | "nsid": "36317684-5150-449a-81c6-d983b5c104d2", |
| | "mgmt-net-id": "bdfbd02d-f75c-4bb4-8c04-1f251f9d3d44", |
| | "OS::stack_id": "6f60223d-7b19-40dc-8123-de8263c2b4cc", |
| | "left-net-rt": "[u'64512:37052']", |
| | "left-subnet-cidr": "57.19.52.0/24" |
| | } |
| stack_name | jsm-36317684-5150-449a-81c6-d983b5c104d2 |
| stack_status | DELETE_FAILED |
| stack_status_reason | Resource DELETE failed: NeutronClientException: |
| | <html><body><h1>504 Gateway Time-out</h1> The server |
| | didn't respond in time. </body></html> |
| template_description | HOT template to create network service |
| timeout_mins | None |
| updated_time | 2015-12-18T17:50:56Z |
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
root@ccra-06:~# heat stack-show 45c7ebe0-45f2-4dd6-b964-a59e4ced32de
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| capabilities | [] |
| creation_time | 2015-12-18T06:27:54Z |
| description | HOT template to create network service |
| disable_rollback | True |
| id | 45c7ebe0-45f2-4dd6-b964-a59e4ced32de |
| links | http://10.102.28.59:8004/v1/9e20cb419c194e2283b364e14bbc4f2d/stacks/jsm-a5b45f54-049c-4e2f-9792-d42876594cc3/45c7ebe0-45f2-4dd6-b964-a59e4ced32de |
| notification_topics | [] |
| parameters | { |
| | "right-net-id": "641bec47-9e5b-462c-a458-872508071245", |
| | "OS::stack_name": "jsm-a5b45f54-049c-4e2f-9792-d42876594cc3", |
| | "nsid": "a5b45f54-049c-4e2f-9792-d42876594cc3", |
| | "mgmt-net-id": "bdfbd02d-f75c-4bb4-8c04-1f251f9d3d44", |
| | "OS::stack_id": "45c7ebe0-45f2-4dd6-b964-a59e4ced32de", |
| | "left-net-rt": "[u'64512:37041']", |
| | "left-subnet-cidr": "57.19.41.0/24" |
| | } |
| stack_name | jsm-a5b45f54-049c-4e2f-9792-d42876594cc3 |
| stack_status | DELETE_FAILED |
| stack_status_reason | Resource DELETE failed: NeutronClientException: |
| | <html><body><h1>504 Gateway Time-out</h1> The server |
| | didn't respond in time. </body></html> |
| template_description | HOT template to create network service |
| timeout_mins | None |
| updated_time | 2015-12-18T17:50:34Z |
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+

Tasks: 838 total, 3 running, 835 sleeping, 0 stopped, 0 zombie
%Cpu(s): 16.5 us, 3.7 sy, 2.2 ni, 77.0 id, 0.4 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem: 26404764+total, 26337996+used, 667684 free, 239588 buffers
KiB Swap: 26832486+total, 94280 used, 26823057+free. 19900704+cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 2869 contrail 20 0 9801172 7.492g 5820 S 1599 3.0 13046:59 contrail-collec
 2863 root 20 0 0.172t 0.019t 1.088g S 349.3 7.9 4736:25 java
 2828 root 20 0 1496604 799632 6676 R 100.7 0.3 557:25.49 contrail-api
45983 root 20 0 125740 44820 2508 R 53.3 0.0 0:00.12 puppet
 2858 heat 20 0 297708 112208 5772 S 41.4 0.0 29:39.25 heat-engine
 4582 mysql 20 0 14.441g 337040 6696 S 41.4 0.1 52:15.12 mysqld
 2850 contrail 20 0 3598904 151252 5812 S 35.5 0.1 193:25.99 contrail-dns
 2925 rabbitmq 20 0 3548180 456340 2720 S 23.7 0.2 282:02.33 beam.smp
 2827 root 20 0 390144 37688 4472 S 11.8 0.0 50:27.13 contrail-discov
 2848 contrail 20 0 9509832 6.947g 8044 S 11.8 2.8 5472:10 contrail-contro

root@ccra-06:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 2.21.1-14 14
contrail-config 2.21.1-14 14
contrail-config-openstack 2.21.1-14 14
contrail-control 2.21.1-14 14
contrail-dns 2.21.1-14 14
contrail-f5 2.21.1-14 14
contrail-fabric-utils 2.21.1-14 14
contrail-heat 2.21.1-14 14
contrail-install-packages 2.21.1-14~icehouse 14
contrail-lib 2.21.1-14 14

information type: Proprietary → Public
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.