Contrail 2.21-14: heat stack-create/stack-update failure due to keystone failure

Bug #1537884 reported by Deepak Jeyaraman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
New
Undecided
Unassigned

Bug Description

Build:
======

Contrail 2.21-14

Logs:
======

Logs are stored here :

$ pwd
/volume/vmcores/sagrawal/jdeepak_temp/NFV/contrail_bug/1537884
jdeepak@ttsv-shell03 /volume/vmcores/sagrawal/jdeepak_temp/NFV/contrail_bug/1537884
$ ls
total 252192
-rwxrwxrwx 1 jdeepak software 257228800 Jan 25 11:52 logs.tar
jdeepak@ttsv-shell03 /volume/vmcores/sagrawal/jdeepak_temp/NFV/contrail_bug/1537884

Problem:
========

Multi node contrail setup , with 3 servers as contrail config, control and db .

Instantiating 30 instances parallely, occassionally one heat stack fails to get created

root@ccra-15:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 2.21.1-14 14
contrail-config 2.21.1-14 14
contrail-config-openstack 2.21.1-14 14
contrail-control 2.21.1-14 14
contrail-dns 2.21.1-14 14
contrail-f5 2.21.1-14 14
contrail-fabric-utils 2.21.1-14 14
contrail-heat 2.21.1-14 14
contrail-install-packages 2.21.1-14~icehouse 14
contrail-lib 2.21.1-14 14
contrail-nodemgr 2.21.1-14 14
contrail-nova-networkapi 2.21.1-14 14

root@ccra-15:~# heat stack-list
+--------------------------------------+------------------------------------------+-----------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------------------------------------+-----------------+----------------------+
| 8277e40f-e909-4a12-abf9-218101ae7252 | jsm-e68c0d30-7cb7-4ec0-81cd-5615194a3b80 | UPDATE_COMPLETE | 2016-01-25T19:31:12Z |
| f84aff88-4c77-47d7-8ba0-c3aa82c1e803 | jsm-21421f1e-a3e5-41bc-b03f-13b5dc8e8ef4 | UPDATE_COMPLETE | 2016-01-25T19:31:18Z |
| febe2ee9-c039-48bc-af1d-500a63e94729 | jsm-d6ec3bca-acfd-4c5f-8157-04db2264bd7d | UPDATE_COMPLETE | 2016-01-25T19:31:22Z |
| 3dafabd8-3e6d-40f4-a006-b3db56594277 | jsm-f0cac78e-c5a6-4e7e-b067-3a8bb9e6c810 | UPDATE_COMPLETE | 2016-01-25T19:31:23Z |
| dac8df41-d5cd-48f2-83ce-a7c56b899b7d | jsm-c1076470-4a9e-426c-bfc2-5321b303b0f8 | UPDATE_COMPLETE | 2016-01-25T19:31:27Z |
| 7a191697-a5a5-4388-80ec-5e23a1499409 | jsm-6c70fa12-bc37-45b6-a95d-fbf9e1060493 | UPDATE_COMPLETE | 2016-01-25T19:31:32Z |
| 147128f3-8caa-46e0-bfdb-688f08237289 | jsm-653c7bc5-1bd2-47d4-abe2-399f1fdb9af5 | UPDATE_COMPLETE | 2016-01-25T19:31:51Z |
| f11d3f44-6521-4d4b-8cae-66838241fbf4 | jsm-a989efbe-abf9-44e4-92ef-b0959cecc298 | UPDATE_COMPLETE | 2016-01-25T19:31:53Z |
| 734003db-d937-4b17-8ca7-b182932a80f0 | jsm-247ca8e5-cac2-419e-997f-0faf7c337727 | UPDATE_COMPLETE | 2016-01-25T19:31:56Z |
| 371e2de0-a020-4583-9bd1-7d0ba7feb0e7 | jsm-499e7d53-a4a8-4a5e-be2d-5f4c4f28e623 | UPDATE_COMPLETE | 2016-01-25T19:32:00Z |
| 335fea1c-c017-42d4-a7ad-5deaf83e6dc9 | jsm-5ff31375-1fcb-4581-9e71-2eef3a15e8d7 | UPDATE_COMPLETE | 2016-01-25T19:32:04Z |
| 94102c8c-b67c-41e7-9103-96bb43f7a7c1 | jsm-10dd2119-b54f-467b-9407-c6ce56a0874c | UPDATE_COMPLETE | 2016-01-25T19:32:05Z |
| a60a1523-34d6-4520-b5e3-d31d45ce2541 | jsm-81219095-bfea-47b2-b884-ea3add2237c8 | UPDATE_COMPLETE | 2016-01-25T19:32:07Z |
| d11552b8-f278-4c2d-b17f-dfac82b4e206 | jsm-95686564-e3a8-4d59-9bbb-f8eb63c9bcb7 | UPDATE_COMPLETE | 2016-01-25T19:32:07Z |
| 2b7a7344-ee3f-4439-8a09-fc6511c7ded7 | jsm-b14dc14d-1709-4a93-9f71-bf510f3cb3e6 | UPDATE_COMPLETE | 2016-01-25T19:32:12Z |
| 305d068c-e9b6-4862-8214-f729329dbea3 | jsm-262f71d9-935e-420f-9651-3c57d3786d5a | UPDATE_COMPLETE | 2016-01-25T19:32:18Z |
| 2d446d90-6ee5-4f26-9de0-68e7e281f077 | jsm-3fe475b8-e95e-4392-a41c-731876318142 | UPDATE_COMPLETE | 2016-01-25T19:32:22Z |
| 69828e36-7821-45bb-bf00-329b2febfba2 | jsm-98f33eea-c3a3-44fe-8927-4c4282480777 | UPDATE_COMPLETE | 2016-01-25T19:32:30Z |
| 2f0027f3-37f1-48bc-a591-66d823adc860 | jsm-0a85d672-c137-4987-b1bb-f4d144ce067f | CREATE_FAILED | 2016-01-25T19:32:36Z |
| 3bf5a35a-a5d9-403d-9511-9dc4a8ce5ec9 | jsm-be2572cc-ad77-4e31-b29f-5b1b2844bfb9 | UPDATE_COMPLETE | 2016-01-25T19:32:36Z |
| 63bab1de-ca4e-4499-b385-0ab11e65e414 | jsm-ca90baad-a7e3-46cb-b623-6449e551079d | UPDATE_COMPLETE | 2016-01-25T19:32:36Z |
| cd03a655-23e4-40f4-a635-d38035c38413 | jsm-10bcab34-db09-4fe0-b0ed-7cb135b65a86 | UPDATE_COMPLETE | 2016-01-25T19:32:36Z |
| d043cabe-d53b-4828-b284-9ffa627122f9 | jsm-747c7f8e-b749-4a69-8a90-8624fd141e3d | UPDATE_COMPLETE | 2016-01-25T19:32:38Z |
| 96184320-b5b1-49a8-a451-3419658c5f08 | jsm-61ccc3eb-b974-42e9-acc6-f2f1e2a61390 | UPDATE_COMPLETE | 2016-01-25T19:32:42Z |
| 99e56337-9411-4e8f-b7da-c3c75bdfeb85 | jsm-70ed5598-d356-490c-99a4-228ff59c8a1c | UPDATE_COMPLETE | 2016-01-25T19:32:46Z |
| 10a78965-d77b-4f63-91bc-52ac39d5edd4 | jsm-f0a6e4c7-68c3-4deb-856d-646f440ef6fc | UPDATE_COMPLETE | 2016-01-25T19:32:47Z |
| deb3b2e5-93d9-4952-aaf0-688061db4a77 | jsm-dcdfb8d2-35a1-4f74-966a-63702e03f327 | UPDATE_COMPLETE | 2016-01-25T19:32:47Z |
| 3b6cb4a8-3b5c-45fc-9edb-c5e3eb7c79f4 | jsm-5a3e2ad5-53c0-4e89-8fbd-9f3b3d67cd92 | UPDATE_COMPLETE | 2016-01-25T19:32:54Z |
| 8305cf54-e714-4273-9c41-0247591b3da4 | jsm-51a7bdc6-9523-4309-8926-89b72c5b0f43 | UPDATE_COMPLETE | 2016-01-25T19:33:00Z |
| 26be0736-d769-4f71-a7fe-1f509386b3e3 | jsm-f6a01c48-0a29-4ebc-a30e-c3b3407ba499 | UPDATE_COMPLETE | 2016-01-25T19:33:03Z |
+--------------------------------------+------------------------------------------+-----------------+----------------------+

root@ccra-15:~# heat stack-show jsm-0a85d672-c137-4987-b1bb-f4d144ce067f
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| capabilities | [] |
| creation_time | 2016-01-25T19:32:36Z |
| description | HOT template to create network service |
| disable_rollback | True |
| id | 2f0027f3-37f1-48bc-a591-66d823adc860 |
| links | http://10.102.28.148:8004/v1/7459eae2bd9449dd928ee70f0c98dd06/stacks/jsm-0a85d672-c137-4987-b1bb-f4d144ce067f/2f0027f3-37f1-48bc-a591-66d823adc860 |
| notification_topics | [] |
| parameters | { |
| | "right-net-id": "edb8569a-a3da-4d80-b1da-003679024738", |
| | "OS::stack_name": "jsm-0a85d672-c137-4987-b1bb-f4d144ce067f", |
| | "nsid": "0a85d672-c137-4987-b1bb-f4d144ce067f", |
| | "mgmt-net-id": "05d94019-145e-46ab-a01c-9e40005fbdd7", |
| | "OS::stack_id": "2f0027f3-37f1-48bc-a591-66d823adc860", |
| | "left-net-rt": "[u'64512:67024']", |
| | "left-subnet-cidr": "67.19.24.0/24" |
| | } |
| stack_name | jsm-0a85d672-c137-4987-b1bb-f4d144ce067f |
| stack_status | CREATE_FAILED |
| stack_status_reason | Resource CREATE failed: RuntimeError: Unable to connect |
| | to keystone for authentication. Verify keystone server |
| | details |
| template_description | HOT template to create network service |
| timeout_mins | None |
| updated_time | None |
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+

information type: Proprietary → Public
description: updated
Revision history for this message
Deepak Jeyaraman (jdeepak) wrote : Re: Contrail 2.21-14: heat stack-create failure due to keystone failure
Download full text (4.8 KiB)

Also, seeing the same error for heat stack update .

| 91794e80-3455-4e01-ac23-5d3d65e331d2 | jsm-83eecde6-58ed-4855-8dcf-03a3c66d8f53 | UPDATE_FAILED | 2016-01-25T21:02:02Z |

root@ccra-15:~# heat stack-show jsm-83eecde6-58ed-4855-8dcf-03a3c66d8f53
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| capabilities | [] |
| creation_time | 2016-01-25T21:02:02Z |
| description | HOT template to create network service |
| disable_rollback | True |
| id | 91794e80-3455-4e01-ac23-5d3d65e331d2 |
| links | http://10.102.28.148:8004/v1/7459eae2bd9449dd928ee70f0c98dd06/stacks/jsm-83eecde6-58ed-4855-8dcf-03a3c66d8f53/91794e80-3455-4e01-ac23-5d3d65e331d2 |
| notification_topics | [] |
| parameters | { |
| | "right-net-id": "edb8569a-a3da-4d80-b1da-003679024738", |
| | "OS::stack_name": "jsm-83eecde6-58ed-4855-8dcf-03a3c66d8f53", |
| | "nsid": "83eecde6-58ed-4855-8dcf-03a3c66d8f53", |
| | "mgmt-net-id": "05d94019-145e-46ab-a01c-9e40005fbdd7", |
| | "OS::stack_id": "91794e80-3455-4e01-ac23-5d3d65e331d2", |
| | "left-net-rt": "[u'64512:67092']", ...

Read more...

summary: - Contrail 2.21-14: heat stack-create failure
+ Contrail 2.21-14: heat stack-create failure due to keystone failure
summary: - Contrail 2.21-14: heat stack-create failure due to keystone failure
+ Contrail 2.21-14: heat stack-create/stack-update failure due to keystone
+ failure
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.