Fresh install 3.2 #29 provisioning fails with keystone issues

[Sarath]

This issue seen multiple times on Sanity automated runs of Vcenter-compute
Amudha debugged this and below findings,
---------------
keystone was already started and when supervisor-openstack tries to start it, it fails with port already in use error
v have pre and post install scripts in contrail-openstack that stop keystone and then start it from supervisor-openstack, and they all seemed to be in place on the failed node
that is why wanted to install manually and see whats happening
but seems to go through fine
----------------

 >>> below failed outputs
2017-03-15 14:28:46:158766: [root@10.84.13.33] Executing task 'verify_openstack'
2017-03-15 14:28:46:158879: [root@10.84.13.33] sudo: python -c 'from platform import linux_distribution; print linux_distribution()'
2017-03-15 14:28:46:159015: [root@10.84.13.33] out: ('Ubuntu', '14.04', 'trusty')
2017-03-15 14:28:46:193296: [root@10.84.13.33] out:
2017-03-15 14:28:46:196639:
2017-03-15 14:28:46:225485: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:28:46:225733: [root@10.84.13.33] out: keystone STARTING
2017-03-15 14:28:46:398729: [root@10.84.13.33] out:
2017-03-15 14:28:46:398925:
2017-03-15 14:28:46:399047: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:29:06:419021: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:29:06:586650: [root@10.84.13.33] out:
2017-03-15 14:29:06:586863:
2017-03-15 14:29:06:587126: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:29:26:606640: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:29:26:709554: [root@10.84.13.33] out:
2017-03-15 14:29:26:710907:
2017-03-15 14:29:26:720505: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:29:46:731976: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:29:46:835432: [root@10.84.13.33] out:
2017-03-15 14:29:46:835635:
2017-03-15 14:29:46:836448: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:30:06:855177: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:30:07:022576: [root@10.84.13.33] out:
2017-03-15 14:30:07:022764:
2017-03-15 14:30:07:022995: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:30:27:042361: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:30:27:177038: [root@10.84.13.33] out:
2017-03-15 14:30:27:184424:
2017-03-15 14:30:27:209481: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:30:47:226381: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:30:47:329995: [root@10.84.13.33] out:
2017-03-15 14:30:47:330227:
2017-03-15 14:30:47:331251: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:31:07:338409: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:31:07:474891: [root@10.84.13.33] out:
2017-03-15 14:31:07:475068:
2017-03-15 14:31:07:475267: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:31:27:494378: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:31:27:598384: [root@10.84.13.33] out:
2017-03-15 14:31:27:598592:
2017-03-15 14:31:27:598839: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:31:47:618642: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:31:47:785450: [root@10.84.13.33] out:
2017-03-15 14:31:47:785603:
2017-03-15 14:31:47:785713: Disconnecting from 10.84.13.32... done.
2017-03-15 14:32:07:867950: Disconnecting from 10.84.13.2... done.
2017-03-15 14:32:07:931758: Disconnecting from 10.84.13.222... done.
2017-03-15 14:32:07:963468: Disconnecting from 10.84.13.223... done.
2017-03-15 14:32:08:027281: Disconnecting from 10.84.13.33... done.
2017-03-15 14:32:08:141189: Disconnecting from 10.84.13.38... done.
2017-03-15 14:32:08:211044: Service keystone not running.
2017-03-15 14:32:08:211159: root@a6s32:~/fabric-utils#
root@a6s32:~/fabric-utils#
root@a6s32:~/fabric-utils# contrail-status
== Contrail Control ==
supervisor-control: inactive (disabled on boot)
unix:///var/run/supervisord_control.sockno

== Contrail Analytics ==
supervisor-analytics: inactive (disabled on boot)
unix:///var/run/supervisord_analytics.sockno

== Contrail Config ==
supervisor-config: inactive (disabled on boot)
unix:///var/run/supervisord_config.sockno

== Contrail Web UI ==
supervisor-webui: inactive (disabled on boot)
unix:///var/run/supervisord_webui.sockno

== Contrail Database ==
contrail-database: active

== Contrail Supervisor Database ==
supervisor-database: active
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

root@a6s32:~/fabric-utils#

[Sarath (nsarath)]

nsarath@ubuntu-build04:/auto/cores/1673274$ ls -ltrd *
-rwxrwxrwx 1 nsarath test 8960000 Mar 15 15:28 Ctrl-A-Utils.tar
-rwxrwxrwx 1 nsarath test 427786240 Mar 15 15:28 Ctrl-A-log.tar
-rwxrwxrwx 1 nsarath test 422901760 Mar 15 15:28 Ctrl-B-log.tar
-rwxrwxrwx 1 nsarath test 429762560 Mar 15 15:28 Ctrl-C-log.tar
-rwxrwxrwx 1 nsarath test 9594880 Mar 15 15:36 utils-buildServer.tar

[Jeba Paulaiyan (jebap)]

Similar issue is seen in openstack cluster also:

2017-03-14 20:35:31:393150: [root@10.87.119.12] out: [localhost] local: setup-quantum-in-keystone --ks_server_ip 192.16.7.28 --quant_server_ip 192.16.7.27 --tenant admin --user admin --password c0ntrail123 --svc_password c0ntrail123 --svc_tenant_name service --root_password None --auth_protocol https --insecure --region_name RegionOne
2017-03-14 20:35:31:400578: [root@10.87.119.12] out: /usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
2017-03-14 20:35:32:102125: [root@10.87.119.12] out: SNIMissingWarning
2017-03-14 20:35:32:102320: [root@10.87.119.12] out: /usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
2017-03-14 20:35:32:102364: [root@10.87.119.12] out: InsecurePlatformWarning
2017-03-14 20:35:32:102400: [root@10.87.119.12] out: /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:794: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
2017-03-14 20:35:32:102430: [root@10.87.119.12] out: InsecureRequestWarning)
2017-03-14 20:35:32:102465: [root@10.87.119.12] out: Authorization Failed: Bad Gateway (HTTP 502)
2017-03-14 20:35:32:467432: [root@10.87.119.12] out: Traceback (most recent call last):
2017-03-14 20:35:32:467605: [root@10.87.119.12] out: File "/usr/bin/setup-quantum-in-keystone", line 9, in <module>
2017-03-14 20:35:32:467662: [root@10.87.119.12] out: load_entry_point('ContrailProvisioning==0.1dev', 'console_scripts', 'setup-quantum-in-keystone')()
2017-03-14 20:35:32:467713: [root@10.87.119.12] out: File "/usr/local/lib/python2.7/dist-packages/contrail_provisioning/config/quantum_in_keystone_setup.py", line 306, in main
2017-03-14 20:35:32:467771: [root@10.87.119.12] out: quant_obj = QuantumSetup(args_str)
2017-03-14 20:35:32:467832: [root@10.87.119.12] out: File "/usr/local/lib/python2.7/dist-packages/contrail_provisioning/config/quantum_in_keystone_setup.py", line 72, in __init__
2017-03-14 20:35:32:467893: [root@10.87.119.12] out: raise e
2017-03-14 20:35:32:467966: [root@10.87.119.12] out: keystoneauth1.exceptions.auth.AuthorizationFailure: Authorization Failed: Bad Gateway (HTTP 502)
2017-03-14 20:35:32:468015: [root@10.87.119.12] out:
2017-03-14 20:35:32:471683: [root@10.87.119.12] out: Fatal error: local() encountered an error (return code 1) while executing 'setup-quantum-in-keystone --ks_server_ip 192.16.7.28 --quant_server_ip 192.16.7...

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/29667
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/29668
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/29668
Committed: http://github.org/Juniper/contrail-provisioning/commit/dcafa86868444c27306a1ae101044b34d327f797
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit dcafa86868444c27306a1ae101044b34d327f797
Author: Ignatious Johnson Christopher <email address hidden>
Date: Thu Mar 16 21:58:12 2017 -0700

Removing the unnecessary supervisor-openstack

restart during keystone-server-setup.sh, immediately
after supervisor-openstack start.
Becasue supervisor-openstack start followed by supervisor-openstack
restart without delay cause supervisor to leave the child orphaned.

Change-Id: Ib8b4ac24423b157f8d90f4bbffd0b898f9413ad0
Closes-Bug: 1673274

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/29667
Committed: http://github.org/Juniper/contrail-provisioning/commit/8b4f1cdd27e73d97082a80a3dca00f7a32941342
Submitter: Zuul (<email address hidden>)
Branch: master

commit 8b4f1cdd27e73d97082a80a3dca00f7a32941342
Author: Ignatious Johnson Christopher <email address hidden>
Date: Thu Mar 16 21:58:12 2017 -0700

Removing the unnecessary supervisor-openstack

restart during keystone-server-setup.sh, immediately
after supervisor-openstack start.
Becasue supervisor-openstack start followed by supervisor-openstack
restart without delay cause supervisor to leave the child orphaned.

Change-Id: Ib8b4ac24423b157f8d90f4bbffd0b898f9413ad0
Closes-Bug: 1673274