R3.1.1.x build2 - liberty HA setup, setup_all fails while provisioning ceilometer

[Sudheendra Rao]

setup_all fails while provisioning ceilometer (keystone authentication failure) on R3.1.1.x build2 liberty HA setup.
017-01-06 12:36:20:941292: Warning: sudo() received nonzero return code 1 while executing 'grep ''KEY_FUNCTION': hash_key,' /etc/openstack-dashboard/local_settings.py'!
2017-01-06 12:36:20:941292:
2017-01-06 12:36:20:941292:
2017-01-06 12:36:25:392949: Warning: sudo() received nonzero return code 1 while executing 'source /etc/contrail/openstackrc;keystone --insecure user-get ceilometer'!
2017-01-06 12:36:25:392949:
2017-01-06 12:36:25:392949:
2017-01-06 12:36:25:927226: Fatal error: sudo() received nonzero return code 1 while executing!
2017-01-06 12:36:25:927226:
2017-01-06 12:36:25:927226: Requested: source /etc/contrail/openstackrc;keystone --insecure user-create --name=ceilometer --pass=CEILOMETER_PASS --tenant=service --<email address hidden>
2017-01-06 12:36:25:927226: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "source /etc/contrail/openstackrc;keystone --insecure user-create --name=ceilometer --pass=CEILOMETER_PASS --tenant=service --<email address hidden>"
2017-01-06 12:36:25:927226:
2017-01-06 12:36:25:927248: Aborting.

The request you have made requires authentication. (HTTP 401) (Request-ID: req-0b948376-dc12-4556-943f-c190a5f7fc04)

keystone log file:
2017-01-06 18:49:33.862 28753 WARNING keystone.common.wsgi [req-29f730d5-9d78-468d-ad21-88f3160730bf - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.10.5
2017-01-06 18:49:33.863 28753 INFO eventlet.wsgi.server [req-29f730d5-9d78-468d-ad21-88f3160730bf - - - - -] 192.168.10.5 - - [06/Jan/2017 18:49:33] "POST /v2.0/tokens HTTP/1.1" 401 399 0.023254
2017-01-06 18:49:50.281 28751 INFO keystone.common.wsgi [req-f5f4c86c-5dfa-4a30-ae79-e8cb9ffa616e - - - - -] POST http://localhost:5000/v2.0/tokens
2017-01-06 18:49:50.297 28751 WARNING keystone.common.wsgi [req-f5f4c86c-5dfa-4a30-ae79-e8cb9ffa616e - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.10.5
2017-01-06 18:49:50.298 28751 INFO eventlet.wsgi.server [req-f5f4c86c-5dfa-4a30-ae79-e8cb9ffa616e - - - - -] 192.168.10.5 - - [06/Jan/2017 18:49:50] "POST /v2.0/tokens HTTP/1.1" 401 399 0.017711
2017-01-06 18:50:00.650 28752 INFO keystone.common.wsgi [req-7e64f2fb-8930-4050-a5d6-c159701c1147 - - - - -] POST http://localhost:5000/v2.0/tokens
2017-01-06 18:50:00.671 28752 WARNING keystone.common.wsgi [req-7e64f2fb-8930-4050-a5d6-c159701c1147 - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.10.6
2017-01-06 18:50:00.675 28752 INFO eventlet.wsgi.server [req-7e64f2fb-8930-4050-a5d6-c159701c1147 - - - - -] 192.168.10.6 - - [06/Jan/2017 18:50:00] "POST /v2.0/tokens HTTP/1.1" 401 399 0.026627
2017-01-06 18:50:15.542 28752 INFO keystone.common.wsgi [req-696a7625-0f40-48c0-b884-04d8f4a2e415 - - - - -] POST http://localhost:5000/v2.0/tokens
2017-01-06 18:50:15.556 28752 WARNING keystone.common.wsgi [req-696a7625-0f40-48c0-b884-04d8f4a2e415 - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.10.6
2017-01-06 18:50:15.557 28752 INFO eventlet.wsgi.server [req-696a7625-0f40-48c0-b884-04d8f4a2e415 - - - - -] 192.168.10.6 - - [06/Jan/2017 18:50:15] "POST /v2.0/tokens HTTP/1.1" 401 399 0.015487

Setup details (have locked the setup):
Config Nodes : [u'nodec21', u'nodec19', u'nodec20']
Control Nodes : [u'nodec21', u'nodec20']
Compute Nodes : [u'nodec61', u'nodec60', u'nodec30']
Openstack Node : [u'nodec21', u'nodec19', u'nodec20']
WebUI Node : nodec20
Analytics Nodes : [u'nodec21', u'nodec19', u'nodec20']
Database Nodes : [u'nodec21', u'nodec19', u'nodec20']

[Ranjeet R (rranjeet-n)]

This was root caused to keystone server already running when the setup starts. As part of openstack setup postinstall, we stop all the openstack services but in this particular case, the system had gone through a restart for kernel upgrade which had started the service.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28003
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28004
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28005
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28003
Committed: http://github.org/Juniper/contrail-provisioning/commit/015cf8905feada1e507b55e3980e8ad6fab6d114
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 015cf8905feada1e507b55e3980e8ad6fab6d114
Author: Ranjeet R <email address hidden>
Date: Thu Jan 19 10:19:30 2017 -0800

Fixes: R3.1.1.x build2 - liberty HA setup, setup_all fails while provisioning ceilometer

This was root caused to keystone server already running when the setup
starts. As part of openstack setup postinstall, we stop all the
openstack services but in this particular case, the system had
gone through a restart for kernel upgrade which had started the service.

Change-Id: Iac14a1edb9714d731c60ff5ff4d8076d8c3b1c15
Closes-Bug: 1654556

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28005
Committed: http://github.org/Juniper/contrail-provisioning/commit/27a6c32eb8aa4a39ed87e8f5e117b5d27c986d29
Submitter: Zuul (<email address hidden>)
Branch: master

commit 27a6c32eb8aa4a39ed87e8f5e117b5d27c986d29
Author: Ranjeet R <email address hidden>
Date: Thu Jan 19 10:29:08 2017 -0800

Fixes: R3.1.1.x build2 - liberty HA setup, setup_all fails while provisioning ceilometer

This was root caused to keystone server already running when the setup
starts. As part of openstack setup postinstall, we stop all the
openstack services but in this particular case, the system had
gone through a restart for kernel upgrade which had started the service.

Change-Id: I1e5306b79f5a24d7a079572f045a40ecaf43d5af
Closes-Bug: 1654556

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28816
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28816
Committed: http://github.org/Juniper/contrail-provisioning/commit/dcb30a8ee5c656c1abe631f18fcfc536c1040060
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit dcb30a8ee5c656c1abe631f18fcfc536c1040060
Author: Ranjeet R <email address hidden>
Date: Wed Feb 15 10:55:32 2017 -0800

Fixes: R3.1.1.x build2 - liberty HA setup, setup_all fails while provisioning ceilometer

This was root caused to keystone server already running when the setup
starts. As part of openstack setup postinstall, we stop all the
openstack services but in this particular case, the system had
gone through a restart for kernel upgrade which had started the service.

Change-Id: Ie35c3caa341d047b2bf32371f8e719549d1e2f8b
Closes-Bug: 1654556

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28004
Committed: http://github.org/Juniper/contrail-provisioning/commit/d575c880899639e58e7cf3949469f9cfc46d94bf
Submitter: Zuul (<email address hidden>)
Branch: R3.1.1.x

commit d575c880899639e58e7cf3949469f9cfc46d94bf
Author: Ranjeet R <email address hidden>
Date: Thu Jan 19 10:27:15 2017 -0800

Fixes: R3.1.1.x build2 - liberty HA setup, setup_all fails while provisioning ceilometer

This was root caused to keystone server already running when the setup
starts. As part of openstack setup postinstall, we stop all the
openstack services but in this particular case, the system had
gone through a restart for kernel upgrade which had started the service.

Change-Id: I184bbf709ae43bf3bf80507688c2bf24bd46afc4
Closes-Bug: 1654556