R3.1.1.x build2 - liberty HA setup, setup_all fails while provisioning ceilometer
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R3.1 |
Fix Committed
|
Critical
|
Ranjeet R | |||
R3.1.1.x |
Fix Committed
|
Critical
|
Ranjeet R | |||
R3.2 |
Fix Committed
|
Critical
|
Ranjeet R | |||
Trunk |
Fix Committed
|
Critical
|
Ranjeet R |
Bug Description
setup_all fails while provisioning ceilometer (keystone authentication failure) on R3.1.1.x build2 liberty HA setup.
017-01-06 12:36:20:941292: Warning: sudo() received nonzero return code 1 while executing 'grep ''KEY_FUNCTION': hash_key,' /etc/openstack-
2017-01-06 12:36:20:941292:
2017-01-06 12:36:20:941292:
2017-01-06 12:36:25:392949: Warning: sudo() received nonzero return code 1 while executing 'source /etc/contrail/
2017-01-06 12:36:25:392949:
2017-01-06 12:36:25:392949:
2017-01-06 12:36:25:927226: Fatal error: sudo() received nonzero return code 1 while executing!
2017-01-06 12:36:25:927226:
2017-01-06 12:36:25:927226: Requested: source /etc/contrail/
2017-01-06 12:36:25:927226: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "source /etc/contrail/
2017-01-06 12:36:25:927226:
2017-01-06 12:36:25:927248: Aborting.
The request you have made requires authentication. (HTTP 401) (Request-ID: req-0b948376-
keystone log file:
2017-01-06 18:49:33.862 28753 WARNING keystone.
2017-01-06 18:49:33.863 28753 INFO eventlet.
2017-01-06 18:49:50.281 28751 INFO keystone.
2017-01-06 18:49:50.297 28751 WARNING keystone.
2017-01-06 18:49:50.298 28751 INFO eventlet.
2017-01-06 18:50:00.650 28752 INFO keystone.
2017-01-06 18:50:00.671 28752 WARNING keystone.
2017-01-06 18:50:00.675 28752 INFO eventlet.
2017-01-06 18:50:15.542 28752 INFO keystone.
2017-01-06 18:50:15.556 28752 WARNING keystone.
2017-01-06 18:50:15.557 28752 INFO eventlet.
Setup details (have locked the setup):
Config Nodes : [u'nodec21', u'nodec19', u'nodec20']
Control Nodes : [u'nodec21', u'nodec20']
Compute Nodes : [u'nodec61', u'nodec60', u'nodec30']
Openstack Node : [u'nodec21', u'nodec19', u'nodec20']
WebUI Node : nodec20
Analytics Nodes : [u'nodec21', u'nodec19', u'nodec20']
Database Nodes : [u'nodec21', u'nodec19', u'nodec20']
This was root caused to keystone server already running when the setup starts. As part of openstack setup postinstall, we stop all the openstack services but in this particular case, the system had gone through a restart for kernel upgrade which had started the service.