vcenter-as-compute: fab setup_all aborts due to Invalid OpenStack Identity credentials

[Sarath]

This is regression bug seen on Vcenter-as-compute sanity
Build#43 was good and issue seen on Build#45

Please find below the Jenkins logs of this failure sequence/logs,
http://anamika.englab.juniper.net:8080/job/ubuntu-14-04_Kilo_vcenter_compute_HA_Sanity/358/consoleFull

Please find below the details,

::
::
2017-07-27 19:58:00:973372: [root@10.87.36.10] sudo: openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT auth_strategy keystone
2017-07-27 19:58:01:057052: [root@10.87.36.10] sudo: openstack-config --set /etc/ceilometer/ceilometer.conf database time_to_live 7200
2017-07-27 19:58:01:157229: [root@10.87.36.10] sudo: source /etc/contrail/openstackrc;keystone --insecure user-get ceilometer
2017-07-27 19:58:01:254283: [root@10.87.36.10] out: The request you have made requires authentication. (HTTP 401) (Request-ID: req-a2160d78-30d8-4497-a9b8-7501b6522f94)
2017-07-27 19:58:01:689172: [root@10.87.36.10] out:
2017-07-27 19:58:01:720862:
2017-07-27 19:58:01:721144: [root@10.87.36.10] sudo: source /etc/contrail/openstackrc;keystone --insecure user-create --name=ceilometer --pass=CEILOMETER_PASS --tenant=service --<email address hidden>
2017-07-27 19:58:01:721342: [root@10.87.36.10] out: Invalid OpenStack Identity credentials.
2017-07-27 19:58:02:361267: [root@10.87.36.10] out:
2017-07-27 19:58:02:392827: o return code 3 while executing 'service contrail-database status'!
2017-07-27 19:39:35:649540:
2017-07-27 19:39:35:649540:
2017-07-27 19:39:56:057310: Warning: sudo() received nonzero return code 1 while executing 'grep replSet=rs-ceilometer /etc/mongodb.conf'!
2017-07-27 19:39:56:057310:
2017-07-27 19:39:56:057310:
2017-07-27 19:39:56:276137: Warning: sudo() received nonzero return code 1 while executing 'service mongodb status | grep not'!
2017-07-27 19:39:56:276137:
2017-07-27 19:39:56:276137:
2017-07-27 19:39:56:874190: Warning: sudo() received nonzero return code 1 while executing 'grep replSet=rs-ceilometer /etc/mongodb.conf'!
2017-07-27 19:39:56:874190:
2017-07-27 19:39:56:874190:
2017-07-27 19:39:57:134235: Warning: sudo() received nonzero return code 1 while executing 'service mongodb status | grep not'!
2017-07-27 19:39:57:134235:
2017-07-27 19:39:57:134235:
2017-07-27 19:39:57:526317: Warning: sudo() received nonzero return code 1 while executing 'grep replSet=rs-ceilometer /etc/mongodb.conf'!
2017-07-27 19:39:57:526317:
2017-07-27 19:39:57:526317:
2017-07-27 19:39:57:767015: Warning: sudo() received nonzero return code 1 while executing 'service mongodb status | grep not'!
2017-07-27 19:39:57:767015:
2017-07-27 19:39:57:767015:
2017-07-27 19:39:58:097190: Warning: sudo() received nonzero return code 252 while executing 'mongo --host 10.87.36.10 --quiet --eval 'rs.conf()._id''!
2017-07-27 19:39:58:097190:
2017-07-27 19:39:58:097190:
2017-07-27 19:40:35:526879: Warning: sudo() received nonzero return code 2 while executing 'sed -i '/umask 007/ a\limit nofile 10240 10240' /etc/init/mysql.conf'!
2017-07-27 19:40:35:526879:
2017-07-27 19:40:35:526879:
2017-07-27 19:40:35:930106: Warning: sudo() received nonzero return code 2 while executing 'sed -i '/umask 007/ a\limit nofile 10240 10240' /etc/init/mysql.conf'!
2017-07-27 19:40:35:930106:
2017-07-27 19:40:35:930106:
2017-07-27 19:40:36:325025: Warning: sudo() received nonzero return code 2 while executing 'sed -i '/umask 007/ a\limit nofile 10240 10240' /etc/init/mysql.conf'!
2017-07-27 19:40:36:325025:
2017-07-27 19:40:36:325025:
2017-07-27 19:46:59:060873: Warning: sudo() received nonzero return code 1 while executing 'grep 'def hash_key' /etc/openstack-dashboard/local_settings.py'!
2017-07-27 19:46:59:060873:
2017-07-27 19:46:59:060873:
2017-07-27 19:46:59:889769: Warning: sudo() received nonzero return code 1 while executing 'grep ''KEY_FUNCTION': hash_key,' /etc/openstack-dashboard/local_settings.py'!
2017-07-27 19:46:59:889769:
2017-07-27 19:46:59:889769:
2017-07-27 19:52:24:787780: Warning: sudo() received nonzero return code 1 while executing 'grep 'def hash_key' /etc/openstack-dashboard/local_settings.py'!
2017-07-27 19:52:24:787780:
2017-07-27 19:52:24:787780:
2017-07-27 19:52:25:735508: Warning: sudo() received nonzero return code 1 while executing 'grep ''KEY_FUNCTION': hash_key,' /etc/openstack-dashboard/local_settings.py'!
2017-07-27 19:52:25:735508:
2017-07-27 19:52:25:735508:
2017-07-27 19:57:51:390117: Warning: sudo() received nonzero return code 1 while executing 'grep 'def hash_key' /etc/openstack-dashboard/local_settings.py'!
2017-07-27 19:57:51:390117:
2017-07-27 19:57:51:390117:
2017-07-27 19:57:52:168950: Warning: sudo() received nonzero return code 1 while executing 'grep ''KEY_FUNCTION': hash_key,' /etc/openstack-dashboard/local_settings.py'!
2017-07-27 19:57:52:168950:
2017-07-27 19:57:52:168950:
2017-07-27 19:58:01:721229: Warning: sudo() received nonzero return code 1 while executing 'source /etc/contrail/openstackrc;keystone --insecure user-get ceilometer'!
2017-07-27 19:58:01:721229:
2017-07-27 19:58:01:721229:
2017-07-27 19:58:02:393110: Fatal error: sudo() received nonzero return code 1 while executing!
2017-07-27 19:58:02:393110:
2017-07-27 19:58:02:393110: Requested: source /etc/contrail/openstackrc;keystone --insecure user-create --name=ceilometer --pass=CEILOMETER_PASS --tenant=service --<email address hidden>
2017-07-27 19:58:02:393110: Executed: sudo -S -p 'sudo password:' /bin/bash
2017-07-27 19:58:02:393011: Disconnecting from 10.87.36.20... done.
2017-07-27 19:58:02:507122: Disconnecting from 10.87.36.19... done.
2017-07-27 19:58:02:510638: Disconnecting from 10.87.36.18... done.
2017-07-27 19:58:02:514308: Disconnecting from 10.87.36.12... done.
2017-07-27 19:58:02:529971: Disconnecting from 10.87.36.11... done.
2017-07-27 19:58:02:597566: Disconnecting from 10.87.36.10... done.
2017-07-27 19:58:02:711480: -l -c "source /etc/contrail/openstackrc;keystone --insecure user-create --name=ceilometer --pass=CEILOMETER_PASS --tenant=service --<email address hidden>"
2017-07-27 19:58:02:393110:
2017-07-27 19:58:02:393154: Aborting.
2017-07-27 19:58:02:393154: root@5a10s31:~/fabric-utils#
root@5a10s31:~/fabric-utils#
root@5a10s31:~/fabric-utils#

[Sarath (nsarath)]

complete Fab logs @

nsarath@ubuntu-build04:/auto/cores/1707125$ ls -ltrd *
-rwxrwxrwx 1 nsarath test 5836800 Jul 27 23:26 Ctrl-A-Utils.tar

[Jeba Paulaiyan (jebap)]

@jebap, we know what went wrong and now setup_all is going through…but not yet identified why it went wrong in the first place

[amudhar (amudha)]

Fix committed into R3.2 through https://review.opencontrail.org/#/c/34172/.

[amudhar (amudha)]

Issue is already fixed in mainline through #1689640.