3029:mainline centos/redhat kilo setup_all fails as contrail-database fails to start

[Sudheendra Rao]

setup_all fails as contrail-database fails to start on centos/redhat kilo on mainline build 3029.

The setup nodeb8 is in failed state.

2017-01-19 15:36:47:760374: [root@10.204.216.201] out: Jan 19 15:34:04 nodeb8.englab.juniper.net cassandra[5351]: Jan 19 15:33:59 nodeb8.englab.juniper.net cassandra[5351]: Shutdown Cassandra: bash: line 0: kill: (3547) - No such process
2017-01-19 15:36:47:760527: [root@10.204.216.201] out:
2017-01-19 15:36:47:762222:
2017-01-19 15:36:47:762685: [root@10.204.216.201] sudo: service contrail-database status | head -10
2017-01-19 15:37:07:783140: [root@10.204.216.201] out: cassandra.service - SYSV: Starts and stops Cassandra
2017-01-19 15:37:07:988025: [root@10.204.216.201] out: Loaded: loaded (/etc/rc.d/init.d/cassandra)
2017-01-19 15:37:07:988289: [root@10.204.216.201] out: Active: active (exited) since Thu 2017-01-19 15:34:05 IST; 3min 2s ago
2017-01-19 15:37:07:988449: [root@10.204.216.201] out: Process: 5351 ExecStop=/etc/rc.d/init.d/cassandra stop (code=exited, status=1/FAILURE)
2017-01-19 15:37:07:988637: [root@10.204.216.201] out: Process: 5473 ExecStart=/etc/rc.d/init.d/cassandra start (code=exited, status=0/SUCCESS)
2017-01-19 15:37:07:988800: [root@10.204.216.201] out:
2017-01-19 15:37:07:988950: [root@10.204.216.201] out: Jan 19 15:34:04 nodeb8.englab.juniper.net cassandra[5351]: Jan 19 15:32:38 nodeb8.englab.juniper.net systemd[1]: Started SYSV: Starts and stops Cassandra.
2017-01-19 15:37:07:989102: [root@10.204.216.201] out: Jan 19 15:34:04 nodeb8.englab.juniper.net cassandra[5351]: Jan 19 15:33:59 nodeb8.englab.juniper.net systemd[1]: Stopping SYSV: Starts and stops Cassandra...
2017-01-19 15:37:07:989261: [root@10.204.216.201] out: Jan 19 15:34:04 nodeb8.englab.juniper.net cassandra[5351]: Jan 19 15:33:59 nodeb8.englab.juniper.net su[5359]: (to cassandra) root on none
2017-01-19 15:37:07:989450: [root@10.204.216.201] out: Jan 19 15:34:04 nodeb8.englab.juniper.net cassandra[5351]: Jan 19 15:33:59 nodeb8.englab.juniper.net cassandra[5351]: Shutdown Cassandra: bash: line 0: kill: (3547) - No such process
2017-01-19 15:37:07:989634: [root@10.204.216.201] out:
2017-01-19 15:37:07:991072:
2017-01-19 15:37:07:991425: Disconnecting from stack@10.204.216.49... done.
2017-01-19 15:37:28:076273: Disconnecting from 10.204.216.201... done.
2017-01-19 15:37:28:108426: Service contrail-database not running.
2017-01-19 15:37:28:108685: Connection to 10.204.216.201 closed.
+ rv=1
+ return 1
+ debug_and_die 'fab setup_interface or setup_all task failed'
+ local 'message=fab setup_interface or setup_all task failed'

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28420
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28420
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/fc35a37e4c2c99c12b268c67d44ac6f44ef9d881
Submitter: Zuul (<email address hidden>)
Branch: master

commit fc35a37e4c2c99c12b268c67d44ac6f44ef9d881
Author: Megh Bhatt <email address hidden>
Date: Fri Feb 3 15:50:59 2017 -0800

Fix verify_database on centos7 and redhat

On centos7 and redhat, contrail-database is still a SYSV service and
hence its status needs to be checked by looking at the return code of
service contrail-database status rather than the output since the output
does not contain the word running. Pipe-ing head -n 10 changes the return
code to always return 0. The above was broken due to commit for bug #1638670
commit f378fac60aa4372bc50c8f96eefef1a5df0c5d3b which made setup_all work
on xenial. Revert the change to verify_service made in that commit.

Change-Id: If38f781713145d28f8725634685f8f98661bc056
Closes-Bug: #1657934