Contrail Version: 3.1.3-75~mitaka
We have noticed that whenever we run `fab stop_collector` after `fab stop_database`, at times, the task doesn’t end and contrail-alarm-gen-0-stdout.log has the following logs:
Connect attempt to <BrokerConnection host=192.168.0.131 port=9092> returned error 111. Disconnecting.
Skipping unconnected connection: <BrokerConnection host=192.168.0.131 port=9092>
Connect attempt to <BrokerConnection host=192.168.0.133 port=9092> returned error 111. Disconnecting.
Skipping unconnected connection: <BrokerConnection host=192.168.0.133 port=9092>
Connect attempt to <BrokerConnection host=192.168.0.132 port=9092> returned error 111. Disconnecting.
Skipping unconnected connection: <BrokerConnection host=192.168.0.132 port=9092>
FailedPayloadsError for -uve-5:0
There are 2 functions in our code that uses this order:
def restore_cassandra_db():
def stop_contrail_control_services():
So, whenever this is run, there is a manual intervention that is required to get things back up and running. We don’t see this, when we change the order as below:
def restore_cassandra_db():
<snip>
try:
execute(stop_cfgm)
execute(stop_collector)
execute(stop_database)
#execute(stop_collector)
execute(restore_cassandra, backup_data_path, store_db,cassandra_backup)
execute(start_cfgm)
execute(start_database)
execute(start_collector)
root@sv-31:/opt/contrail/utils# fab restore_cassandra_db
(Finished without manual intervention.)
I am in discussion with Megh Bhatt from BU on this. He is aware of this and has the relevant logs and info. This bug is filed for tracking purposes.
Analytics Team - We need to speed up the progress on this. Can I please get an update?
-Sandeep.