Log message about fencing rabbit node from cluster is missed

[Andrey Sledzinskiy]

fuel version - 9.0-mos-427

Steps:
1. Deploy cluster with default configuration, 3 controllers, 2 compute, 1 cinder
2. SSH to first controller and put corosync cluster to maintenance mode:
   crm configure property maintenance-mode=true
3. Stop rabbit and corosync service on first controller
4. Check on master node that rabbit-fence.log contains
'grep -P 'Forgetting cluster node rabbit@node-1'

Expected - message is present in log

Actual result - log contain next error:

2016-06-02T00:52:15.920052+00:00 info: 2016-06-02 00:52:15,923 INFO Starting rabbit fence script main loop
2016-06-02T03:16:09.791401+00:00 debug: 2016-06-02 03:16:09,788 DEBUG Command hiera node_name_prefix_for_messaging
2016-06-02T03:16:09.792463+00:00 debug: 2016-06-02 03:16:09,792 DEBUG Stderr: /usr/lib/ruby/vendor_ruby/hiera/filecache.rb:52:in `read': Permission denied - /etc/hiera/deleted_nodes.yaml (Errno::EACCES)
 from /usr/lib/ruby/vendor_ruby/hiera/filecache.rb:52:in `read_file'
 from /usr/lib/ruby/vendor_ruby/hiera/backend/yaml_backend.rb:22:in `block in lookup'
 from /usr/lib/ruby/vendor_ruby/hiera/backend.rb:73:in `block in datasources'
 from /usr/lib/ruby/vendor_ruby/hiera/backend.rb:71:in `map'
 from /usr/lib/ruby/vendor_ruby/hiera/backend.rb:71:in `datasources'
 from /usr/lib/ruby/vendor_ruby/hiera/backend/yaml_backend.rb:16:in `lookup'
 from /usr/lib/ruby/vendor_ruby/hiera/backend.rb:174:in `block in lookup'
 from /usr/lib/ruby/vendor_ruby/hiera/backend.rb:171:in `each'
 from /usr/lib/ruby/vendor_ruby/hiera/backend.rb:171:in `lookup'
 from /usr/lib/ruby/vendor_ruby/hiera.rb:60:in `lookup'
 from /usr/bin/hiera:221:in `<main>'
2016-06-02T03:16:09.809679+00:00 info: 2016-06-02 03:16:09,794 INFO Got messaging-node-1.test.domain.local that left cluster

[Maksim Malchuk (mmalchuk)]

This is not a High bug, because not affected deploy.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/337852

[Maksim Malchuk (mmalchuk)]

JFYI, the message is present in the log, steps should be fixed:
[root@fuel ~]# cat /var/log/remote/node-1.domain.tld/rabbit-fence.log
2016-07-06T08:25:05.671045+00:00 info: 2016-07-06 08:25:05,671 INFO Starting rabbit fence script main loop
[root@fuel ~]# cat /var/log/remote/node-2.domain.tld/rabbit-fence.log
2016-07-06T08:29:41.542098+00:00 info: 2016-07-06 08:29:41,541 INFO Starting rabbit fence script main loop
2016-07-06T10:20:28.423097+00:00 debug: 2016-07-06 10:20:28,343 DEBUG Command hiera node_name_prefix_for_messaging
2016-07-06T10:20:28.471356+00:00 debug: 2016-07-06 10:20:28,477 DEBUG Stdout: messaging-
2016-07-06T10:20:28.600910+00:00 info: 2016-07-06 10:20:28,511 INFO Got messaging-node-1.domain.tld that left cluster
2016-07-06T10:20:28.603017+00:00 debug: 2016-07-06 10:20:28,607 DEBUG {'message': <dbus.lowlevel.SignalMessage path: /org/corosync, iface: org.corosync, member: NodeStateChange, dest: (broadcast)>, 'type': 'NodeStateChange'}
2016-07-06T10:20:28.603017+00:00 debug: 2016-07-06 10:20:28,609 DEBUG messaging-node-1.domain.tld
2016-07-06T10:20:28.603017+00:00 debug: 2016-07-06 10:20:28,609 DEBUG 1
2016-07-06T10:20:28.603017+00:00 debug: 2016-07-06 10:20:28,609 DEBUG 192.168.0.4
2016-07-06T10:20:28.603017+00:00 debug: 2016-07-06 10:20:28,609 DEBUG left
2016-07-06T10:20:28.603017+00:00 info: 2016-07-06 10:20:28,609 INFO Preparing to fence node rabbit@messaging-node-1 from rabbit cluster
2016-07-06T10:20:29.730122+00:00 debug: 2016-07-06 10:20:29,736 DEBUG Command rabbitmqctl eval "mnesia:system_info(running_db_nodes)."| grep -o rabbit@messaging-node-1
2016-07-06T10:20:30.318220+00:00 debug: 2016-07-06 10:20:30,322 DEBUG Command rabbitmqctl eval "mnesia:system_info(db_nodes)."| grep -o rabbit@messaging-node-1
2016-07-06T10:20:30.324712+00:00 debug: 2016-07-06 10:20:30,330 DEBUG Stdout: rabbit@messaging-node-1
2016-07-06T10:20:30.328272+00:00 info: 2016-07-06 10:20:30,334 INFO Disconnecting node rabbit@messaging-node-1
2016-07-06T10:20:31.036736+00:00 debug: 2016-07-06 10:20:31,042 DEBUG Command rabbitmqctl eval "disconnect_node(list_to_atom(\"rabbit@messaging-node-1\"))."
2016-07-06T10:20:31.038797+00:00 debug: 2016-07-06 10:20:31,045 DEBUG Stdout: false
2016-07-06T10:20:31.045717+00:00 info: 2016-07-06 10:20:31,052 INFO Forgetting cluster node rabbit@messaging-node-1
2016-07-06T10:20:32.214535+00:00 debug: 2016-07-06 10:20:32,220 DEBUG Command rabbitmqctl forget_cluster_node rabbit@messaging-node-1
2016-07-06T10:20:32.215189+00:00 debug: 2016-07-06 10:20:32,221 DEBUG Stdout: Removing node 'rabbit@messaging-node-1' from cluster ...

the log from the first node doesn't contain the messages at all because of stopped corosync.

the proposed commit will fix the hiera issue as you can see above.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/337852
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=5bf3ab609dbf2d8a8c456a6342c3ea6f4f9a6bbc
Submitter: Jenkins
Branch: master

commit 5bf3ab609dbf2d8a8c456a6342c3ea6f4f9a6bbc
Author: Maksim Malchuk <email address hidden>
Date: Tue Jul 5 23:21:26 2016 +0300

    Fix permissions for hiera files

    All hiera files should have world-readable permissions.

    Change-Id: I19afc6dcd9586da68918cda8ea8898acd0635ad9
    Closes-Bug: #1599460
    Closes-Bug: #1588434
    Signed-off-by: Maksim Malchuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0 release.