Logging format for Pacemaker components isn't consistent

[Simon Pasquier]

This has been noticed with recent ISO builds of Fuel.

The messages logged in /var/log/pacemaker.log aren't formatted consistently:
<29>Apr 29 13:23:46 node-32 corosync[13486]: [TOTEM ] memb_state_operational_enter A new membership (192.168.0.6:8) was formed. Members joined: 1
<29>Apr 29 13:23:46 node-32 corosync[13486]: [MAIN ] corosync_sync_completed Completed service synchronization, ready to provide service.
<29>Apr 29 13:23:46 node-32 corosync[13486]: [QUORUM] quorum_api_set_quorum This node is within the primary component and will provide service.
<29>Apr 29 13:23:46 node-32 corosync[13486]: [QUORUM] log_view_list Members[2]: 1 3
Set r/w permissions for uid=108, gid=115 on /var/log/pacemaker.log
Apr 29 13:23:46 [13545] node-32.domain.tld pacemakerd: info: crm_log_init: Changed active directory to /var/lib/heartbeat/cores/root
Apr 29 13:23:46 [13545] node-32.domain.tld pacemakerd: info: crm_ipc_connect: Could not establish pacemakerd connection: Connection refused (111)
Apr 29 13:23:46 [13545] node-32.domain.tld pacemakerd: info: get_cluster_type: Detected an active 'corosync' cluster
Apr 29 13:23:46 [13545] node-32.domain.tld pacemakerd: info: mcp_read_config: Reading configure for stack: corosync
Apr 29 13:23:46 [13545] node-32.domain.tld pacemakerd: notice: mcp_read_config: Configured corosync to accept connections from group 115: OK (1)
Apr 29 13:23:46 [13545] node-32.domain.tld pacemakerd: notice: main: Starting Pacemaker 1.1.12 (Build: 561c4cf): generated-manpages agent-manpages ncurses libqb-logging libqb-ipc lha-fencing nagios heartbeat corosync-native snmp libesmtp acls
Apr 29 13:23:46 [13545] node-32.domain.tld pacemakerd: info: main: Maximum core file size is: 18446744073709551615
Apr 29 13:23:46 [13545] node-32.domain.tld pacemakerd: info: qb_ipcs_us_publish: server name: pacemakerd

This snippet show that some lines are prefixed by the Syslog priority and others aren't.

Diving into the Pacemaker/Rsyslog configurations, it seems that important messages (eg NOTICE level and above) are sent to Syslog while other messages (eg INFO and below) are sent to the debug file [1].

The end result is that parsing logs is harder than expected and messages without priority cannot be classifier by severity.

[1] http://blog.clusterlabs.org/blog/2013/pacemaker-logging/

[Simon Pasquier (simon-pasquier)]

I updated the following parameters in /etc/sysconfig/pacemaker (in a CentOS environment):

PCMK_logfile=/var/log/pacemaker_debug.log
PCMK_logpriority=notice

I don't why but it didn't change anything, all log messages ended up in /var/log/pacemaker.log. I also removed the /etc/rsyslog.d/02-ha.conf file and the NOTICE and higher message logs ended up in /var/log/daemon.log as expected.

I will investigate further why PCMK_logfile is ineffective.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/186343

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/186343
Committed: https://git.openstack.org/cgit/stackforge/fuel-plugin-lma-collector/commit/?id=d4307937125f8e2f8694df37af22d1d13016da2a
Submitter: Jenkins
Branch: master

commit d4307937125f8e2f8694df37af22d1d13016da2a
Author: Simon Pasquier <email address hidden>
Date: Thu May 28 12:05:40 2015 +0200

    Fix parsing of Pacemaker logs

    This changes fixes the parsing of the Pacemaker logs. Pacemaker logs
    via Syslog (for NOTICE and higher messages) and directly to a file for
    other messages. Unfortunately since all messages end up in the same
    /var/log/pacemaker.log, we need a special decoder that can handle both
    formats.

    Change-Id: I7389b4fd17d8f5e1b14bc17a2edeece579e1f4e7
    Related-Bug: #1450045

[Bogdan Dobrelya (bogdando)]

Fixed in the scope of LMA plugin

[Nastya Urlapova (aurlapova)]

It was fixed in plugin, moved to Won't Fix for fuel project.