Failover replication mode is broken

[Paul Goins]

As currently implemented, failover replication mode does not work.

This apparently is related to the $ActionQueueType parameter. In commit fed1684ce5b0df4845a82026507f5540bc0fb33a it was changed to LinkedList. This generally is considered to be advisable per docs. However, in a blog post by one of the rsyslog developers (https://rainer.gerhards.net/2011/03/using-failover-and-asynchornous-actions.html), it's noted that this apparently causes things to be processed asynchronously in a way which breaks their normal recipe for failover (https://rsyslog.readthedocs.io/en/latest/tutorials/failover_syslog_server.html).

Setting $ActionQueueType to Direct, while less performant (maybe significantly - not tested), allows this to work. $ActionQueueType's default value appears to be Direct as well, so an alternative is to remove that directive.

I'm curious if there is a better but still compatible way to fix this? But if not (or in the meantime), we may want to make the above change in order to fix the "failover" replication mode.

[Diko Parvanov (dparv)]

In general what I can see is most of the configuration we have in the charm is outdated and depricated; the way we should handle failover is quite different than what we have (it was used for version 7.x, current release is 8.x)

rsyslogd: error during parsing file /etc/rsyslog.d/40-rsyslog-imfile.conf, on or before line 8: parameter 'statefile' deprecated but accepted, consider removing or replacing it [v8.16.0 try http://www.rsyslog.co
m/e/2207 ]
rsyslogd: error during parsing file /etc/rsyslog.d/40-rsyslog-imfile.conf, on or before line 13: parameter 'statefile' deprecated but accepted, consider removing or replacing it [v8.16.0 try http://www.rsyslog.c
om/e/2207 ]
rsyslogd: invalid number [v8.16.0 try http://www.rsyslog.com/e/2010 ]

we currently follow https://www.rsyslog.com/doc/v7-stable/tutorials/failover_syslog_server.html

$ActionQueueDiscardSeverity info
*.* @@SERVER:PORT

$ActionExecOnlyWhenPreviousIsSuspended on

& @@SERVER:PORT

<etc>

& /var/log/localbuffer
$ActionExecOnlyWhenPreviousIsSuspended off

where we should be following: https://www.rsyslog.com/doc/v8-stable/tutorials/failover_syslog_server.html

ii rsyslog 8.16.0-1ubuntu3.1 amd64 reliable system and kernel logging daemon

if($msg contains "info") then {
     action(type="omfwd" target="SERVER" port="XXX" protocol="udp")
     action(type="omfwd" target="SERVER" port="PORT" action.execOnlyWhenPreviousIsSuspended="on")
     action(type="omfwd" target="SERVER" port="PORT" action.execOnlyWhenPreviousIsSuspended="on")
     action(type="omfile" tag="failover" file="/var/log/localbuffer" action.execOnlyWhenPreviousIsSuspended="on")
}

[Eric Chen (eric-chen)]

This charm is no longer being actively maintained. Please consider using the new Canonical Observability Stack instead.
(https://charmhub.io/topics/canonical-observability-stack)