CentOS 7.6 rsyslogd 8.24.0-34 bug causing not processing imfile logs after logrotate

[Satish Patel]

Issue:

I have deployed CentOS 7.6 with stein and found none of component sending logs to remote syslog, and after investigation found logrotate causing

Step to reproduce issue:

1. systemctl stop rsyslog
2. rm -rf /var/spool/rsyslog/*
3. systemctl start rsyslog
4. lets wait for logrotate
5. after logrotate syslog stop forwarding logs to remote syslog host.

Impacted version of rsyslog:

rsyslogd 8.24.0-34.el7

Solution:

Upgrade rsyslog with following process:

cd /etc/yum.repos.d/
wget http://rpms.adiscon.com/v8-stable/rsyslog.repo
yum upgrade rsyslog

Notes: you have to stop rsyslog and remove all file in /var/spool/rsyslog/* and then start to make it work properly.

I would suggest we should push newer version of rsyslog in openstack-ansible package so people like me don't waste 2 week to troubleshoot this issue, or make documentation clear.

[Satish Patel (satish-txt)]

Incase if folks already deployed cloud and want to upgrade rsyslog with latest version then here is the playbook.

$cat rsyslog-upgrade.yml

---
- hosts: "{{ openstack_host_group|default('hosts') }}"
- hosts: "{{ openstack_host_group | default('hosts') }}:all_containers"
  tasks:
    - name: Install rsyslog-v8 yum repo
      template:
        src: files/rsyslog.repo
        dest: /etc/yum.repos.d/rsyslog.repo

    - name: Install rsyslog-v8 package
      yum:
        name: rsyslog
        state: latest

    - name: Stop rsyslog
      systemd:
        name: rsyslog
        state: stopped

    - name: cleann up /var/spool/rsyslog
      shell: /bin/rm -rf /var/spool/rsyslog/*

    - name: Start rsyslog
      systemd:
        name: rsyslog
        state: started

Run playbook:

openstack-ansible rsyslog-upgrade.yml --limit <group_name>

[Dmitriy Rabotyagov (noonedeadpunk)]

We have deprecated and removed support for rsyslogd in favor of logging to systemd-journald.