rsyslogd restarts every few minutes

[Vasilios Tzanoudakis]

Fuel 5.1.1 nightly 28/10/2014
Ubuntu 3 controllers HA + GRE

Controller node 1 example

dmesg
========================================================
after each restart the system appears the following:

[ 204.060497] init: rsyslog main process (1540) killed by SEGV signal
[ 204.060521] init: rsyslog main process ended, respawning

/var/log/messages
=================================================================================
<46>Nov 5 15:00:11 node-1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="23317" x-info="http://www.rsyslog.com"] start
<46>Nov 5 15:00:11 node-1 rsyslogd: rsyslogd's groupid changed to 103
<46>Nov 5 15:00:11 node-1 rsyslogd: rsyslogd's userid changed to 101
<46>Nov 5 15:05:56 node-1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="28901" x-info="http://www.rsyslog.com"] start
<46>Nov 5 15:05:56 node-1 rsyslogd: rsyslogd's groupid changed to 103
<46>Nov 5 15:05:56 node-1 rsyslogd: rsyslogd's userid changed to 101
<46>Nov 5 15:11:05 node-1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="1540" x-info="http://www.rsyslog.com"] start
<46>Nov 5 15:11:05 node-1 rsyslogd: rsyslogd's groupid changed to 103
<46>Nov 5 15:11:05 node-1 rsyslogd: rsyslogd's userid changed to 101
<46>Nov 5 15:14:15 node-1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="14311" x-info="http://www.rsyslog.com"] start
<46>Nov 5 15:14:15 node-1 rsyslogd: rsyslogd's groupid changed to 103
<46>Nov 5 15:14:15 node-1 rsyslogd: rsyslogd's userid changed to 101

[Dmitry Borodaenko (angdraug)]

If rsyslog crashes with SEGV, this has to be addressed by MOS Linux team.

Vasilios, in case we won't be able to reproduce this locally, did you get any core dumps for the reported rsyslog crashes?

[Aleksander Mogylchenko (amogylchenko)]

We made no changes in rsyslog and should be using upstream package. Could you please attach rsyslog config here and provide the output of this command:
~>apt-cache policy rsyslog

But if you happen to find any related core dumps, please attach them as well.

[Vasilios Tzanoudakis (vtzanoudakis)]

Rsyslog Config: http://paste.openstack.org/show/129837/

latest dmesg: http://paste.openstack.org/show/129838/

root@node-1:~# apt-cache policy rsyslog
rsyslog:
  Installed: 5.8.6-1ubuntu8.6
  Candidate: 5.8.6-1ubuntu8.9
  Version table:
     5.8.6-1ubuntu8.9 0
        500 http://gr.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
 *** 5.8.6-1ubuntu8.6 0
        500 http://10.20.0.2/ubuntu/fuelweb/x86_64/ precise/main amd64 Packages
        100 /var/lib/dpkg/status
     5.8.6-1ubuntu8 0
        500 http://gr.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

unfortunately there is no core dump file created so far.

[Vasilios Tzanoudakis (vtzanoudakis)]

just added the official repo to show you that there is a difference in the version.

[Dmitry Borodaenko (angdraug)]

Changelog of the difference:

rsyslog (5.8.6-1ubuntu8.9) precise-security; urgency=medium

  * SECURITY UPDATE: denial of service and possible code execution via
    invalid PRI value
    - debian/patches/CVE-2014-3634.patch: limit PRI values in ChangeLog,
      configure.ac, configure, plugins/imfile/imfile.c,
      plugins/imklog/imklog.c, plugins/imklog/ksym.c,
      plugins/imsolaris/imsolaris.c, plugins/imtemplate/imtemplate.c,
      plugins/imuxsock/imuxsock.c, runtime/msg.c, runtime/nsd_gtls.c,
      runtime/parser.c, runtime/rsyslog.h, runtime/rule.c,
      runtime/srutils.c, runtime/syslogd-types.h, tools/syslogd.c.
    - CVE-2014-3634
    - CVE-2014-3683

 -- Marc Deslauriers <email address hidden> Thu, 02 Oct 2014 11:34:40 -0400

rsyslog (5.8.6-1ubuntu8.7) precise; urgency=medium

  * Enable non-kernel facility klog messages. (LP: #1274444)

 -- Chris J Arges <email address hidden> Tue, 19 Aug 2014 15:00:53 -0500

Whether or not this particular SEGV is related, it's worth upgrading.

[Vasilios Tzanoudakis (vtzanoudakis)]

upgrading to 5.8.6-1ubuntu8.9 solved the issue. I have upgraded all 3 controller nodes manually and everything is running smoothly now.

[Aleksander Mogylchenko (amogylchenko)]

5.8.6-1ubuntu8.9 is available via Security Ubuntu updates for 12.04.
Should be fetched when mirrors are updated.

[Vitaly Parakhin (vparakhin)]

Our internal upstream mirrors used for building 5.1.1 ISO are already frozen: http://mirrors-local-msk.msk.mirantis.net/ubuntu-5.1.1/

[Aleksander Mogylchenko (amogylchenko)]

I'm not sure when 5.1.1 mirrors were "unfrozen", since no security update got there.

Anyway, this package is already on our mirrors:
http://mirrors-local-msk.msk.mirantis.net/files/ubuntu-latest/pool/main/r/rsyslog/

And I see no sense building the same package from same sources and pushing it to 5.1.1 via Gerrit review process, instead of fetching updated package from upstream.

[Aleksander Mogylchenko (amogylchenko)]

Will be fixed in #1392877, when all security updates are merged into ISO.

[Anastasia Palkina (apalkina)]

Verified on ISO #19

"build_id": "2014-11-17_21-00-23", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "19", "auth_required": true, "api": "1.0", "nailgun_sha": "2fcab95dc43a248ba867065e96ab764ee73882d1", "production": "docker", "fuelmain_sha": "ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9", "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-11-17_21-00-23", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "19", "api": "1.0", "nailgun_sha": "2fcab95dc43a248ba867065e96ab764ee73882d1", "production": "docker", "fuelmain_sha": "ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9", "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "add3fdd3e2af57b20dbb73a6bc53a9ccc4701c9a"}}}, "fuellib_sha": "add3fdd3e2af57b20dbb73a6bc53a9ccc4701c9a"