Fuel for OpenStack

Duplicate lines in OpenStack components' log files at Fuel master node

Bug #1579081 reported by Ilya Shakhat
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Maksim Malchuk
Fuel for OpenStack 10.0
Mitaka
Fix Released
High
Maksim Malchuk
Fuel for OpenStack 9.0

Bug Description

Detailed bug description:
 In the log file located at Fuel master log lines are duplicated.

e.g.

root@node-1:~# less /var/log/nova/nova-conductor.log

  2016-05-06 13:52:16.858 5983 DEBUG oslo_messaging._drivers.amqpdriver [req-66b51b09-5942-4cd0-9b9b-273dba5a701a - - - - -] REPLY msg_id: 23c6277493c1430d8732beb0d04638ff size: 4117 reply queue: reply_9775c2f60c014ab4a60fcb00a8ca3aa1 time elapsed: 0.0500315539539s _send_reply /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:96
2016-05-06 13:52:16.976 5986 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: 41a29abd270c46d1b37ef22a500927c2 size: 1811 reply to: reply_9775c2f60c014ab4a60fcb00a8ca3aa1 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:230

[root@fuel ~]# less /var/log/remote/node-1.domain.tld/nova-conductor.log

2016-05-06T13:52:16.809210+00:00 debug: 2016-05-06 13:52:16.800 5983 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: 23c6277493c1430d8732beb0d04638ff size: 1231 reply to: reply_9775c2f60c014ab4a60fcb00a8ca3aa1 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:230
2016-05-06T13:52:16.809706+00:00 debug: 2016-05-06 13:52:16.800 5983 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: 23c6277493c1430d8732beb0d04638ff size: 1231 reply to: reply_9775c2f60c014ab4a60fcb00a8ca3aa1 __call__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:230
2016-05-06T13:52:16.862563+00:00 debug: 2016-05-06 13:52:16.858 5983 DEBUG oslo_messaging._drivers.amqpdriver [req-66b51b09-5942-4cd0-9b9b-273dba5a701a - - - - -] REPLY msg_id: 23c6277493c1430d8732beb0d04638ff size: 4117 reply queue: reply_9775c2f60c014ab4a60fcb00a8ca3aa1 time elapsed: 0.0500315539539s _send_reply /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:96
2016-05-06T13:52:16.865470+00:00 debug: 2016-05-06 13:52:16.858 5983 DEBUG oslo_messaging._drivers.amqpdriver [req-66b51b09-5942-4cd0-9b9b-273dba5a701a - - - - -] REPLY msg_id: 23c6277493c1430d8732beb0d04638ff size: 4117 reply queue: reply_9775c2f60c014ab4a60fcb00a8ca3aa1 time elapsed: 0.0500315539539s _send_reply /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:96

Steps to reproduce:
 Deploy. Look

Expected results:
 No dups

Actual result:
 There're dups

Reproducibility:
 always

Workaround:
 n/a

Impact:
 debuggability, support

Description of the environment:
 Operation system: Ubuntu 14.04
 Versions of components: MOS 9
 Reference architecture: MOS
 Network model: no matter
 Related projects installed: none

Oleksiy Molchanov (omolchanov)
tags: added: area-library
Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-agent (master)

Fix proposed to branch: master
Review: https://review.openstack.org/315692

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/315746

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-agent (master)

Reviewed: https://review.openstack.org/315692
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=9967f88e0ad8c68150c05c86af946056f87cc93d
Submitter: Jenkins
Branch: master

commit 9967f88e0ad8c68150c05c86af946056f87cc93d
Author: Maksim Malchuk <email address hidden>
Date: Tue May 10 13:34:49 2016 +0300

    Use same rsyslog config files for remote logging

    The cloud-init should use the same filenames for the rsyslog remote
    logging as used in the fuel-library templates which would be
    overwritten during the deployment by the puppet. This is partially
    fix the problem with duplicate messages sent from slave nodes to the
    Fuel master node rsyslog.

    Change-Id: Ib95db5545889e8ae2b86dcf787c4adb7413d5698
    Partial-Bug: #1579081
    Partial-Bug: #1580200
    Signed-off-by: Maksim Malchuk <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-agent (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/316299

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-agent (stable/mitaka)

Reviewed: https://review.openstack.org/316299
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=3538a7a9583fc2c0db9e4e39c7621e3bbefd2202
Submitter: Jenkins
Branch: stable/mitaka

commit 3538a7a9583fc2c0db9e4e39c7621e3bbefd2202
Author: Maksim Malchuk <email address hidden>
Date: Tue May 10 13:34:49 2016 +0300

    Use same rsyslog config files for remote logging

    The cloud-init should use the same filenames for the rsyslog remote
    logging as used in the fuel-library templates which would be
    overwritten during the deployment by the puppet. This is partially
    fix the problem with duplicate messages sent from slave nodes to the
    Fuel master node rsyslog.

    Change-Id: Ib95db5545889e8ae2b86dcf787c4adb7413d5698
    Partial-Bug: #1579081
    Partial-Bug: #1580200
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit 9967f88e0ad8c68150c05c86af946056f87cc93d)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/315746
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=50aa72b3d41c6c4be24bea03ea8ad2ba0e2fbb3e
Submitter: Jenkins
Branch: master

commit 50aa72b3d41c6c4be24bea03ea8ad2ba0e2fbb3e
Author: Maksim Malchuk <email address hidden>
Date: Thu May 12 22:12:36 2016 +0300

    Remove duplicate configurations of the rsyslog

    This commit removes the duplicate configuration settings configured
    by the incorrect usage of the rsyslog::client class for local and
    remote logging. Also, this commit adds some comments, notes and
    contains styling fixes.

    Depends-On: Ib95db5545889e8ae2b86dcf787c4adb7413d5698
    Change-Id: Iefef30379a62c8fab123e03c35c302de8e39a4a3
    Closes-Bug: #1579081
    Partial-Bug: #1580200
    Signed-off-by: Maksim Malchuk <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/316528

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/316528
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=86cf7bdddb683a5cfb1bf0f95416d396fcd3ae17
Submitter: Jenkins
Branch: stable/mitaka

commit 86cf7bdddb683a5cfb1bf0f95416d396fcd3ae17
Author: Maksim Malchuk <email address hidden>
Date: Thu May 12 22:12:36 2016 +0300

    Remove duplicate configurations of the rsyslog

    This commit removes the duplicate configuration settings configured
    by the incorrect usage of the rsyslog::client class for local and
    remote logging. Also, this commit adds some comments, notes and
    contains styling fixes.

    Depends-On: Ib95db5545889e8ae2b86dcf787c4adb7413d5698
    Change-Id: Iefef30379a62c8fab123e03c35c302de8e39a4a3
    Closes-Bug: #1579081
    Partial-Bug: #1580200
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit 50aa72b3d41c6c4be24bea03ea8ad2ba0e2fbb3e)

Revision history for this message
Dmitry Tyzhnenko (dtyzhnenko) wrote :

Verified on CI
Version 10.0-235
https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.ha_neutron/39/

Logs have no duplicates

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Dmitry Tyzhnenko (dtyzhnenko) wrote :

Verified on CI
Version 9.0-328
https://product-ci.infra.mirantis.net/job/9.0.main.ubuntu.bvt_2/328/

Logs have no duplicates

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.