Syslog exceed NOFILE limit

Bug #1626092 reported by David Burnazyan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Oleksiy Molchanov
Mitaka
Fix Released
High
Oleksiy Molchanov

Bug Description

Detailed bug description:
 Syslog
Steps to reproduce:
 1. Create environment with: 3 controller, 20 ceph-osd, 1000 computes
 2. Start deployment
Expected results:
 Logs from all nodes collected on Fuel node
Actual result:
 Logs collected not from all nodes. Syslog use near of 100% of CPU
Reproducibility:
 Each time
Workaround:
 Increase NOFILE limit^
 1. On fuel node create file /etc/systemd/system/rsyslog.service.d/limits.conf with content:
[Service]
LimitNOFILE=16384

  2. execute systemctl daemon-reload
Impact:
 Fuel can't get logs from all nodes
Description of the environment:
 Operation system: Ubuntu-14
 Versions of components: Fuel 9.0, Mitaka
 Reference architecture: 3 controller, 20 ceph-osd, 1000 computes
 Network model: VXLAN
 Related projects installed: none

description: updated
tags: added: area-library
no longer affects: fuel/newton
tags: added: scale
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Maksim Malchuk (mmalchuk)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/374253

Changed in fuel:
assignee: Maksim Malchuk (mmalchuk) → Oleksiy Molchanov (omolchanov)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/374253
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=eee7623705708123668c514c9794467b54c8bfc8
Submitter: Jenkins
Branch: master

commit eee7623705708123668c514c9794467b54c8bfc8
Author: Oleksiy Molchanov <email address hidden>
Date: Wed Sep 21 18:43:27 2016 +0300

    Increase MaxOpenFiles to 16384 for syslog on master by default

    Change-Id: If730b41b902f4c84c509bc3d98a0955732162fc0
    Closes-Bug: 1626092

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/384895

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 10.0.0rc1

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/384895
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=8779d8eddeabc360f78c215f697558cbf2e698ce
Submitter: Jenkins
Branch: stable/mitaka

commit 8779d8eddeabc360f78c215f697558cbf2e698ce
Author: Oleksiy Molchanov <email address hidden>
Date: Wed Sep 21 18:43:27 2016 +0300

    Increase MaxOpenFiles to 16384 for syslog on master by default

    Change-Id: If730b41b902f4c84c509bc3d98a0955732162fc0
    Closes-Bug: 1626092
    (cherry picked from commit eee7623705708123668c514c9794467b54c8bfc8)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 10.0.0

This issue was fixed in the openstack/fuel-library 10.0.0 release.

Revision history for this message
Michael Semenov (msemenov) wrote :

9.2 is not certified with more than 200 nodes. So, set as Fix Released in 9.2.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.