9.2 stops collect logs after some time
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Invalid
|
High
|
Fuel Sustaining | ||
Mitaka |
Invalid
|
High
|
Fuel Sustaining | ||
Newton |
Invalid
|
High
|
Fuel Sustaining |
Bug Description
Steps to reproduce:
1. Install 9.0
2. Upgrade to 9.2 from http://
3. Start deploying cluster (~300 nodes in my case)
After some times logs in /var/log/remote are not updated.
On example, one of controller - node-2009
[root@fuel remote]# date
Wed Jan 25 08:56:38 UTC 2017
[root@fuel remote]# tail -n1 node-2009.
2017-01-
[root@fuel remote]# ssh node-2009.
Warning: Permanently added 'node-2009.
<4>Jan 25 03:18:07 node-2009 kernel: [44816.762676] NOHZ: local_softirq_
all controllers does not have puppet-apply.log in /var/log/remote
Another example - random compute node
[root@fuel remote]# date
Wed Jan 25 09:02:46 UTC 2017
[root@fuel remote]# tail -n1 node-2228.
2017-01-
[root@fuel remote]# ssh node-2228.
Warning: Permanently added 'node-2228.
<4>Jan 25 06:27:02 node-2228 kernel: [52101.701376] NOHZ: local_softirq_
/var/log has a lot of free space
/dev/mapper/
rsyslogd in the top of load every time and load 90%-120% CPUs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25707 root 20 0 950796 70852 31100 S 97.3 0.1 2934:18 rsyslogd
Changed in fuel: | |
milestone: | none → 10.1 |
milestone: | 10.1 → 9.3 |
assignee: | nobody → Fuel Sustaining (fuel-sustaining-team) |
status: | New → Confirmed |
importance: | Undecided → High |
tags: | added: area-library |
it seems that we have rsyslog not accepting logs from the nodes due to it getting into high CPU load. we need to investigate this behaviour deeper with some SMEs