api servers hang with 100% CPU if syslog restarted
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Glance |
Invalid
|
Undecided
|
Unassigned | ||
OpenStack Compute (nova) |
Invalid
|
Undecided
|
Unassigned | ||
neutron |
Invalid
|
Undecided
|
Unassigned | ||
oslo.log |
Invalid
|
Undecided
|
Unassigned | ||
python-eventlet (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
Affected:
glance-api
glance-registry
neutron-server
nova-api
If service was configured to use rsyslog and rsyslog was restarted after API server started, it hangs on next log line with 100% CPU. If server have few workers, each worker will eat own 100% CPU share.
Steps to reproduce:
1. Configure syslog:
use_syslog=true
syslog_
2. restart api service
3. restart rsyslog
Execute some command to force logging. F.e.: neutron net-create foo, nova boot, etc.
Expected result: normal operation
Actual result:
with some chance (about 30-50%) api server will hung with 100% CPU usage and will not reply to request.
Strace on hung service:
gettimeofday(
poll([{fd=3, events=
sendto(3, "<151>keystonem
gettimeofday(
poll([{fd=3, events=
sendto(3, "<151>keystonem
gettimeofday(
Tested on:
nova, glance, neutron: 1:2014.2.3, Ubuntu version.
description: | updated |
tags: | added: ops |
Changed in glance: | |
status: | New → Invalid |
Changed in nova: | |
status: | New → Invalid |
Changed in neutron: | |
status: | New → Invalid |
Changed in oslo.log: | |
status: | New → Invalid |
It sounds similar with: /bugs.launchpad .net/ubuntu/ +source/ python- eventlet/ +bug/1452312 /github. com/eventlet/ eventlet/ issues/ 192
https:/
https:/