glance-registry process spins if rsyslog restarted with syslog logging enabled

Bug #1452312 reported by James Page
52
This bug affects 9 people
Affects Status Importance Assigned to Milestone
glance (Ubuntu)
Trusty
Undecided
Unassigned
Utopic
Undecided
Unassigned
Vivid
Undecided
Unassigned
Wily
Undecided
Unassigned
python-eventlet (Ubuntu)
High
Unassigned
Trusty
High
Unassigned
Utopic
High
Unassigned
Vivid
High
Unassigned
Wily
High
Unassigned

Bug Description

[Impact]
Glance, when configured to log to syslog, will spin consuming 100% cpu if syslog is restarted, locking all function.

[Test Case]
Install OpenStack Glance as part of a OpenStack Cloud
Configure to log to syslog
Restart syslog (sudo service rsyslog restart)

[Regression Potential]
Minimal - disconnected sockets are added to the list of types to re-raise exceptions for.

[Original Bug Report]
When configured to log to syslog, if the rsyslog process is restarted, then the glance-registry process goes into a spin, consuming 100% of CPU - strace:

poll([{fd=3, events=POLLOUT|POLLERR|POLLHUP}, {fd=5, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 2, 60000) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "<14>2015-05-06 14:58:16.186 2579"..., 112, 0, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
gettimeofday({1430924440, 430244}, NULL) = 0
poll([{fd=3, events=POLLOUT|POLLERR|POLLHUP}, {fd=5, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 2, 60000) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "<14>2015-05-06 14:58:16.186 2579"..., 112, 0, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)

This maybe related to bug 1081022 but that should have only impacted 12.04 deployments.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: glance-registry 1:2014.1.4-0ubuntu2
ProcVersionSignature: User Name 3.13.0-51.84-generic 3.13.11-ckt18
Uname: Linux 3.13.0-51-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.10
Architecture: amd64
Date: Wed May 6 15:00:00 2015
Ec2AMI: ami-00000350
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small.osci
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
PackageArchitecture: all
SourcePackage: glance
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.glance.glance.registry.conf: [deleted]
modified.conffile..etc.glance.glance.registry.paste.ini: [deleted]

Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :

Using the reproducer on bug 1081022 would indicate that this is something different, as the installation does not exhibit quite the same issue.

Revision history for this message
James Page (james-page) wrote :

Raising a task for eventlet as this appears to be related to https://gist.github.com/temoto/8259860

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in glance (Ubuntu):
status: New → Confirmed
Changed in python-eventlet (Ubuntu):
status: New → Confirmed
Revision history for this message
James Page (james-page) wrote :

I'm reticent to pull in the patch detail in #3 as this is not upstream - the codebase in the master branch is fundamentally the same as that found in eventlet in 14.04.

Changed in python-eventlet (Ubuntu):
importance: Undecided → High
Changed in glance (Ubuntu):
importance: Undecided → High
no longer affects: glance (Ubuntu)
Changed in glance (Ubuntu Trusty):
status: New → Invalid
Changed in glance (Ubuntu Utopic):
status: New → Invalid
Changed in glance (Ubuntu Vivid):
status: New → Invalid
Changed in glance (Ubuntu Wily):
status: Confirmed → Invalid
Changed in python-eventlet (Ubuntu Trusty):
status: New → Confirmed
importance: Undecided → High
Changed in python-eventlet (Ubuntu Wily):
status: Confirmed → New
Changed in python-eventlet (Ubuntu Vivid):
importance: Undecided → High
Changed in python-eventlet (Ubuntu Utopic):
importance: Undecided → High
Changed in glance (Ubuntu Wily):
importance: High → Undecided
Revision history for this message
James Page (james-page) wrote :

Also confirmed with kilo - using eventlet 0.16.0

Changed in python-eventlet (Ubuntu Vivid):
status: New → Confirmed
Changed in python-eventlet (Ubuntu Wily):
status: New → Confirmed
Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :
James Page (james-page)
Changed in python-eventlet (Ubuntu Utopic):
status: New → Confirmed
James Page (james-page)
description: updated
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello James, or anyone else affected,

Accepted python-eventlet into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.13.0-1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-eventlet (Ubuntu Trusty):
status: Confirmed → Fix Committed
tags: added: verification-needed
Changed in python-eventlet (Ubuntu Utopic):
status: Confirmed → Fix Committed
Revision history for this message
Chris J Arges (arges) wrote :

Hello James, or anyone else affected,

Accepted python-eventlet into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.13.0-1ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-eventlet (Ubuntu Vivid):
status: Confirmed → Fix Committed
Revision history for this message
Chris J Arges (arges) wrote :

Hello James, or anyone else affected,

Accepted python-eventlet into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.16.1-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Stuart McLaren (stuart-mclaren) wrote :

Note that this behaviour was originally reported here:

https://bugs.launchpad.net/oslo.log/+bug/1076466

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-eventlet - 0.17.3-2ubuntu2

---------------
python-eventlet (0.17.3-2ubuntu2) wily; urgency=medium

  * d/p/handle-ENOTCONN.patch: Ensure that ENOTCONN socket errors are
    raised correctly, resolving issues with disconnected syslog sockets
    in OpenStack Glance (LP: #1452312).

 -- James Page <email address hidden> Tue, 26 May 2015 16:45:17 +0100

Changed in python-eventlet (Ubuntu Wily):
status: Confirmed → Fix Released
Revision history for this message
George Shuklin (george-shuklin) wrote :

Hello.

I can't see update nor in trusty-updates nor in trusty-updates/juno/main. It still contains 0.13.0-1ubuntu3.2~cloud0 instead of 0.13.0-1ubuntu3.3

Revision history for this message
Edward Hope-Morley (hopem) wrote :

@george-shuklin the package will remain in trusty-proposed until it has been verified. If you want to help test this package please enable the proposed pocket in your /etc/apt/sources.list, run apt-get update and install the package. Thanks!

Revision history for this message
Seyeong Kim (seyeongkim) wrote :

0. set up keystone, glance
1. stop rsyslogd while glance-api, registry is running
2. run comamnd "glance image-list"
3. CPU usage is getting high

trusty
ii python-eventlet 0.13.0-1ubuntu2.2 all concurrent networking library for Python

utopic
ii python-eventlet 0.13.0-1ubuntu3.3 all concurrent networking library for Python

vivid
ii python-eventlet 0.16.1-0ubuntu1.1 all concurrent networking library - Python 2.x

tags: added: verification-done
removed: verification-needed
Revision history for this message
George Shuklin (george-shuklin) wrote :

@hopem, We've running on this (proposed) code for few weeks already, works fine.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-eventlet - 0.13.0-1ubuntu2.2

---------------
python-eventlet (0.13.0-1ubuntu2.2) trusty; urgency=medium

  * d/p/handle-ENOTCONN.patch: Ensure that ENOTCONN socket errors are
    raised correctly, resolving issues with disconnected syslog sockets
    in OpenStack Glance (LP: #1452312).

 -- James Page <email address hidden> Tue, 26 May 2015 17:22:24 +0100

Changed in python-eventlet (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for python-eventlet has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-eventlet - 0.13.0-1ubuntu3.3

---------------
python-eventlet (0.13.0-1ubuntu3.3) utopic; urgency=medium

  * d/p/handle-ENOTCONN.patch: Ensure that ENOTCONN socket errors are
    raised correctly, resolving issues with disconnected syslog sockets
    in OpenStack Glance (LP: #1452312).

 -- James Page <email address hidden> Tue, 26 May 2015 17:20:06 +0100

Changed in python-eventlet (Ubuntu Utopic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-eventlet - 0.16.1-0ubuntu1.1

---------------
python-eventlet (0.16.1-0ubuntu1.1) vivid; urgency=medium

  * d/p/handle-ENOTCONN.patch: Ensure that ENOTCONN socket errors are
    raised correctly, resolving issues with disconnected syslog sockets
    in OpenStack Glance (LP: #1452312).

 -- James Page <email address hidden> Tue, 26 May 2015 17:28:40 +0100

Changed in python-eventlet (Ubuntu Vivid):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers