glance-registry process spins if rsyslog restarted with syslog logging enabled

[James Page]

[Impact]
Glance, when configured to log to syslog, will spin consuming 100% cpu if syslog is restarted, locking all function.

[Test Case]
Install OpenStack Glance as part of a OpenStack Cloud
Configure to log to syslog
Restart syslog (sudo service rsyslog restart)

[Regression Potential]
Minimal - disconnected sockets are added to the list of types to re-raise exceptions for.

[Original Bug Report]
When configured to log to syslog, if the rsyslog process is restarted, then the glance-registry process goes into a spin, consuming 100% of CPU - strace:

poll([{fd=3, events=POLLOUT|POLLERR|POLLHUP}, {fd=5, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 2, 60000) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "<14>2015-05-06 14:58:16.186 2579"..., 112, 0, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
gettimeofday({1430924440, 430244}, NULL) = 0
poll([{fd=3, events=POLLOUT|POLLERR|POLLHUP}, {fd=5, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 2, 60000) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "<14>2015-05-06 14:58:16.186 2579"..., 112, 0, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)

This maybe related to bug 1081022 but that should have only impacted 12.04 deployments.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: glance-registry 1:2014.1.4-0ubuntu2
ProcVersionSignature: User Name 3.13.0-51.84-generic 3.13.11-ckt18
Uname: Linux 3.13.0-51-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.10
Architecture: amd64
Date: Wed May 6 15:00:00 2015
Ec2AMI: ami-00000350
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small.osci
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
PackageArchitecture: all
SourcePackage: glance
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.glance.glance.registry.conf: [deleted]
modified.conffile..etc.glance.glance.registry.paste.ini: [deleted]

[James Page (james-page)]

Using the reproducer on bug 1081022 would indicate that this is something different, as the installation does not exhibit quite the same issue.

[James Page (james-page)]

Raising a task for eventlet as this appears to be related to https://gist.github.com/temoto/8259860

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Page (james-page)]

I'm reticent to pull in the patch detail in #3 as this is not upstream - the codebase in the master branch is fundamentally the same as that found in eventlet in 14.04.

[James Page (james-page)]

Also confirmed with kilo - using eventlet 0.16.0

[James Page (james-page)]

https://github.com/eventlet/eventlet/issues/192

[James Page (james-page)]

Potential fix:

https://github.com/eventlet/eventlet/commit/ed0655333007fe97624b745e92deb407d192022c

[Chris J Arges (arges)]

Hello James, or anyone else affected,

Accepted python-eventlet into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.13.0-1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello James, or anyone else affected,

Accepted python-eventlet into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.13.0-1ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello James, or anyone else affected,

Accepted python-eventlet into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.16.1-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Stuart McLaren (stuart-mclaren)]

Note that this behaviour was originally reported here:

https://bugs.launchpad.net/oslo.log/+bug/1076466

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-eventlet - 0.17.3-2ubuntu2

---------------
python-eventlet (0.17.3-2ubuntu2) wily; urgency=medium

  * d/p/handle-ENOTCONN.patch: Ensure that ENOTCONN socket errors are
    raised correctly, resolving issues with disconnected syslog sockets
    in OpenStack Glance (LP: #1452312).

 -- James Page <email address hidden> Tue, 26 May 2015 16:45:17 +0100

[George Shuklin (george-shuklin)]

Hello.

I can't see update nor in trusty-updates nor in trusty-updates/juno/main. It still contains 0.13.0-1ubuntu3.2~cloud0 instead of 0.13.0-1ubuntu3.3

[Edward Hope-Morley (hopem)]

@george-shuklin the package will remain in trusty-proposed until it has been verified. If you want to help test this package please enable the proposed pocket in your /etc/apt/sources.list, run apt-get update and install the package. Thanks!

[Seyeong Kim (xtrusia)]

0. set up keystone, glance
1. stop rsyslogd while glance-api, registry is running
2. run comamnd "glance image-list"
3. CPU usage is getting high

trusty
ii python-eventlet 0.13.0-1ubuntu2.2 all concurrent networking library for Python

utopic
ii python-eventlet 0.13.0-1ubuntu3.3 all concurrent networking library for Python

vivid
ii python-eventlet 0.16.1-0ubuntu1.1 all concurrent networking library - Python 2.x

[George Shuklin (george-shuklin)]

@hopem, We've running on this (proposed) code for few weeks already, works fine.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-eventlet - 0.13.0-1ubuntu2.2

---------------
python-eventlet (0.13.0-1ubuntu2.2) trusty; urgency=medium

  * d/p/handle-ENOTCONN.patch: Ensure that ENOTCONN socket errors are
    raised correctly, resolving issues with disconnected syslog sockets
    in OpenStack Glance (LP: #1452312).

 -- James Page <email address hidden> Tue, 26 May 2015 17:22:24 +0100

[Chris J Arges (arges)]

The verification of the Stable Release Update for python-eventlet has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-eventlet - 0.13.0-1ubuntu3.3

---------------
python-eventlet (0.13.0-1ubuntu3.3) utopic; urgency=medium

  * d/p/handle-ENOTCONN.patch: Ensure that ENOTCONN socket errors are
    raised correctly, resolving issues with disconnected syslog sockets
    in OpenStack Glance (LP: #1452312).

 -- James Page <email address hidden> Tue, 26 May 2015 17:20:06 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-eventlet - 0.16.1-0ubuntu1.1

---------------
python-eventlet (0.16.1-0ubuntu1.1) vivid; urgency=medium

  * d/p/handle-ENOTCONN.patch: Ensure that ENOTCONN socket errors are
    raised correctly, resolving issues with disconnected syslog sockets
    in OpenStack Glance (LP: #1452312).

 -- James Page <email address hidden> Tue, 26 May 2015 17:28:40 +0100