[SRU] python-eventlet (0.9.16-1ubuntu4) on 12.04 raising AssertionError exceptions.

[Adam Gandelman]

python-eventlet (0.9.16-1ubuntu2) introduced a cherry-picked upstream patch that fixed a memory leak manifested by Nova's use of the library. Since that time, the various nova services often log AssertionError exceptions, ie:

2012-07-01 17:59:27 DEBUG nova.utils [-] Got file lock "iptables" for method "apply"... from (pid=23798) inner /usr/lib/python2.7/dist-packages/nova/utils.py:942
2012-07-01 17:59:27 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap iptables-save -t filter from (pid=23798) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
2012-07-01 17:59:27 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap iptables-restore from (pid=23798) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
2012-07-01 17:59:27 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap iptables-save -t nat from (pid=23798) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
2012-07-01 17:59:28 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap iptables-restore from (pid=23798) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
2012-07-01 17:59:28 DEBUG nova.network.linux_net [-] IPTablesManager.apply completed with success from (pid=23798) apply /usr/lib/python2.7/dist-packages/nova/network/linux_net.py:335

As was pointed out on a recent mailing list post [1], this is due to two missing patches that were subsequently applied upstream to avoid the exceptions.

The issue has been fixed in Debian unstable as of 0.9.16-2 (a more complete patch applied), and in Ubuntu as of the 0.9.16-2ubuntu1 merge.

We should consider SRU'ing a fix for this as it is unsettling for users deploying Nova into production, and annoying for upstream developers (12.04 is the current target platform for the current Nova dev cycle).

[1] http://<email address hidden>/msg13739.html

[Adam Gandelman (gandelman-a)]

*** Impact ***

The incomplete patch set applied to python-eventlet causes nova to frequently log AssertionError exceptions. 12.04 is the development platform for the current development release of Openstack, and will likely continue to be for a number of future releases. For developers, these messages pollutes debug logs, test suite runs, etc.

*** Development Fix ***

The issue has been fixed upstream since March, in Debian since March, and in Ubuntu 12.10 since May.

*** Test Case ***

sudo apt-get -y install nova-api
sudo service nova-api stop
sudo nova-api

Observe numerous "Exception AssertionError: AssertionError()..." messages logged to stderr during the lifetime of the process, as well as 'KeyError' exception on process exit.

*** Regression Potential ***

Minimal

[Adam Conrad (adconrad)]

Hello Adam, or anyone else affected,

Accepted python-eventlet into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-eventlet/1.4-0ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Qiu Yu (unicell)]

Just verified with Nova latest source in Precise, "AssertionError exceptions" is fixed with this 0.9.16-1ubuntu4.1

Steps to verify:
1. enable proposed in Ubuntu 12.04
2. verify package version using 'python -c "import eventlet; print eventlet.version_info'
3. apt-get install python-eventlet=0.9.16-1ubuntu4.1
4. start nova-api and no more AssertionError exceptions
5. downgrade to 0.9.16-1ubuntu4, and this exception happens again.

Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-eventlet - 0.9.16-1ubuntu4.1

---------------
python-eventlet (0.9.16-1ubuntu4.1) precise-proposed; urgency=low

  * debian/patches/assertionerror-exception-fix.patch: Patch
    additonal symbols missed by eventlet-leak.patch, silence AssertionError
    exceptions. (LP: #1019904)
 -- Adam Gandelman <email address hidden> Sun, 01 Jul 2012 18:21:02 -0700