python unit test jobs failing due to subunit log being too big

[Matt Riedemann]

http://logs.openstack.org/60/144760/1/check/gate-nova-python26/6eb86b3/console.html#_2015-01-05_10_20_01_178

2015-01-05 10:20:01.178 | + [[ 72860 -gt 50000 ]]
2015-01-05 10:20:01.178 | + echo
2015-01-05 10:20:01.178 |
2015-01-05 10:20:01.178 | + echo 'sub_unit.log was > 50 MB of uncompressed data!!!'
2015-01-05 10:20:01.178 | sub_unit.log was > 50 MB of uncompressed data!!!
2015-01-05 10:20:01.179 | + echo 'Something is causing tests for this project to log significant amounts'
2015-01-05 10:20:01.179 | Something is causing tests for this project to log significant amounts
2015-01-05 10:20:01.179 | + echo 'of data. This may be writers to python logging, stdout, or stderr.'
2015-01-05 10:20:01.179 | of data. This may be writers to python logging, stdout, or stderr.
2015-01-05 10:20:01.179 | + echo 'Failing this test as a result'
2015-01-05 10:20:01.179 | Failing this test as a result
2015-01-05 10:20:01.179 | + echo

Looks like the subunit log is around 73 MB, this could be due to the new pip because I'm seeing a ton of these:

"DeprecationWarning: `require` parameter is deprecated. Use EntryPoint._load instead."

The latest pip was released on 1/3/15:

https://pypi.python.org/pypi/pip/6.0.6

That's also when those warnings showed up:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRGVwcmVjYXRpb25XYXJuaW5nOiBgcmVxdWlyZWAgcGFyYW1ldGVyIGlzIGRlcHJlY2F0ZWQuIFVzZSBFbnRyeVBvaW50Ll9sb2FkIGluc3RlYWQuXCIgQU5EIHRhZ3M6XCJjb25zb2xlXCIgYW5kIHByb2plY3Q6XCJvcGVuc3RhY2svbm92YVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDIwNDc2OTk3NTI3fQ==

[Matt Riedemann (mriedem)]

upstream pip bug: https://github.com/pypa/pip/issues/2326

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/145026

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/145037

[Matt Riedemann (mriedem)]

This should be fixed by the 1.2.0 release of stevedore:

https://pypi.python.org/pypi/stevedore/1.2.0

Which has this fix:

https://review.openstack.org/#/c/145042/

for bug 1407778.

I'm going to recheck the stable/juno nova change which should run with the latest stevedore and if it passes we can just close this as a duplicate of bug 1407778.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/145026

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/145037

[Matt Riedemann (mriedem)]

We still have a problem here because the paste library is hitting the same deprecation warnings, so re-opening this bug.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/145611

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/145611
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b718b52feba694bfe832021dbfbb2f8de5bffaab
Submitter: Jenkins
Branch: master

commit b718b52feba694bfe832021dbfbb2f8de5bffaab
Author: Matt Riedemann <email address hidden>
Date: Wed Jan 7 14:10:01 2015 -0800

    Add WarningsFixture to only emit DeprecationWarning once in a test run

    The nova unit test console output can be flooded with deprecation
    warnings from dependent libraries, this change makes it so they are only
    logged once.

    It's worth noting that Keystone has an alternative implementation where
    they filter deprecation warnings and only error out if they are coming
    from Keystone code. See commit 9ae6ffe8a.

    Partial-Bug: #1407736

    Co-authored-by: Sean Dague <email address hidden>

    Change-Id: Id8d8866baaf64721fda2b6b2e8358db18920c8ba

[Matt Riedemann (mriedem)]

Something merged in the last 24 hours that really made this drop off sharply in stable/juno:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRGVwcmVjYXRpb25XYXJuaW5nOiBQYXJhbWV0ZXJzIHRvIGxvYWQgYXJlIGRlcHJlY2F0ZWQuICBDYWxsIC5yZXNvbHZlIGFuZCAucmVxdWlyZSBzZXBhcmF0ZWx5LlwiIEFORCB0YWdzOlwiY29uc29sZVwiIEFORCBidWlsZF9uYW1lOlwiZ2F0ZS1ub3ZhLXB5dGhvbjI3XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MjA3MzE5Njc3MjN9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/145832

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/145927

[Matt Riedemann (mriedem)]

bknudson pointed out the real issue, sqlalchemy-migrate is always logging deprecation warnings, that's why moving the deprecation warnings fixture in nova to after the db fixture fixed the problem for nova:

https://github.com/stackforge/sqlalchemy-migrate/blob/master/migrate/changeset/__init__.py#L13

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/145938

[Brent Eagles (beagles)]

Brant,

I posted a patch *ages* ago that hadn't been merged that I think addresses this issue.

https://review.openstack.org/#/c/99657/

[Brent Eagles (beagles)]

Should've also mentioned that there was a related bug:
https://bugs.launchpad.net/oslo.db/+bug/1328647

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brant Knudson (<email address hidden>) on branch: master
Review: https://review.openstack.org/145938
Reason: See https://review.openstack.org/#/c/99657/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/145938
Committed: https://git.openstack.org/cgit/stackforge/sqlalchemy-migrate/commit/?id=244c6c55d7aaa1f99e56027f4f5151fca51e7377
Submitter: Jenkins
Branch: master

commit 244c6c55d7aaa1f99e56027f4f5151fca51e7377
Author: Brant Knudson <email address hidden>
Date: Thu Jan 8 15:38:02 2015 -0600

    Don't add warnings filter on import

    The changeset module was adding a warnings filter on import. This
    affects all applications that wind up importing it. A library
    shouldn't modify the warnings filters unless asked.

    Closes-Bug: #1407736
    Change-Id: I893f8be48efd3d3642e977ab587c9e6dc867258b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/145927
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bcaff21f07763e9dbd859ea81f72cbca83a6e113
Submitter: Jenkins
Branch: master

commit bcaff21f07763e9dbd859ea81f72cbca83a6e113
Author: Brant Knudson <email address hidden>
Date: Thu Jan 8 14:39:45 2015 -0600

    Move WarningsFixture after DatabaseFixture so emit once

    The DatabaseFixture must be messing with the warnings module.

    Change-Id: Ic8b22569a4a8fb05b4d875e34432ada97a52f10c
    Partial-Bug: #1407736

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/147490

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/147491

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/147490
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9ee1a0f5756eec1c4ba42bd823b61b09c58e23fa
Submitter: Jenkins
Branch: stable/juno

commit 9ee1a0f5756eec1c4ba42bd823b61b09c58e23fa
Author: Sean Dague <email address hidden>
Date: Thu Jan 15 07:29:41 2015 -0500

    only emit deprecation warnings once

    This is a simplified version of what's in master (where all of test.py
    was broken up into fixtures, so backporting directly is way too
    complicated).

    The python warnings module allows you to dial down warnings by
    type. This makes a particular warning only emit once per python run,
    not on every function call.

    Before sqlalchemy-migration >= 0.9.4, migrate reset the warnings
    filters everytime migrations are run, so this has to run after any
    database code in the setup (otherwise you could make this the first
    like of setUp()).

    Change-In-Master: b718b52feba694bfe832021dbfbb2f8de5bffaab

    Change-Id: I54cb62b64dfd28c4c0981db6a29619e9f7ced488
    Partial-Bug: #1407736

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/147491
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=54f9225a3d6d3b4196ece7700809def3e38b9482
Submitter: Jenkins
Branch: stable/icehouse

commit 54f9225a3d6d3b4196ece7700809def3e38b9482
Author: Sean Dague <email address hidden>
Date: Thu Jan 15 07:29:41 2015 -0500

    only emit deprecation warnings once

    This is a simplified version of what's in master (where all of test.py
    was broken up into fixtures, so backporting directly is way too
    complicated).

    The python warnings module allows you to dial down warnings by
    type. This makes a particular warning only emit once per python run,
    not on every function call.

    Before sqlalchemy-migration >= 0.9.4, migrate reset the warnings
    filters everytime migrations are run, so this has to run after any
    database code in the setup (otherwise you could make this the first
    like of setUp()).

    Change-In-Master: b718b52feba694bfe832021dbfbb2f8de5bffaab

    Change-Id: I54cb62b64dfd28c4c0981db6a29619e9f7ced488
    Partial-Bug: #1407736
    (cherry picked from commit 9ee1a0f5756eec1c4ba42bd823b61b09c58e23fa)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/145026

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/145832
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c3d10b7a62ad2032cde401f91e764c59d2e11b44
Submitter: Jenkins
Branch: master

commit c3d10b7a62ad2032cde401f91e764c59d2e11b44
Author: Davanum Srinivas <email address hidden>
Date: Thu Jan 8 11:38:28 2015 -0500

    Do not use deprecated assertRaisesRegexp()

    The unit test log ends up with DeprecationWarning(s) from the outdated
    calls to assertRaisesRegexp. We should switch to using assertRaisesRegex
    instead. This commit eliminates these warnings from the log and the hacking
    rule N344 ensures that folks don't end up adding fresh code down the line
    with the outdated assertRaisesRegexp as well

    Partial-Bug: #1407736

    Change-Id: Ifba672f7568d5159c63bf88c534812e4e3a26d5a