unittests: ec2 tests escape to network

Bug #1714117 reported by Joshua Powers
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fix Released
Chad Smith

Bug Description

On Aug 10, 2017 the nightly and CI tests began taking 20 additional minutes. The additional time is occurring during the tox run, specifically during the 'py27' and 'xenial' environments. Here are comparisons of run times:

Aug 8: 1 min 52 seconds
Aug 9: 1 min 47 seconds
Aug 10: 19 mins

These tests are run at ~23:40 UTC each day, therefore a change would have occurred during the day resulting in the end-of-day run to see the effect.

From the Aug 10 run these two tox environments are adding a huge amount of time:
py27: Ran 980 tests in 525.871s
xenial: Ran 980 tests in 518.678s

Here are timing results from the Aug 9 run:
py27: Ran 978 tests in 9.051s
xenial: Ran 978 tests in 14.122s

Per the git log I believe it is tied to this revision:

Which added two new tests:
hence the increase the number of tests run. I am not however, saying that these two tests are the cause of the additional test time.

Related branches

Joshua Powers (powersj)
Changed in cloud-init:
status: New → Confirmed
Revision history for this message
Joshua Powers (powersj) wrote :

Diagnosed which test it was with:

$ git clone https://git.launchpad.net/cloud-init
$ cd cloud-init
$ nosetests --with-timer --timer-ok 1 --timer-warning 5

This results in a sorted listed list of all tests with timing in green if less than 1 second, yellow if between 1 and 5 seconds, and red if greater than 5 seconds. All tests were less than one second, except these:

[success] 19.54% tests.unittests.test_datasource.test_ec2.TestEc2.test_unknown_platform_with_strict_false: 127.3695s
[success] 19.54% tests.unittests.test_datasource.test_ec2.TestEc2.test_valid_platform_with_strict_true: 127.3614s
[success] 19.54% tests.unittests.test_datasource.test_ec2.TestEc2.test_valid_platform_with_strict_false: 127.3573s
[success] 19.54% tests.unittests.test_datasource.test_ec2.TestEc2.test_network_config_property_returns_version_1_network_data: 127.3403s
[success] 19.54% tests.unittests.test_datasource.test_ec2.TestEc2.test_ec2_local_performs_dhcp_on_non_bsd: 127.3302s
[success] 0.78% tests.unittests.test_datasource.test_openstack.TestOpenStackDataSource.test_datasource: 5.0584s
[success] 0.77% tests.unittests.test_datasource.test_openstack.TestOpenStackDataSource.test_bad_datasource_meta: 5.0298s

These account for 99.25% of the run time.

Joshua Powers (powersj)
summary: - tox: long CI times
+ unittests: ec2 tests escape to network
Chad Smith (chad.smith)
Changed in cloud-init:
status: Confirmed → In Progress
importance: Undecided → Medium
assignee: nobody → Chad Smith (chad.smith)
Chad Smith (chad.smith)
Changed in cloud-init:
status: In Progress → Fix Committed
Revision history for this message
Scott Moser (smoser) wrote : Fixed in Cloud-init 17.1

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.