unittests: ec2 tests escape to network

[Joshua Powers]

On Aug 10, 2017 the nightly and CI tests began taking 20 additional minutes. The additional time is occurring during the tox run, specifically during the 'py27' and 'xenial' environments. Here are comparisons of run times:

Aug 8: 1 min 52 seconds
Aug 9: 1 min 47 seconds
Aug 10: 19 mins

These tests are run at ~23:40 UTC each day, therefore a change would have occurred during the day resulting in the end-of-day run to see the effect.

From the Aug 10 run these two tox environments are adding a huge amount of time:
py27: Ran 980 tests in 525.871s
xenial: Ran 980 tests in 518.678s

Here are timing results from the Aug 9 run:
py27: Ran 978 tests in 9.051s
xenial: Ran 978 tests in 14.122s

Per the git log I believe it is tied to this revision:
https://git.launchpad.net/cloud-init/commit/?id=d5f855dd96ccbea77f61b0515b574ad2c43d116d

Which added two new tests:
test_ec2_local_returns_false_on_bsd
test_ec2_local_performs_dhcp_on_non_bsd
hence the increase the number of tests run. I am not however, saying that these two tests are the cause of the additional test time.

[Joshua Powers (powersj)]

Diagnosed which test it was with:

$ git clone https://git.launchpad.net/cloud-init
$ cd cloud-init
$ nosetests --with-timer --timer-ok 1 --timer-warning 5

This results in a sorted listed list of all tests with timing in green if less than 1 second, yellow if between 1 and 5 seconds, and red if greater than 5 seconds. All tests were less than one second, except these:

[success] 19.54% tests.unittests.test_datasource.test_ec2.TestEc2.test_unknown_platform_with_strict_false: 127.3695s
[success] 19.54% tests.unittests.test_datasource.test_ec2.TestEc2.test_valid_platform_with_strict_true: 127.3614s
[success] 19.54% tests.unittests.test_datasource.test_ec2.TestEc2.test_valid_platform_with_strict_false: 127.3573s
[success] 19.54% tests.unittests.test_datasource.test_ec2.TestEc2.test_network_config_property_returns_version_1_network_data: 127.3403s
[success] 19.54% tests.unittests.test_datasource.test_ec2.TestEc2.test_ec2_local_performs_dhcp_on_non_bsd: 127.3302s
[success] 0.78% tests.unittests.test_datasource.test_openstack.TestOpenStackDataSource.test_datasource: 5.0584s
[success] 0.77% tests.unittests.test_datasource.test_openstack.TestOpenStackDataSource.test_bad_datasource_meta: 5.0298s

These account for 99.25% of the run time.

[Scott Moser (smoser)]

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2988