manila 'cover' tox job shows different results after reruns

[Valeriy Ponomaryov]

We have tox job "cover" that calculate total code coverage with unit tests.
And this job reports different results on the same state if run it again and again.

Results for three runs:

Run№ statements missing excluded coverage
1 43573 4568 0 90%
2 43573 4580 0 89%
3 43573 4577 0 89%
4 43573 4584 0 89%

Looks like we have some concurrency issues in unit tests that lead to really different code lines be called.

Some debug discovered several problem places, that have different coverage:

manila/db/sqlalchemy/api
manila/service
manila/share/drivers/generic
manila/test
manila/tests/network/test_share_network_db
manila/tests/share/drivers/test_generic
manila/tests/share/test_manager
manila/tests/test_service
manila/openstack/common/scheduler/weights/__init__
manila/share/drivers/hds/sop

[Valeriy Ponomaryov (vponomaryov)]

Debuging case of manila/db/sqlalchemy/api found out that "some" of not covered code lines are taken as "covered" sometimes. So, we can fix it with improving test coverage.

[Valeriy Ponomaryov (vponomaryov)]

Observation shows that one of the reasons - presense of periodic tasks. They either catch on while tests are running or not depending on execution time that differs from time to time.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/205549

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/205549
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=4a87762adf03654c81e23fc5cff061d09db49719
Submitter: Jenkins
Branch: master

commit 4a87762adf03654c81e23fc5cff061d09db49719
Author: Valeriy Ponomaryov <email address hidden>
Date: Fri Jul 24 16:25:01 2015 +0300

    Add test coverage for periodic tasks

    Add test coverage for methods that are periodic tasks for two reasons:

    - Test it in general
    - Reduce test coverage reporting dispersion, where periodic tasks often have
    different coverage with each subsequent run.

    Change-Id: I539ac140aa4f89650c093350063e37f3afa46922
    Partial-Bug: #1477547

[Tom Barron (tpb)]

This issue should be resolved by https://review.openstack.org/#/c/515118