[integration tests][nova] TimeoutExpired: Timeout of 60 seconds expired waiting for volume to be deleted

[Georgy Dyuldin]

Test result:

https://mirantis.testrail.com/index.php?/tests/view/6075224

Trace:

cls = <mos_tests.nova.nova_test.TestNovaDeferredDelete object at 0x7fc9fc255950>
os_conn = <mos_tests.environment.os_actions.OpenStackActions object at 0x7fc9fd3313d0>

    @classmethod
    @pytest.yield_fixture
    def volumes(cls, os_conn):
        """Volumes cleanUp"""
        volumes = []
        yield volumes
        for volume in volumes:
> common_functions.delete_volume(os_conn.cinder, volume)

mos_tests/nova/nova_test.py:625:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cinder_client = <cinderclient.v2.client.Client object at 0x7fc9fcebeb50>
volume = <Volume: dbe45541-bd83-4653-b842-0665c11106a4>

    def delete_volume(cinder_client, volume):
        """Delete volume and check that it is absent in the list
            :param cinder_client: Cinder API client connection point
            :param volume: volume
        """
        if volume in cinder_client.volumes.list():
            volume.get()
            if len(volume.attachments) > 0:
                volume.detach()
            cinder_client.volumes.delete(volume)
            volume_id = volume.id
            wait(lambda: not is_volume_exists(cinder_client, volume_id),
                 timeout_seconds=60,
> waiting_for='volume to be deleted')
E TimeoutExpired: Timeout of 60 seconds expired waiting for volume to be deleted

mos_tests/functions/common.py:461: TimeoutExpired

[Anna Babich (ababich)]

As we can see from http://cz7776.bud.mirantis.net:8080/jenkins/job/9.0_Neutron_NovaCeph/2/testReport/mos_tests.nova.nova_test/TestNovaDeferredDelete/test_restore_deleted_instance_86400___842493__/ , it's a cleanup issue, not related to the test scenario.
Anyway, I've tried to reproduce it on env (Cinder Ceph + Nova Ceph):
[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 398
cat /etc/fuel_build_number:
 398
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6346.noarch
 fuel-bootstrap-cli-9.0.0-1.mos282.noarch
 fuel-migrate-9.0.0-1.mos8386.noarch
 rubygem-astute-9.0.0-1.mos746.noarch
 fuel-misc-9.0.0-1.mos8386.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-mirror-9.0.0-1.mos136.noarch
 fuel-openstack-metadata-9.0.0-1.mos8693.noarch
 fuel-notify-9.0.0-1.mos8386.noarch
 nailgun-mcagents-9.0.0-1.mos746.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8693.noarch
 python-fuelclient-9.0.0-1.mos315.noarch
 fuelmenu-9.0.0-1.mos270.noarch
 fuel-9.0.0-1.mos6346.noarch
 fuel-utils-9.0.0-1.mos8386.noarch
 fuel-setup-9.0.0-1.mos6346.noarch
 fuel-library9.0-9.0.0-1.mos8386.noarch
 shotgun-9.0.0-1.mos89.noarch
 fuel-agent-9.0.0-1.mos282.noarch
 fuel-ui-9.0.0-1.mos2699.noarch
 fuel-ostf-9.0.0-1.mos934.noarch
 python-packetary-9.0.0-1.mos136.noarch
 fuel-nailgun-9.0.0-1.mos8693.noarch

and the results are: http://paste.openstack.org/show/506315/ , the volume has been detached and deleted successfully.

Also, I have a question concerning:
def delete_volume(cinder_client, volume):
        """Delete volume and check that it is absent in the list
            :param cinder_client: Cinder API client connection point
            :param volume: volume
        """
        if volume in cinder_client.volumes.list():
            volume.get()
            if len(volume.attachments) > 0:
                volume.detach()
            cinder_client.volumes.delete(volume)
            volume_id = volume.id
            wait(lambda: not is_volume_exists(cinder_client, volume_id),
                 timeout_seconds=60,
> waiting_for='volume to be deleted')

we can see that detach() method is called within cinder-client, but in fact it's Nova's action (nova-attach, nova-detach), and I'm not sure now that it's cinder-client's one as well. Is the nova-client enabled earlier for these operations?

[Dina Belova (dbelova)]

More than a month in the incomplete state -> moving to invalid. Please mark back to confirmed if this will be reproduced again.