test_shelve_unshelve_server fails on Ceph jobs

[Eric Harney]

An additional check was added to test_shelve_unshelve_server here:

"Check image after unshelve"
https://review.openstack.org/#/c/463248/

This new check has never succeeded on Ceph jobs, which is causing widespread CI failures in various projects.

This check is proposed for removal here:

https://review.openstack.org/#/c/471352/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/477821

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/477821
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=43a40654f403237bc167d7b9d2edaf5a5188f147
Submitter: Jenkins
Branch: master

commit 43a40654f403237bc167d7b9d2edaf5a5188f147
Author: ghanshyam <email address hidden>
Date: Tue Jun 27 11:23:27 2017 +0300

    Fix image deletion checks after unshelve server

    Tempest test checks and expect image to be deleted immediately
    when server is unshelved and in active state. But there is no
    guarantee of immediate image deletion from glance side.

    API ref does not state anything even it explicitly states
    about async postcondition
    -https://developer.openstack.org/api-ref/compute/?expanded=unshelve-restore-shelved-server-unshelve-action-detail#unshelve-restore-shelved-server-unshelve-action

    Discussed in below mail thread-
    - http://lists.openstack.org/pipermail/openstack-dev/2017-June/118574.html

    Test should wait for image deletion and if it does not then fail.
    Closes-Bug: #1697953

    Change-Id: I6552a3e4a7133a9fdaf203ab31cfef6dff764a62

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 16.1.0 release.

[Eric Harney (eharney)]

This test still fails on Ceph RBD.

[Eric Harney (eharney)]

See: http://logs.openstack.org/05/474205/3/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/69b8874/console.html

[Ghanshyam Mann (ghanshyammann)]

Thanks Eric for information.

from log it seems like glance not able to delete the image with in time-

http://logs.openstack.org/05/474205/3/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/69b8874/console.html#_2017-07-11_15_17_52_482929

glance api log says image cannot be deleted as it used by backend store

http://logs.openstack.org/05/474205/3/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/69b8874/logs/screen-g-api.txt.gz#_Jul_11_14_48_06_086901

need glance team to look further

[Attila Fazekas (afazekas)]

See also:https://bugs.launchpad.net/nova/+bug/1653953

[Artem Hrechanychenko (ahrechan)]

reproduced
        Body: None
    Response - Headers: {'status': '200', u'content-length': '1556', 'content-location': 'https://10.0.0.101:13774/v2.1/servers/cd83b1b9-6207-4c3a-892a-c8a060494060', u'x-compute-request-id': 'req-899731ef-29cc-41ac-876d-1bb436026850', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Mon, 23 Oct 2017 11:42:20 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-899731ef-29cc-41ac-876d-1bb436026850'}
        Body: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"tempest-ServerActionsTestJSON-1172139153-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:57:20:8e", "version": 4, "addr": "10.100.0.6", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://10.0.0.101:13774/v2.1/servers/cd83b1b9-6207-4c3a-892a-c8a060494060", "rel": "self"}, {"href": "https://10.0.0.101:13774/servers/cd83b1b9-6207-4c3a-892a-c8a060494060", "rel": "bookmark"}], "image": {"id": "8e57b8fa-871e-4299-a742-b6feedb5f016", "links": [{"href": "https://10.0.0.101:13774/images/8e57b8fa-871e-4299-a742-b6feedb5f016", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2017-10-23T11:37:19.000000", "flavor": {"id": "77b26b98-9182-4afc-8560-728bfd84e8ca", "links": [{"href": "https://10.0.0.101:13774/flavors/77b26b98-9182-4afc-8560-728bfd84e8ca", "rel": "bookmark"}]}, "id": "cd83b1b9-6207-4c3a-892a-c8a060494060", "security_groups": [{"name": "default"}], "user_id": "c3b9dd20a4eb4ee68b11dd5406722261", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "status": "ACTIVE", "updated": "2017-10-23T11:37:19Z", "hostId": "c26060b2344c009f8c61122b329c9950178cc1c63a0bd1eaecf10f1c", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest-ServerActionsTestJSON-server-1950192548", "created": "2017-10-23T11:36:25Z", "tenant_id": "25b0c5d99e784198ba5dbca95977d47f", "os-extended-volumes:volumes_attached": [], "metadata": {}}}
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_actions.py", line 547, in test_shelve_unshelve_server
    glance_client.wait_for_resource_deletion(images[0]['id'])
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 871, in wait_for_resource_deletion
    raise exceptions.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (ServerActionsTestJSON:test_shelve_unshelve_server) Failed to delete image 8f838e1f-9d75-4efe-ac72-d510e42d77c9 within the required time (300 s).
Ran 1 tests in 377.869s (+375.440s)
FAILED (id=4, failures=1 (+1))

[OpenStack Infra (hudson-openstack)]

Change abandoned by Erno Kuvaja (<email address hidden>) on branch: master
Review: https://review.openstack.org/471352

[Abhishek Kekane (abhishek-kekane)]

Marked invalid, no updates since last 6 months.

[Matt Riedemann (mriedem)]

Looks like this is a duplicate of bug 1653953 which has a fix proposed here:

https://review.opendev.org/#/c/457886/

We just need to remove this line in the ceph job:

https://github.com/openstack/devstack-plugin-ceph/blob/51f8550b024a8197c329c76f894772b090a8e0b1/devstack/plugin.sh#L102

And depends-on the nova fix to verify.

[Matt Riedemann (mriedem)]

The nova fix is being tested here: https://review.opendev.org/662327