OSTF tests CRUD image action under glance v1 failed by timeout

[Ivan Kolodyazhny]

Staging job failed:
Actual:
Failed tests, fails: 1 should fail: 0 failed tests name: [{u'Check that user can create, update and delete image using Glance v1 (failure)': u'Time limit exceeded while waiting for Checking image status to finish. Please refer to OpenStack logs for more details.'}]

Build verification test for Centos

Case: ha_nova_vlan
1. Create cluster
2. Add 3 nodes with controller roles
3. Add 2 nodes with compute roles
4. Set up cluster to use Network VLAN manager with 8 networks
5. Deploy the cluster
6. Validate cluster was set up correctly, there are no dead
services, there are no errors in logs
7. Run network verification
8. Run OSTF
9. Create snapshot

[Stanislav Makar (smakar)]

could you please provide the link to this Stage job ?

[Sergey Otpuschennikov (sotpuschennikov)]

http://jenkins-product.srt.mirantis.net:8080/job/6.1.staging.centos.bvt_1/133/console

[Tatyanka (tatyana-leontovich)]

from the ostf log:
when tests was interrupted by timeout iname has saving state
fuel_health.test: DEBUG: Sleeping for 5 seconds
glanceclient.common.http: DEBUG: curl -i -X HEAD -H 'User-Agent: python-glanceclient' -H 'Content-Type: application/octet-stream' -H 'Accept-Encoding: gzip, deflate, compress' -H 'Accept: */*' -H 'X-Auth-Token: {SHA1}cf76de7a472fff8990b90846bf5d13a155f590f3' http://10.109.6.2:9292/v1/images/96108fcd-4f69-4c05-a660-21566663302e
urllib3.connectionpool: DEBUG: Setting read timeout to <object object at 0x7f7f1a79c0d0>
urllib3.connectionpool: DEBUG: "HEAD http://10.109.6.2:9292/v1/images/96108fcd-4f69-4c05-a660-21566663302e HTTP/1.1" 200 0
glanceclient.common.http: DEBUG:
HTTP/1.1 200 OK
content-length: 20
via: 1.1 apache_api_proxy:9292
vary: Accept-Encoding
x-image-meta-status: saving
x-image-meta-owner: dd108a78fdde44a6976410ea9af7ff70
x-image-meta-name: ostf_test-image_glance-899962118
x-image-meta-container_format: bare
x-image-meta-created_at: 2015-03-13T09:41:43
x-image-meta-min_ram: 0
x-image-meta-updated_at: 2015-03-13T09:41:44
x-image-meta-id: 96108fcd-4f69-4c05-a660-21566663302e
date: Fri, 13 Mar 2015 09:43:19 GMT
x-openstack-request-id: req-5abbe598-6bfe-41d6-b41e-ae768b107a8d
x-image-meta-deleted: False
content-encoding: gzip
x-image-meta-protected: False
x-image-meta-min_disk: 0
x-image-meta-size: 13200896
connection: close
x-image-meta-is_public: False
content-type: text/html; charset=UTF-8
x-image-meta-disk_format: qcow2

fuel_health.test: DEBUG: Sleeping for 5 seconds
fuel_health.common.test_mixins: INFO: Timeout 100s exceeded for Checking image status
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 179, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 225, in __exit__
    raise AssertionError(msg)
AssertionError: Time limit exceeded while waiting for Checking image status to finish.

from glance registry:
we can see that Updating image 96108fcd-4f69-4c05-a660-21566663302e with metadata: {u'status': u'active', u'location_data': [{u'url': u'swift+http://services%3Aglance:vvO0nZmV@10.109.7.2:5000/v2.0/glance/96108fcd-4f69-4c05-a660-21566663302e', u'status': u'active', u'metadata': {}}]} update /usr/lib/python2.6/site-packages/glance/registry/api/v1/images.py:467
2015-03-13T09:43:22.040377+00:00 info: 2015-03-13 09:43:22.035 5485 INFO glance.registry.api.v1.images [25a8fd33-8c6d-47b0-bfd2-ac05b804afbd 640c7aea0b8d432d8ae669b550a2dd30 dd108a78fdde44a6976410ea9af7ff70 - - -] Updating metadata for image 96108fcd-4f69-4c05-a660-21566663302e
2015-03-13T09:43:22.041701+00:00 info: 2015-03-13 09:43:22.037 5485 INFO glance.wsgi.server [25a8fd33-8c6d-47b0-bfd2-ac05b804afbd 640c7aea0b8d432d8ae669b550a2dd30 dd108a78fdde44a6976410ea9af7ff70 - - -]

image going to be active, so seems we need set timeout bigger

[Tatyanka (tatyana-leontovich)]

move to the medium priority, according to deployment pass successfully, also re-execution of tests says that test passed

[Tatyanka (tatyana-leontovich)]

link to the failed job http://jenkins-product.srt.mirantis.net:8080/view/6.1/job/6.1.staging.centos.bvt_1/133/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/164228

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/164228
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=8ec3ac1099340e68bde94c8bfb44daccd9a21b34
Submitter: Jenkins
Branch: master

commit 8ec3ac1099340e68bde94c8bfb44daccd9a21b34
Author: Tatyana Leontovich <email address hidden>
Date: Fri Mar 13 17:51:47 2015 +0200

    Set timeout bigger for wait image status

    If our test server is under load,
    test create/update/ delete image via glnce v1 failed by timeout

    So set timeout bigger, to avoid false positive result in bvt

    Change-Id: I94e146d3f84c3dec623714ba9ec57c6723806fda
    Closes-Bug: #1431858

[Tatyanka (tatyana-leontovich)]

verified 329 iso