tripleo

tempest failing in centos7 third party RDO jobs due to corrupted cirros image

Bug #1860094 reported by yatin on 2020-01-17

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	High	Unassigned	tripleo ussuri-2 "tripleo ussuri-2"

Bug Description

Jobs are failing with Errors like below:-
- Stderr: u"qemu-img: Could not open '/var/lib/cinder/conversion/tmpFgX776standalone.localdomain@tripleo_iscsi': Image is not in qcow2 format\n"
- DeviceDetachFailed: Device detach failed for vdb: Unable to detach the device from the live config.

Example logs:-
http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-stein/0bc481d/logs/tempest.html.gz
http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-stein/0bc481d/logs/undercloud/var/log/containers/cinder/cinder-volume.log.txt.gz
http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario002-standalone-stein/361cf98/logs/tempest.html.gz
http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario002-standalone-stein/361cf98/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz

cirros image is corrupted in nodepool image. confirmed by checking it like:-
file /opt/cache/files/cirros-0.3.5-x86_64-disk.img
/opt/cache/files/cirros-0.3.5-x86_64-disk.img: HTML document, ASCII text, with CRLF line terminators

cat /opt/cache/files/cirros-0.3.5-x86_64-disk.img
<html><body onload='resend();'>
<script>
function resend() {
document.cookie="sto-idd=1f225573927eddae; max-age=1; path=/";
window.location.reload();
}
</script>
</body>
</html>

NOTE: this affect all jobs in all releases which are relying on cached cirros image. Some examples:-
https://trunk-primary.rdoproject.org/api-centos-stein/api/civotes_detail.html?commit_hash=e669382b251b5fec28dba8e40a03c7b8f23f51d3&distro_hash=9b5439527e6cc440aa3130d9d07fb495c94ad660

Possibly this was caused by issues is http://download.cirros-cloud.net/ yesterday. Currently the site seems responding fine, building new nodepool image should fix it. @jpena is working on creating new image.

See original description

Tags:

yatin (yatinkarel) on 2020-01-17

description:	updated
Changed in tripleo:
status:	New → Triaged
milestone:	none → ussuri-2
importance:	Undecided → High

Revision history for this message

Javier Peña (jpena-c) wrote on 2020-01-17:

The updated upstream-centos-7 image has been uploaded. After the cirros website issues, the cached qcow2 images were wrong. After removing the cached images, proper ones were downloaded.

wes hayutin (weshayutin) on 2020-01-17

tags:

added: tempest

Revision history for this message

yatin (yatinkarel) wrote on 2020-01-20:

Jobs are passing now after new image is updated, https://trunk-primary.rdoproject.org/api-centos-stein/api/civotes_detail.html?commit_hash=581b8a2096f1128cf60b967d69d52f91193b3554&distro_hash=58dbe344b9cbade0c3fae69dc09a60eda62dec12

This can be closed now.

wes hayutin (weshayutin) on 2020-01-20

Changed in tripleo:
status:	Triaged → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.