Fail heat_integrationtests.scenario.test_server_cfn_init.CfnInitIntegrationTest.test_server_cfn_init due to Nova error

Bug #1534026 reported by Sergey Kraynev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Medium
Thomas Herve

Bug Description

Functional test heat_integrationtests.scenario.test_server_cfn_init.CfnInitIntegrationTest.test_server_cfn_init fails due to error related with Nova service.

http://logs.openstack.org/59/263659/6/gate/gate-heat-dsvm-functional-orig-mysql/c37dc34/console.html

Error in Heat engine log:

2016-01-14 03:10:18.603 | 2016-01-14 03:10:18.567 | {0} heat_integrationtests.scenario.test_server_cfn_init.CfnInitIntegrationTest.test_server_cfn_init [273.761970s] ... FAILED
2016-01-14 03:10:18.608 | 2016-01-14 03:10:18.572 |
2016-01-14 03:10:18.611 | 2016-01-14 03:10:18.575 | Captured traceback:
2016-01-14 03:10:18.619 | 2016-01-14 03:10:18.583 | ~~~~~~~~~~~~~~~~~~~
2016-01-14 03:10:18.626 | 2016-01-14 03:10:18.591 | Traceback (most recent call last):
2016-01-14 03:10:18.629 | 2016-01-14 03:10:18.594 | File "heat_integrationtests/scenario/test_server_cfn_init.py", line 121, in test_server_cfn_init
2016-01-14 03:10:18.631 | 2016-01-14 03:10:18.596 | self.check_stack(stack_id)
2016-01-14 03:10:18.662 | 2016-01-14 03:10:18.599 | File "heat_integrationtests/scenario/test_server_cfn_init.py", line 30, in check_stack
2016-01-14 03:10:18.662 | 2016-01-14 03:10:18.601 | sid, res, 'CREATE_COMPLETE')
2016-01-14 03:10:18.662 | 2016-01-14 03:10:18.605 | File "heat_integrationtests/common/test.py", line 252, in _wait_for_resource_status
2016-01-14 03:10:18.662 | 2016-01-14 03:10:18.608 | resource_status_reason=res.resource_status_reason)
2016-01-14 03:10:18.662 | 2016-01-14 03:10:18.611 | heat_integrationtests.common.exceptions.StackResourceBuildErrorException: Resource SmokeServer in stack CfnInitIntegrationTest-1690615954/e2459003-e7f9-4a4e-9c7e-1d9a4f8ceed7 is in CREATE_FAILED status due to 'ResourceInError: resources.SmokeServer: Went to status ERROR due to "Message: Unknown, Code: Unknown"'
2016-01-14 03:10:18.662 | 2016-01-14 03:10:18.614 |

Corresponding error in Nova-cond log:

2016-01-14 03:10:04.218 ERROR nova.scheduler.utils [req-fac76492-ead6-458e-bda9-8358646de38a demo demo] [instance: 03903507-319e-40f1-ad74-b9cba80a9750] Error from last host: devstack-trusty-ovh-bhs1-7187016 (node devstack-trusty-ovh-bhs1-7187016): [u'Traceback (most recent call last):\n', u' File "/opt/stack/new/nova/nova/compute/manager.py", line 1916, in _do_build_and_run_instance\n filter_properties)\n', u' File "/opt/stack/new/nova/nova/compute/manager.py", line 2080, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance 03903507-319e-40f1-ad74-b9cba80a9750 was re-scheduled: Unexpected error while running command.\nCommand: qemu-img convert -O raw /opt/stack/data/nova/instances/_base/9cf6f64b97de957a3e42a6f9bda69a63b26bcc67.part /opt/stack/data/nova/instances/_base/9cf6f64b97de957a3e42a6f9bda69a63b26bcc67.converted -f qcow2\nExit code: 1\nStdout: u''\nStderr: u'qemu-img: error while reading sector 2397568: Input/output error\\n'\n"]
2016-01-14 03:10:04.326 WARNING nova.scheduler.utils [req-fac76492-ead6-458e-bda9-8358646de38a demo demo] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 150, in inner
    return func(*args, **kwargs)

  File "/opt/stack/new/nova/nova/scheduler/manager.py", line 78, in select_destinations
    dests = self.driver.select_destinations(ctxt, spec_obj)

  File "/opt/stack/new/nova/nova/scheduler/filter_scheduler.py", line 74, in select_destinations
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.

Revision history for this message
Thomas Herve (therve) wrote :
Changed in heat:
milestone: mitaka-3 → mitaka-rc1
Changed in heat:
milestone: mitaka-rc1 → newton-1
Angus Salkeld (asalkeld)
Changed in heat:
status: New → Triaged
tags: added: gate-failure
Thomas Herve (therve)
Changed in heat:
milestone: newton-1 → ongoing
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

I'm not seeing evidence of this failure in Kibana, we can always reopen

Changed in heat:
status: Triaged → Invalid
Revision history for this message
Thomas Herve (therve) wrote :

Got it again here: http://logs.openstack.org/06/279406/19/gate/gate-heat-dsvm-functional-orig-mysql-lbaasv2/774b57b/

We get a new clearer error now though: Image a494b0a9-b49a-4b41-bc7c-ed10e15e898e is unacceptable: Unable to convert image to raw.

Changed in heat:
status: Invalid → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/338516

Changed in heat:
assignee: nobody → Thomas Herve (therve)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/338516
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=150a013597da2605be34a3971430283fc25946da
Submitter: Jenkins
Branch: master

commit 150a013597da2605be34a3971430283fc25946da
Author: Thomas Herve <email address hidden>
Date: Wed Jul 6 21:43:57 2016 +0200

    Fail the test run if image download fails

    When we run the integration tests, we download the test image that we
    need and upload it on the fly. If the download is aborted because of
    some network problem, the image is incorrect, but we still try to run
    the tests and boot servers with it. We should fail early instead.

    Change-Id: I8797440086206749c9098dcd42b84d95426fbf3e
    Closes-Bug: #1534026

Changed in heat:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/338612

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/heat 7.0.0.0b2

This issue was fixed in the openstack/heat 7.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/352177

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/mitaka)

Reviewed: https://review.openstack.org/352177
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=c3ac439ed232642134778f980bb82e5aeb783bb4
Submitter: Jenkins
Branch: stable/mitaka

commit c3ac439ed232642134778f980bb82e5aeb783bb4
Author: Thomas Herve <email address hidden>
Date: Wed Jul 6 21:43:57 2016 +0200

    Use Fedora 24 base image

    This is to fix mitaka CI, which is failing since the heat-test-image
    rebuild failed.

    Closes-Bug: #1534026
    (cherry picked from commits 150a013597da2605be34a3971430283fc25946da
                            and 59fc53a66c4dec45e4d150bce0a1d674477f710c)
    Change-Id: Ifaccf340f0ff4eac52b1eeef5914b7e90f2687da

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (master)

Change abandoned by Steve Baker (<email address hidden>) on branch: master
Review: https://review.openstack.org/338612

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 6.1.0

This issue was fixed in the openstack/heat 6.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.