Problems with images bubble up as a simple "There are not enough hosts available"

[Julian Edwards]

When starting a new instance, I received the generic "There are not enough hosts available" error, but the real reason was buried in logs, which was that the image I was trying to use was corrupt.

[Davanum Srinivas (DIMS) (dims-v)]

Which version of Openstack? Nova? Can you please share some logs from both the nova api and the nova compute components?

[Julian Edwards (julian-edwards)]

Sorry, I missed info off. This was done in devstack after I ran stack.sh.

n-cond.log shows:

RescheduledException: Build of instance fd530fd4-e4e6-476a-aa2d-5f4ad6b02cbb was re-scheduled: Unexpected error while running command.\nCommand: qemu-img convert -O raw /opt/stack/data/nova/instances/_base/89faaaa5ec41988d9b775c2fdc678fb4a4b974d9.part /opt/stack/data/nova/instances/_b
ase/89faaaa5ec41988d9b775c2fdc678fb4a4b974d9.converted\nExit code: 1\nStdout: u''\nStderr: u'qemu-img: error while reading sector 702336: Input/output error\\n'

[Davanum Srinivas (DIMS) (dims-v)]

Julian,

Can you please see if you can find the original traceback for "qemu-img: error while reading sector 702336: Input/output error" in the nova-compute log?

thanks,
dims

[Julian Edwards (julian-edwards)]

This is all I have:

[req-79b024be-ebb6-473e-a832-bb7f1fd58672 demo invisible_to_admin] [instance: fd530fd4-e4e6-476a-aa2d-5f4ad6b02cbb] Error from last host: devstack (node devstack): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2175, in _do_build_and_run_instance\n filter_properties)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2318, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance fd530fd4-e4e6-476a-aa2d-5f4ad6b02cbb was re-scheduled: Unexpected error while running command.\nCommand: qemu-img convert -O raw /opt/stack/data/nova/instances/_base/89faaaa5ec41988d9b775c2fdc678fb4a4b974d9.part /opt/stack/data/nova/instances/_base/89faaaa5ec41988d9b775c2fdc678fb4a4b974d9.converted\nExit code: 1\nStdout: u''\nStderr: u'qemu-img: error while reading sector 702336: Input/output error\\n'\n"]

I'd like to help fix this, if you wouldn't mind mentoring me?

[Davanum Srinivas (DIMS) (dims-v)]

Julian,

if you see line 2318 mentioned, you will see we are catching an uncategorized exception:
http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n2318

if you grep for "Unexpected error" you will see it comes from:
http://git.openstack.org/cgit/openstack/oslo.concurrency/tree/oslo_concurrency/processutils.py#n69

So we should be catching processutils.ProcessExecutionError just like we catch exception.FlavorDiskTooSmall earlier in manager.py

you may end up with something like this...

http://paste.openstack.org/show/197374/

-- dims

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/169436

[Julian Edwards (julian-edwards)]

Damn, my email reply to the bug got lost. Anyway, I was saying thanks and that I'd try to do a review, but you beat me to it. I +1ed your review, as it tests out OK.

Thanks.

[Davanum Srinivas (DIMS) (dims-v)]

Julian,

Thanks, so a related bug https://bugs.launchpad.net/nova/+bug/1431291 was bumped up in priority and since we are short on time, i ended up filing this one. Please pick another bug and ping me, i'll try to help get you going.

-- dims

[Julian Edwards (julian-edwards)]

No worries, understood!

[OpenStack Infra (hudson-openstack)]

Change abandoned by Davanum Srinivas (dims) (<email address hidden>) on branch: master
Review: https://review.openstack.org/169436

[Markus Zoeller (markus_z) (mzoeller)]

Solving an inconsistency: The bug is 'In Progress' but without an assignee. I set the status back to the last known status before the change to 'In Progress'.

Feel free to assign the bug to yourself. If you do so, please set it to 'In Progress'.

[Stephen Finucane (stephenfinucane)]

I'm not able to figure out how to boot a corrupt image :S so unassigning myself

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/264349

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264349
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9a4ecfd96dad32fd4726c46dc6d89e956f1f2a29
Submitter: Jenkins
Branch: master

commit 9a4ecfd96dad32fd4726c46dc6d89e956f1f2a29
Author: Chris Dent <email address hidden>
Date: Mon Jan 11 12:05:00 2016 +0000

    Propagate qemu-img errors to compute manager

    When qemu-img is called with oslo_concurrency.process_utils.execute
    the ProcessExecutionError was raised when qemu-img either fails to
    execute or has a non-zero exit code. This error did not propagate
    up to the compute manager with any meaningful information meaning
    that if an instance build fails the error message is the generic
    "There are not enough hosts available".

    This change captures ProcessExecutionError and re-raises the
    exception as either InvalidDiskInfo (in qemu_img_info) or
    ImageUnacceptable (in convert_image and fetch_to_raw) and makes the
    manager accept this as a cause for a BuildAbortException on the
    logic that if the image is bad, things are dire, let's bail.

    Based on the code in qemu_img_info it appears there was a
    misunderstanding of how process_utils.execute behaves so it seems
    likely this problem is present elsewhere in the code. This change
    attempts to only address the issue as it shows up on the new
    instance path described in the related bug.

    Change-Id: I4fa1c258db58c70dfbf0178b7bb13978fda3a11f
    Closes-Bug: #1436166

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/nova 13.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/288594

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/288594
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b806adb032567c2c84d61834e6f8d2684723c194
Submitter: Jenkins
Branch: stable/liberty

commit b806adb032567c2c84d61834e6f8d2684723c194
Author: Chris Dent <email address hidden>
Date: Mon Jan 11 12:05:00 2016 +0000

    Propagate qemu-img errors to compute manager

    When qemu-img is called with oslo_concurrency.process_utils.execute
    the ProcessExecutionError was raised when qemu-img either fails to
    execute or has a non-zero exit code. This error did not propagate
    up to the compute manager with any meaningful information meaning
    that if an instance build fails the error message is the generic
    "There are not enough hosts available".

    This change captures ProcessExecutionError and re-raises the
    exception as either InvalidDiskInfo (in qemu_img_info) or
    ImageUnacceptable (in convert_image and fetch_to_raw) and makes the
    manager accept this as a cause for a BuildAbortException on the
    logic that if the image is bad, things are dire, let's bail.

    Based on the code in qemu_img_info it appears there was a
    misunderstanding of how process_utils.execute behaves so it seems
    likely this problem is present elsewhere in the code. This change
    attempts to only address the issue as it shows up on the new
    instance path described in the related bug.

    Conflicts:
     nova/virt/images.py

    Change-Id: I4fa1c258db58c70dfbf0178b7bb13978fda3a11f
    Closes-Bug: #1436166
    (cherry picked from commit 9a4ecfd96dad32fd4726c46dc6d89e956f1f2a29)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 12.0.2 release.