failure to download image does not give helpful information

Bug #1512186 reported by Ian Wienand
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
High
Galyna Zholtkevych
ironic-python-agent
Fix Released
High
Jim Rollenhagen

Bug Description

I am trying to see why the ironic job fails in [1]

What is consistent is that the bm logs [2] show the vm stuck at "Unpacking initramfs..."

When digging a bit futher, I can see in the ironic conductor logs [3]

---
2015-10-30 07:48:47.444 11552 ERROR ironic.drivers.modules.agent [-] node 65a1e175-1342-41e7-bc84-9eafe227172d command status errored: {u'message': u'Error downloading image: Download of image id e942e5cc-3749-42d5-8275-f70ecb3102a2 failed: Image download failed for all URLs.', u'code': 500, u'type': u'ImageDownloadError', u'details': u'Download of image id e942e5cc-3749-42d5-8275-f70ecb3102a2 failed: Image download failed for all URLs.'}
---

That is basically it (searching for e942e5cc-* in the logs doesn't show anything else useful at all)

So I'm really still quite in the dark about what failed, why it failed, or what you might be able to do about it. It would be really nice if this gave you enough details to start debugging further.

(From some IRC logs [4], i'm guessing this does have to do with tempurls, and not surprisingly the original change is modifying that area ... so there's clearly some interaction here)

[1] https://review.openstack.org/#/c/237860/ (change `swift post` to openstackclient cli command)
[2] http://logs.openstack.org/60/237860/2/check/gate-tempest-dsvm-ironic-agent_ssh/8121f13/logs/ironic-bm-logs/baremetalbrbm_0_console.txt.gz
[3] http://logs.openstack.org/60/237860/2/check/gate-tempest-dsvm-ironic-agent_ssh/8121f13/logs/screen-ir-cond.txt.gz#_2015-10-30_07_48_47_444
[4] http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/%23openstack-ironic.2015-10-12.log.html

Revision history for this message
Ian Wienand (iwienand) wrote :

So in the end, it turned out the credentials were wrong and the temp-url wsan't being setup correctly [1]

but there was really nothing to help diagnose this

http://logs.openstack.org/60/237860/3/check/gate-tempest-dsvm-ironic-agent_ssh/fe125c2/logs/devstacklog.txt.gz#_2015-11-02_09_22_47_399

Revision history for this message
Dmitry Tantsur (divius) wrote :

Yeah, we at least need to have better error message in IPA, maybe also some checks in ironic itself that image can be accessed

Changed in ironic:
status: New → Triaged
importance: Undecided → High
Changed in ironic-python-agent:
status: New → Triaged
importance: Undecided → High
Changed in ironic:
assignee: nobody → Madhu Mohan Nelemane (mmohan-9)
Changed in ironic-python-agent:
assignee: nobody → Madhu Mohan Nelemane (mmohan-9)
Revision history for this message
Jay Faulkner (jason-oldos) wrote :

This has been assigned for >3 months with no patches posted. Unassigning.

Changed in ironic-python-agent:
assignee: Madhu Mohan Nelemane (mmohan-9) → nobody
Changed in ironic:
assignee: Madhu Mohan Nelemane (mmohan-9) → nobody
Changed in ironic:
assignee: nobody → Galyna Zholtkevych (gzholtkevych)
Changed in ironic-python-agent:
assignee: nobody → Galyna Zholtkevych (gzholtkevych)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic-python-agent (master)

Fix proposed to branch: master
Review: https://review.openstack.org/370824

Changed in ironic-python-agent:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/370875

Changed in ironic:
status: Triaged → In Progress
Changed in ironic-python-agent:
assignee: Galyna Zholtkevych (gzholtkevych) → Jim Rollenhagen (jim-rollenhagen)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic-python-agent (master)

Reviewed: https://review.openstack.org/370824
Committed: https://git.openstack.org/cgit/openstack/ironic-python-agent/commit/?id=993149cfb491dd81b9e9fda155c26d06e16d6fc2
Submitter: Jenkins
Branch: master

commit 993149cfb491dd81b9e9fda155c26d06e16d6fc2
Author: Galyna Zholtkevych <email address hidden>
Date: Thu Sep 15 16:26:29 2016 +0300

    Improve error message while download image

    Collecting warning logs in the case of download failure
    and write them to error logs in the end. This will help
    a user to diagnose a problem when warning log was not
    enabled.

    Change-Id: I4198d7be08fc11b616b3f95c595ff53794436e24
    Partial-Bug: 1512186

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (master)

Reviewed: https://review.openstack.org/370875
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=22dcfc65eebb6dcafee8bf3afe5749cf4e7c7f20
Submitter: Jenkins
Branch: master

commit 22dcfc65eebb6dcafee8bf3afe5749cf4e7c7f20
Author: Galyna Zholtkevych <email address hidden>
Date: Fri Sep 16 11:00:29 2016 +0300

    Validate the generated swift temp url

    During deployment, a swift temporary URL may be generated,
    to be used for getting the image for an instance.

    This validates the generated URL to make sure that it
    can be accessed.
    If validation fails, an exception is raised and the error
    is logged. This is helpful when diagnosing the problem with
    a deployment.

    Change-Id: I1f62bc9b5646d173ec6a3beb6282a5e6002c2030
    Closes-Bug: 1512186

Changed in ironic:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ironic 7.0.0

This issue was fixed in the openstack/ironic 7.0.0 release.

Revision history for this message
Vladyslav Drok (vdrok) wrote :

This can be closed I think

Changed in ironic-python-agent:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.