agent_ssh driver did not check RAM size

Bug #1471081 reported by Haomeng,Wang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
High
Zhenguo Niu

Bug Description

agent_ssh will download the image from glance via *swift_temp_url* into in-memory-folder /tmp/<img_id> first then run 'qemu-img' to 'copy' image to local harddisk.

However if the RAM size is less than the converted image, will break the ipa deployments as below log:

ir-cond.log:2015-06-26 07:28:27.258 ERROR ironic.drivers.modules.agent [-] node db1a8a8f-885d-4433-a772-551f917457f0 command status errored: {u'message': u'Error downloading image.', u'code': 500, u'type': u'ImageDownloadError', u'details': u'Download of image id e18a422d-94d8-4389-b345-21be90b6f745 failed: Unable to write image to /tmp/e18a422d-94d8-4389-b345-21be90b6f745. Error: [Errno 28] No space left on device'}

Tags: agent
Haomeng,Wang (whaom)
description: updated
description: updated
description: updated
Revision history for this message
Haomeng,Wang (whaom) wrote :

And it is better that we detect the *real* image size in raw format, that can be as input to compare with the physical RAM size to see if it is enough, any comments?

Revision history for this message
Dmitry Tantsur (divius) wrote :

I think we should stream an image to the hard drive instead of downloading it into RAM...

Changed in ironic:
status: New → Confirmed
importance: Undecided → High
tags: added: agent
Changed in ironic:
assignee: nobody → Zhenguo Niu (niu-zglinux)
Revision history for this message
Zhenguo Niu (niu-zglinux) wrote :

@Dmitry Tantsur if the image format is not raw, we need to download the whole image and convert it before write to disk.

Revision history for this message
Lucas Alvares Gomes (lucasagomes) wrote :

I think we can convert it on the flight no? With qemu-img... worth investigating

Revision history for this message
Ramakrishnan G (rameshg87) (rameshg87) wrote :

As far as I know, it's not possible for qemu-img to convert on the fly.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=513451

>> This is technically not possible. Given the formats involved, the file
>> is not read/written linearly, which render the use of stdin/stdout
>> impossible. Tagging the bug as wontfix.

Revision history for this message
Zhenguo Niu (niu-zglinux) wrote :

yeah, I agree we can't convert on the fly, so I'd like to add check for the node ram property and the image.min_disk before provisioning, when using agent deploy.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/215254

Changed in ironic:
status: Confirmed → In Progress
Changed in ironic:
assignee: Zhenguo Niu (niu-zglinux) → Haomeng,Wang (whaom)
Changed in ironic:
assignee: Haomeng,Wang (whaom) → Zhenguo Niu (niu-zglinux)
Changed in ironic:
milestone: none → 4.2.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (master)

Reviewed: https://review.openstack.org/215254
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=a715e3dcf94af4cd94ba5a51b745d411bc473182
Submitter: Jenkins
Branch: master

commit a715e3dcf94af4cd94ba5a51b745d411bc473182
Author: Zhenguo Niu <email address hidden>
Date: Fri Aug 21 01:11:12 2015 +0800

    Check image size before provisioning for agent driver

    agent will download the image from glance into in-memory-folder first
    then run 'qemu-img' to 'copy' image to local harddisk, but if the RAM
    size is less than the (image size + reserved RAM size), will break the
    IPA deployments.

    Change-Id: I5ed62a8936e996ffed60c811bd01d3be97f95547
    Closes-Bug: #1471081

Changed in ironic:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ironic:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.