Need clean up ftp transfer faiure in powervm driver

Bug #1188884 reported by ChangBo Guo(gcb)
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
ChangBo Guo(gcb)
Grizzly
Fix Released
Low
ChangBo Guo(gcb)

Bug Description

During spawn with powervm driver , transfer image file from glance server to IVM host may fail in some conditions , like IVM host doesn't have enough disk space . It leaves the garbage file in IVM host , will disturb next spawn after resovle envrioment issue ,need clean it .

In my test case I started an ISO deploy, then while the file was being copied to the IVM I filled up the destination directory to create an out-of-space condition.
The ftp transfer failed and the ISO was not deleted:
-------------------------------------------------------------------------------------------------------
c25ff88-bcae-4850-bb77-70929f67c0b0.bc0_b01.iso to /home/padmin failed
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator Traceback (most recent call last):
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator File "/usr/lib/python2.6/site-packages/nova/virt/powervm/operator.py", line 618, in _create_image
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator context, instance, image_id)
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator File "/usr/lib/python2.6/site-packages/nova/virt/powervm/decorator.py", line 25, in wrapper
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator r = f(*args, **kwds)
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator File "/usr/lib/python2.6/site-packages/nova/virt/powervm/blockdev.py", line 152, in create_volume_from_image
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator image_meta)
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator File "/usr/lib/python2.6/site-packages/nova/virt/powervm/decorator.py", line 25, in wrapper
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator r = f(*args, **kwds)
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 242, in inner
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator retval = f(*args, **kwargs)
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator File "/usr/lib/python2.6/site-packages/nova/virt/powervm/blockdev.py", line 252, in _prepare_iso_image_on_host
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator host_dir)
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator File "/usr/lib/python2.6/site-packages/nova/virt/powervm/blockdev.py", line 178, in copy_image_file
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator remote_path)
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator File "/usr/lib/python2.6/site-packages/nova/virt/powervm/common.py", line 125, in ftp_put_command
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator source_path=local_path, dest_path=remote_dir)
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator PowerVMFTPTransferFailed: FTP PUT from /var/lib/nova/images/ac25ff88-bcae-4850-bb77-70929f67c0b0.bc0_b01.iso to /home/padmin failed
2013-06-06 16:43:32.572 18655 TRACE nova.virt.powervm.operator
-------------------------------------------------------------------------------------------------------

Tags: powervm
tags: added: powervm
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/32238

Changed in nova:
assignee: nobody → ChangBo Guo (guochbo)
status: New → In Progress
Revision history for this message
ChangBo Guo(gcb) (glongwave) wrote :

log into IVM host , the image file transferred with failure has zero size .
$ ls -l
rksh: There is not enough space in the file system.
total 2005984
-rw-r----- 1 padmin staff 0 Apr 24 08:58 403b460c-0d5a-47ac-bf4f-96753e7b4e9b.ip9_114_23_22.iso
-rw-r----- 1 padmin staff 0 Apr 25 03:21 8853af84-f190-4d6e-a6cb-a8511ee32167.ip9_114_23_22.iso

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/32238
Committed: http://github.com/openstack/nova/commit/84b41d944a841bd21cd72d57dd94ccf4810b6433
Submitter: Jenkins
Branch: master

commit 84b41d944a841bd21cd72d57dd94ccf4810b6433
Author: Chang Bo Guo <email address hidden>
Date: Fri Jun 7 22:23:39 2013 -0700

    Remove junk file when ftp transfer failure

    During spawn with powervm driver, the image transfer to the IVM host
    may fail in some conditions, eg. IVM host doesn't have enough disk
    space. It leaves the junk file in IVM host, and this will disturb
    next spawn after resolving environment issue, need clean it.

    Fixes bug #1188884

    Change-Id: I7b8d719995c2702c89cf55cef5f34bde9d01c1b9

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/grizzly)

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/35429

Thierry Carrez (ttx)
Changed in nova:
milestone: none → havana-2
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/grizzly)

Reviewed: https://review.openstack.org/35429
Committed: http://github.com/openstack/nova/commit/fbde109390f32204f6355d937449f0f54f07ffd7
Submitter: Jenkins
Branch: stable/grizzly

commit fbde109390f32204f6355d937449f0f54f07ffd7
Author: Chang Bo Guo <email address hidden>
Date: Tue Jul 2 19:47:56 2013 -0700

    Remove junk file when ftp transfer failure

    During spawn with powervm driver, the image transfer to the IVM host
    may fail in some conditions, eg. IVM host doesn't have enough disk
    space. It leaves the junk file in IVM host, and this will disturb
    next spawn after resolving environment issue, need clean it.

    Fixes bug #1188884

    Change-Id: I6347dec4cfda61bed0960159e8a3402fcffa296b

tags: added: in-stable-grizzly
Alan Pevec (apevec)
tags: removed: in-stable-grizzly
Changed in nova:
importance: Undecided → Low
Thierry Carrez (ttx)
Changed in nova:
milestone: havana-2 → 2013.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.