nbd device can not be released after injecting files into image if mount operation failed

[wangpan]

I got this issue with stable folsom when I wanted to run a windows8 instance, but I believe this issue still exists in grizzly.
My host distro is debian wheezy, and the version of 'mount' is mount_2.20.1-5.3_amd64.deb.
error logs:
2013-03-07 17:52:50 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf kpartx -a /dev/nbd14 from (pid=19455) execute /usr/local/lib/python2.7/dist-packages/nova/utils.py:192
2013-03-07 17:52:50 DEBUG nova.utils [-] Result was 0 from (pid=19455) execute /usr/local/lib/python2.7/dist-packages/nova/utils.py:207
2013-03-07 17:52:50 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/mapper/nbd14p1 /tmp/openstack-disk-mount-tmp9Plrpc from (pid=19455) execute /usr/local/lib/python2.7/dist-packages/nova/utils.py:192
2013-03-07 17:52:51 DEBUG nova.utils [-] Result was 0 from (pid=19455) execute /usr/local/lib/python2.7/dist-packages/nova/utils.py:207
=============here we can see that the 'umount' operation isn't executed=============
2013-03-07 17:52:51 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf kpartx -d /dev/nbd14 from (pid=19455) execute /usr/local/lib/python2.7/dist-packages/nova/utils.py:192
2013-03-07 17:52:51 DEBUG nova.utils [-] Result was 1 from (pid=19455) execute /usr/local/lib/python2.7/dist-packages/nova/utils.py:207
2013-03-07 17:52:51 WARNING nova.virt.libvirt.driver [-] [instance: 172195e0-7074-450e-9840-937e9173c20b] Ignoring error injecting data into image 4bd31030-7410-4dc4-8c01-71bdc06b7f86 ([Errno 16] Device or resource busy: '/tmp/openstack-disk-mount-tmp9Plrpc')

I have debug this issue inline, and find that error occurs in the 'mount' operation, (===but the nbd device has been mounted!===):
        _out, err = utils.trycmd('mount', self.mapped_device, self.mount_dir,
                                 run_as_root=True)
        if err: ######### err is set by the utils.trycmd() method.
            self.error = _('Failed to mount filesystem: %s') % err
            return False #### return here
so the do_umount method will not call the unmnt_dev method because self.mounted is False:
def do_umount(self):
        """Call the unmnt, unmap and unget operations."""
        if self.mounted:
            self.unmnt_dev()
and then the 'kpartx -d' operation will fail(raise ProcessExecutionError), the 'qemu-nbd -d' operation will not be executed.
Finally, the nbd device used by qemu-nbd isn't released!

[wangpan (hzwangpan)]

And I have try this with mount_2.20.1-4_amd64.deb installed in host, the 'mount' operation is failed with nbd device has NOT been mounted, so the nbd device is released successfully.
I guess this is a bug of 'mount', but it's better that we fix it by changing our nova codes.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/23903

[wangpan (hzwangpan)]

    if err:
            self.error = _('Failed to mount filesystem: %s') % err
            LOG.error(self.error) ### I add error log here
The error message is like this while 'mount':
2013-03-08 17:04:31 ERROR nova.virt.disk.mount [-] Failed to mount filesystem: $LogFile version 2.0 is not supported. (This driver supports version 1.1 only.)
$LogFile version 2.0 is not supported. (This driver supports version 1.1 only.)
Did not find any restart pages in $LogFile and it was not empty.
The file system wasn't safely closed on Windows. Fixing.
and the error code is 0, mount successfully.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/23903
Committed: http://github.com/openstack/nova/commit/786424b4561df6408128e2cd1b79b00c8b638cdc
Submitter: Jenkins
Branch: master

commit 786424b4561df6408128e2cd1b79b00c8b638cdc
Author: Wangpan <email address hidden>
Date: Fri Mar 8 18:05:02 2013 +0800

    Fixes nbd device can't be released error

    When the 'mount' operation is successful but stderr is not empty during
    injecting key/file into image, there may be some warning messages output to
    stderr, so we need to discard them and continue to inject files into image.

    Fixes bug #1152519

    Change-Id: Icc7d15196f3ee67674f9d815ddeef4378f363e2c