hard reboot fails with preallocate_images=performance

[Bolke de Bruin]

I have in /etc/nova/nova.conf

preallocate_images=performance

for obvious speed reasons.

I had the need to hard reboot some instances (somehow soft reboot does not seem to work often with me) and I got this error in the logs for every instance:

013-07-10 18:53:40.638 DEBUG nova.utils [req-911ae4d0-6a75-416f-b3d7-b281793cbfdc 93178802ad6f48e8aea05cc7ba195859 876e74dbf00f44418c6a61d83c5b7ce8] Running cmd (subprocess): fallocate -n -l 42949672960 /var/lib/nova/instances/abbb258d-f851-4f4e-81ee-0b8334d331d2/disk execute /usr/lib/python2.7/dist-packages/nova/utils.py:208
2013-07-10 18:53:40.652 DEBUG nova.utils [req-911ae4d0-6a75-416f-b3d7-b281793cbfdc 93178802ad6f48e8aea05cc7ba195859 876e74dbf00f44418c6a61d83c5b7ce8] Result was 1 execute /usr/lib/python2.7/dist-packages/nova/utils.py:232
2013-07-10 18:53:40.653 ERROR nova.compute.manager [req-911ae4d0-6a75-416f-b3d7-b281793cbfdc 93178802ad6f48e8aea05cc7ba195859 876e74dbf00f44418c6a61d83c5b7ce8] [instance: abbb258d-f851-4f4e-81ee-0b8334d331d2] Cannot reboot instance: Unexpected error while running command.
Command: fallocate -n -l 42949672960 /var/lib/nova/instances/abbb258d-f851-4f4e-81ee-0b8334d331d2/disk
Exit code: 1
Stdout: ''
Stderr: 'fallocate: /var/lib/nova/instances/abbb258d-f851-4f4e-81ee-0b8334d331d2/disk: open failed: Permission denied\n'

Basically the hard reboot tries to re-allocate the disk space (which - in this case? - is already allocated) and fails because the disk file is owned by root (I think) rather than nova. Doing a chown nova:nova on the disk files resolves the issue for me.

I see three (combined) options:

1) do not try to reallocate in case it is already the right size
2) execute the fallocate part under root
3) make sure the disk is owned/writeable by nova

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/40864

[Jerico Revote (jrevote)]

Hi,

Any news about the patch?

J

[Jiajun Liu (ljjjustin)]

yes, I restored and push new commit today.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/40864
Committed: http://github.com/openstack/nova/commit/c246fb37e20197be27264a36515725233085df1d
Submitter: Jenkins
Branch: master

commit c246fb37e20197be27264a36515725233085df1d
Author: Jiajun Liu <email address hidden>
Date: Thu Aug 22 11:45:18 2013 +0800

    fallocate image only when user has write access

    Libvirt will change instance's ownership when the instance became power
    off which will make nova-compute service unable to hard reboot instance
    bacause the nova-compute service has not write access. This commit will
    allocate space for image only the first time we create image.

    fixes bug 1200113

    Change-Id: Id81b09ade5f870665f5ffabab3cde144c6dec210

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/50565

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/50932

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/50565
Committed: http://github.com/openstack/nova/commit/317c4c618c6db65e951d16757bb5871519388f68
Submitter: Jenkins
Branch: stable/grizzly

commit 317c4c618c6db65e951d16757bb5871519388f68
Author: Jiajun Liu <email address hidden>
Date: Thu Aug 22 11:45:18 2013 +0800

    fallocate image only when user has write access

    Libvirt will change instance's ownership when the instance became power
    off which will make nova-compute service unable to hard reboot instance
    bacause the nova-compute service has not write access. This commit will
    allocate space for image only the first time we create image.

    Fixes bug: 1200113
    Conflicts: nova/tests/test_imagebackend.py
    Change-Id: Id81b09ade5f870665f5ffabab3cde144c6dec210
    Cherry-picked: c246fb37e20197be27264a36515725233085df1d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/50932
Committed: http://github.com/openstack/nova/commit/19e2b7ddfc7035565e75444e5813a9d195c85bfb
Submitter: Jenkins
Branch: milestone-proposed

commit 19e2b7ddfc7035565e75444e5813a9d195c85bfb
Author: Jiajun Liu <email address hidden>
Date: Thu Aug 22 11:45:18 2013 +0800

    fallocate image only when user has write access

    Libvirt will change instance's ownership when the instance became power
    off which will make nova-compute service unable to hard reboot instance
    bacause the nova-compute service has not write access. This commit will
    allocate space for image only the first time we create image.

    fixes bug 1200113

    Change-Id: Id81b09ade5f870665f5ffabab3cde144c6dec210
    (cherry picked from commit c246fb37e20197be27264a36515725233085df1d)