Preallocated disks are deducted twice from disk_available_least when using preallocated_images = space

[Lee Yarwood]

Description
===========
When using preallocated file based disks (preallocate_images = space) with the Libvirt virt driver the reported allocation for each disk appears doubled, leaving disk_available_least under reporting the amount of available resources on a compute node.

Originally reported and debugged by mschuppert and mdbooth: https://bugzilla.redhat.com/show_bug.cgi?id=1554265

Steps to reproduce
==================

$ crudini --get /etc/nova/nova-cpu.conf DEFAULT preallocate_images
space

$ nova hypervisor-show eb515aa2-b79e-4f38-a6ea-d493a6e0657f
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
[..]
| disk_available_least | 43 |
| free_disk_gb | 48 |
[..]
| local_gb | 48 |
| local_gb_used | 0 |
[..]
+---------------------------+------------------------------------------+

$ nova flavor-show 2
+----------------------------+----------+
| Property | Value |
+----------------------------+----------+
[..]
| disk | 20 |
[..]
+----------------------------+----------+

$ nova boot --image cirros-0.3.5-x86_64-disk --flavor 2 test
[..]
$ nova hypervisor-show eb515aa2-b79e-4f38-a6ea-d493a6e0657f
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
[..]
| disk_available_least | 3 |
| free_disk_gb | 28 |
[..]
| local_gb | 48 |
| local_gb_used | 20 |
[..]
+---------------------------+------------------------------------------+

Expected result
===============
The allocation for each preallocated disk is reported correctly and removed from disk_available_least.

Actual result
=============
The allocation for each preallocated disk appears doubled and removed from disk_available_least.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   fb0b785169e5e422b06e82f2eb58e68f6d2008b3

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libvirt + KVM

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   file / qcow2

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

Logs & Configs
==============
See above.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/561704

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/561704
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=23bd8f62634707fc9896a38ff4dae606c89c6c4b
Submitter: Zuul
Branch: master

commit 23bd8f62634707fc9896a38ff4dae606c89c6c4b
Author: Lee Yarwood <email address hidden>
Date: Mon Apr 16 18:08:56 2018 +0100

    libvirt: Report the allocated size of preallocated file based disks

    At present the Libvirt driver can preallocate file based disks using the
    fallocate command and importantly the `-n` option. This option allocates
    blocks on the filesystem past the initial EOF of a given file:

    ```
    $ touch test.img ; fallocate -n -l $(( 1024 * 1024 )) test.img
    $ ll -lah test.img
    -rw-rw-r--. 1 stack stack 0 Apr 16 13:28 test.img
    $ du -h test.img
    1.0M test.img
    ```

    This results in a miscalculation of the total disk overcommit for file
    based (excluding ploop) disks as os.path.getsize is currently used to
    determine the allocated size of these disks:

    ```
    >>> import os
    >>> os.path.getsize('test.img')
    0
    ```

    Using the above example the disk overcommit would be reported as 1.0M as
    the disk appears empty yet will report a potential (virtual) size of 1.0M.

    However as the required blocks have already been allocated on the
    filesystem the host will report disk_available_least as missing an
    additional 1.0M, essentially doubling the allocation for each disk.

    To correct this the allocated size of file based (excluding ploop) disks
    is reported using `disk_size` from the `qemu-img info` command. This
    should ensure blocks allocated past the EOF of the file are taken into
    account and correctly reported as allocated.

    A future change should ultimately remove the use of the `-n` option with
    fallocate, however as this would not help disks that have already been
    allocated this has not been included in this change to simplify backports.

    Change-Id: If642e51a4e186833349a8e30b04224a3687f5594
    Closes-bug: #1764489

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/562074

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/562077

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/562078

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/562074
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2d50f6e7854543849c4cb7641ae6b88fe04cb6f6
Submitter: Zuul
Branch: stable/queens

commit 2d50f6e7854543849c4cb7641ae6b88fe04cb6f6
Author: Lee Yarwood <email address hidden>
Date: Mon Apr 16 18:08:56 2018 +0100

    libvirt: Report the allocated size of preallocated file based disks

    At present the Libvirt driver can preallocate file based disks using the
    fallocate command and importantly the `-n` option. This option allocates
    blocks on the filesystem past the initial EOF of a given file:

    ```
    $ touch test.img ; fallocate -n -l $(( 1024 * 1024 )) test.img
    $ ll -lah test.img
    -rw-rw-r--. 1 stack stack 0 Apr 16 13:28 test.img
    $ du -h test.img
    1.0M test.img
    ```

    This results in a miscalculation of the total disk overcommit for file
    based (excluding ploop) disks as os.path.getsize is currently used to
    determine the allocated size of these disks:

    ```
    >>> import os
    >>> os.path.getsize('test.img')
    0
    ```

    Using the above example the disk overcommit would be reported as 1.0M as
    the disk appears empty yet will report a potential (virtual) size of 1.0M.

    However as the required blocks have already been allocated on the
    filesystem the host will report disk_available_least as missing an
    additional 1.0M, essentially doubling the allocation for each disk.

    To correct this the allocated size of file based (excluding ploop) disks
    is reported using `disk_size` from the `qemu-img info` command. This
    should ensure blocks allocated past the EOF of the file are taken into
    account and correctly reported as allocated.

    A future change should ultimately remove the use of the `-n` option with
    fallocate, however as this would not help disks that have already been
    allocated this has not been included in this change to simplify backports.

    Change-Id: If642e51a4e186833349a8e30b04224a3687f5594
    Closes-bug: #1764489
    (cherry picked from commit 23bd8f62634707fc9896a38ff4dae606c89c6c4b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/562077
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d88b75e81eabfbd463007f6a4f27e6966a466530
Submitter: Zuul
Branch: stable/pike

commit d88b75e81eabfbd463007f6a4f27e6966a466530
Author: Lee Yarwood <email address hidden>
Date: Mon Apr 16 18:08:56 2018 +0100

    libvirt: Report the allocated size of preallocated file based disks

    At present the Libvirt driver can preallocate file based disks using the
    fallocate command and importantly the `-n` option. This option allocates
    blocks on the filesystem past the initial EOF of a given file:

    ```
    $ touch test.img ; fallocate -n -l $(( 1024 * 1024 )) test.img
    $ ll -lah test.img
    -rw-rw-r--. 1 stack stack 0 Apr 16 13:28 test.img
    $ du -h test.img
    1.0M test.img
    ```

    This results in a miscalculation of the total disk overcommit for file
    based (excluding ploop) disks as os.path.getsize is currently used to
    determine the allocated size of these disks:

    ```
    >>> import os
    >>> os.path.getsize('test.img')
    0
    ```

    Using the above example the disk overcommit would be reported as 1.0M as
    the disk appears empty yet will report a potential (virtual) size of 1.0M.

    However as the required blocks have already been allocated on the
    filesystem the host will report disk_available_least as missing an
    additional 1.0M, essentially doubling the allocation for each disk.

    To correct this the allocated size of file based (excluding ploop) disks
    is reported using `disk_size` from the `qemu-img info` command. This
    should ensure blocks allocated past the EOF of the file are taken into
    account and correctly reported as allocated.

    A future change should ultimately remove the use of the `-n` option with
    fallocate, however as this would not help disks that have already been
    allocated this has not been included in this change to simplify backports.

    Conflicts:
            nova/tests/unit/virt/libvirt/test_driver.py

    NOTE(lyarwood): I11e329ac5f5fe4b9819fefbcc32ff1ee504fc58b made
    get_domain private in Queens.

    Change-Id: If642e51a4e186833349a8e30b04224a3687f5594
    Closes-bug: #1764489
    (cherry picked from commit 23bd8f62634707fc9896a38ff4dae606c89c6c4b)
    (cherry picked from commit 2d50f6e7854543849c4cb7641ae6b88fe04cb6f6)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/562078
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=caca71376fbc2c1195cda71608c8d2a70ef9a678
Submitter: Zuul
Branch: stable/ocata

commit caca71376fbc2c1195cda71608c8d2a70ef9a678
Author: Lee Yarwood <email address hidden>
Date: Mon Apr 16 18:08:56 2018 +0100

    libvirt: Report the allocated size of preallocated file based disks

    At present the Libvirt driver can preallocate file based disks using the
    fallocate command and importantly the `-n` option. This option allocates
    blocks on the filesystem past the initial EOF of a given file:

    ```
    $ touch test.img ; fallocate -n -l $(( 1024 * 1024 )) test.img
    $ ll -lah test.img
    -rw-rw-r--. 1 stack stack 0 Apr 16 13:28 test.img
    $ du -h test.img
    1.0M test.img
    ```

    This results in a miscalculation of the total disk overcommit for file
    based (excluding ploop) disks as os.path.getsize is currently used to
    determine the allocated size of these disks:

    ```
    >>> import os
    >>> os.path.getsize('test.img')
    0
    ```

    Using the above example the disk overcommit would be reported as 1.0M as
    the disk appears empty yet will report a potential (virtual) size of 1.0M.

    However as the required blocks have already been allocated on the
    filesystem the host will report disk_available_least as missing an
    additional 1.0M, essentially doubling the allocation for each disk.

    To correct this the allocated size of file based (excluding ploop) disks
    is reported using `disk_size` from the `qemu-img info` command. This
    should ensure blocks allocated past the EOF of the file are taken into
    account and correctly reported as allocated.

    A future change should ultimately remove the use of the `-n` option with
    fallocate, however as this would not help disks that have already been
    allocated this has not been included in this change to simplify backports.

    Conflicts:
            nova/tests/unit/virt/libvirt/test_driver.py

    NOTE(lyarwood): I11e329ac5f5fe4b9819fefbcc32ff1ee504fc58b made
    get_domain private in Queens.

    Change-Id: If642e51a4e186833349a8e30b04224a3687f5594
    Closes-bug: #1764489
    (cherry picked from commit 23bd8f62634707fc9896a38ff4dae606c89c6c4b)
    (cherry picked from commit 2d50f6e7854543849c4cb7641ae6b88fe04cb6f6)
    (cherry picked from commit d88b75e81eabfbd463007f6a4f27e6966a466530)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.1.1 release.