nova cannot attach volume whose source_type='block' when setting disk_cache_modes as writethrough or writeback

[Arthur Dayne]

Wheh nova.conf is set as

...

[libvirt]
...
disk_cachemodes = block=writethrough
...

OR

[libvirt]
...
disk_cachemodes = block=writeback
...

Nova cannot attache volume whose source_type is 'block', because of a libvirt issue: https://bugzilla.redhat.com/show_bug.cgi?id=1086704

[Matt Riedemann (mriedem)]

Do you mean source_type='blank'?

[Arthur Dayne (palagend)]

There is hard-code in nova.virt.libvirt.volumes.iscsi:get_config where `conf.driver_io=native`, which will result in a problem when we set `disk_cachemodes = block=writethrough` in /etc/nova/nova.conf.

In this case, when we try to attach a block device provided from cinder whose backend is LVM to a VM, the device cannot be successfully attached actually.

[Brin Zhang (zhangbailin)]

Fix commit: https://review.opendev.org/#/c/682772/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/682772
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=af2405e1181d70cdf60bcd0e40b3e80f2db2e3a6
Submitter: Zuul
Branch: master

commit af2405e1181d70cdf60bcd0e40b3e80f2db2e3a6
Author: Arthur Dayne <email address hidden>
Date: Tue Sep 17 19:08:59 2019 +0800

    libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available

    Because of the libvirt issue[1], there is a bug[2] that if we set cache mode
    whose write semantic is not O_DIRECT (.i.e unsafe, writeback or writethrough),
    there will be a problem with the volume drivers
    (.i.e nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver,
    nova.virt.libvirt.volume.LibvirtNFSVolumeDriver and so on), which designate
    native io explicitly.

    That problem will generate a libvirt xml for the instance,
    whose content contains

    ```
    ...
    <disk ... >
      <driver ... cache='unsafe/writeback/writethrough' io='native' />
    </disk>
    ...
    ```
    In turn, it will fail to start the instance or attach the disk.

    > When qemu is configured with a block device that has aio=native set, but
    > the cache mode doesn't use O_DIRECT (i.e. isn't cache=none/directsync or any
    > unnamed mode with explicit cache.direct=on), then the raw-posix block driver
    > for local files and block devices will silently fall back to aio=threads.
    > The blockdev-add interface rejects such combinations, but qemu can't
    > change the existing legacy interfaces that libvirt uses today.

    [1]: https://github.com/libvirt/libvirt/commit/058384003db776c580d0e5a3016a6384e8eb7b92
    [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1086704

    Closes-Bug: #1841363
    Change-Id: If9acc054100a6733f3659a15dd9fc2d462e84d64

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/747609

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/747610

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/747611

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/747612

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747609
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0bd58921a1fcaffcc4fac25f63434c9cab93b061
Submitter: Zuul
Branch: stable/ussuri

commit 0bd58921a1fcaffcc4fac25f63434c9cab93b061
Author: Arthur Dayne <email address hidden>
Date: Tue Sep 17 19:08:59 2019 +0800

    libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available

    Because of the libvirt issue[1], there is a bug[2] that if we set cache mode
    whose write semantic is not O_DIRECT (.i.e unsafe, writeback or writethrough),
    there will be a problem with the volume drivers
    (.i.e nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver,
    nova.virt.libvirt.volume.LibvirtNFSVolumeDriver and so on), which designate
    native io explicitly.

    That problem will generate a libvirt xml for the instance,
    whose content contains

    ```
    ...
    <disk ... >
      <driver ... cache='unsafe/writeback/writethrough' io='native' />
    </disk>
    ...
    ```
    In turn, it will fail to start the instance or attach the disk.

    > When qemu is configured with a block device that has aio=native set, but
    > the cache mode doesn't use O_DIRECT (i.e. isn't cache=none/directsync or any
    > unnamed mode with explicit cache.direct=on), then the raw-posix block driver
    > for local files and block devices will silently fall back to aio=threads.
    > The blockdev-add interface rejects such combinations, but qemu can't
    > change the existing legacy interfaces that libvirt uses today.

    [1]: https://github.com/libvirt/libvirt/commit/058384003db776c580d0e5a3016a6384e8eb7b92
    [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1086704

    Closes-Bug: #1841363
    Change-Id: If9acc054100a6733f3659a15dd9fc2d462e84d64
    (cherry picked from commit af2405e1181d70cdf60bcd0e40b3e80f2db2e3a6)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/751249

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747610
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d92fe4f3e69db51b7bbad52a1fe1740e37bade9a
Submitter: Zuul
Branch: stable/train

commit d92fe4f3e69db51b7bbad52a1fe1740e37bade9a
Author: Arthur Dayne <email address hidden>
Date: Tue Sep 17 19:08:59 2019 +0800

    libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available

    Because of the libvirt issue[1], there is a bug[2] that if we set cache mode
    whose write semantic is not O_DIRECT (.i.e unsafe, writeback or writethrough),
    there will be a problem with the volume drivers
    (.i.e nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver,
    nova.virt.libvirt.volume.LibvirtNFSVolumeDriver and so on), which designate
    native io explicitly.

    That problem will generate a libvirt xml for the instance,
    whose content contains

    ```
    ...
    <disk ... >
      <driver ... cache='unsafe/writeback/writethrough' io='native' />
    </disk>
    ...
    ```
    In turn, it will fail to start the instance or attach the disk.

    > When qemu is configured with a block device that has aio=native set, but
    > the cache mode doesn't use O_DIRECT (i.e. isn't cache=none/directsync or any
    > unnamed mode with explicit cache.direct=on), then the raw-posix block driver
    > for local files and block devices will silently fall back to aio=threads.
    > The blockdev-add interface rejects such combinations, but qemu can't
    > change the existing legacy interfaces that libvirt uses today.

    [1]: https://github.com/libvirt/libvirt/commit/058384003db776c580d0e5a3016a6384e8eb7b92
    [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1086704

    Closes-Bug: #1841363
    Change-Id: If9acc054100a6733f3659a15dd9fc2d462e84d64
    (cherry picked from commit af2405e1181d70cdf60bcd0e40b3e80f2db2e3a6)
    (cherry picked from commit 0bd58921a1fcaffcc4fac25f63434c9cab93b061)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747611
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cc2f45ebb03326c757abbfaef4de3e2c1fa4ead2
Submitter: Zuul
Branch: stable/stein

commit cc2f45ebb03326c757abbfaef4de3e2c1fa4ead2
Author: Arthur Dayne <email address hidden>
Date: Tue Sep 17 19:08:59 2019 +0800

    libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available

    Because of the libvirt issue[1], there is a bug[2] that if we set cache mode
    whose write semantic is not O_DIRECT (.i.e unsafe, writeback or writethrough),
    there will be a problem with the volume drivers
    (.i.e nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver,
    nova.virt.libvirt.volume.LibvirtNFSVolumeDriver and so on), which designate
    native io explicitly.

    That problem will generate a libvirt xml for the instance,
    whose content contains

    ```
    ...
    <disk ... >
      <driver ... cache='unsafe/writeback/writethrough' io='native' />
    </disk>
    ...
    ```
    In turn, it will fail to start the instance or attach the disk.

    > When qemu is configured with a block device that has aio=native set, but
    > the cache mode doesn't use O_DIRECT (i.e. isn't cache=none/directsync or any
    > unnamed mode with explicit cache.direct=on), then the raw-posix block driver
    > for local files and block devices will silently fall back to aio=threads.
    > The blockdev-add interface rejects such combinations, but qemu can't
    > change the existing legacy interfaces that libvirt uses today.

    [1]: https://github.com/libvirt/libvirt/commit/058384003db776c580d0e5a3016a6384e8eb7b92
    [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1086704

    Closes-Bug: #1841363
    Change-Id: If9acc054100a6733f3659a15dd9fc2d462e84d64
    (cherry picked from commit af2405e1181d70cdf60bcd0e40b3e80f2db2e3a6)
    (cherry picked from commit 0bd58921a1fcaffcc4fac25f63434c9cab93b061)
    (cherry picked from commit d92fe4f3e69db51b7bbad52a1fe1740e37bade9a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/747612
Committed: https://opendev.org/openstack/nova/commit/6e6440fe97c07489ab6b70029a22fa0b9b70bc02
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit 6e6440fe97c07489ab6b70029a22fa0b9b70bc02
Author: Arthur Dayne <email address hidden>
Date: Tue Sep 17 19:08:59 2019 +0800

    libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available

    Because of the libvirt issue[1], there is a bug[2] that if we set cache mode
    whose write semantic is not O_DIRECT (.i.e unsafe, writeback or writethrough),
    there will be a problem with the volume drivers
    (.i.e nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver,
    nova.virt.libvirt.volume.LibvirtNFSVolumeDriver and so on), which designate
    native io explicitly.

    That problem will generate a libvirt xml for the instance,
    whose content contains

    ```
    ...
    <disk ... >
      <driver ... cache='unsafe/writeback/writethrough' io='native' />
    </disk>
    ...
    ```
    In turn, it will fail to start the instance or attach the disk.

    > When qemu is configured with a block device that has aio=native set, but
    > the cache mode doesn't use O_DIRECT (i.e. isn't cache=none/directsync or any
    > unnamed mode with explicit cache.direct=on), then the raw-posix block driver
    > for local files and block devices will silently fall back to aio=threads.
    > The blockdev-add interface rejects such combinations, but qemu can't
    > change the existing legacy interfaces that libvirt uses today.

    [1]: https://github.com/libvirt/libvirt/commit/058384003db776c580d0e5a3016a6384e8eb7b92
    [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1086704

    Closes-Bug: #1841363
    Change-Id: If9acc054100a6733f3659a15dd9fc2d462e84d64
    (cherry picked from commit af2405e1181d70cdf60bcd0e40b3e80f2db2e3a6)
    (cherry picked from commit 0bd58921a1fcaffcc4fac25f63434c9cab93b061)
    (cherry picked from commit d92fe4f3e69db51b7bbad52a1fe1740e37bade9a)
    (cherry picked from commit cc2f45ebb03326c757abbfaef4de3e2c1fa4ead2)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova rocky-eol release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/nova/+/751249
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.