libvirt.libvirtError: internal error: unable to execute QEMU command 'object-add': Incorrect number of padding bytes

[Lee Yarwood]

Description
===========
When attempting to attach an encrypted volume the following trace is logged [1]:

File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py",
line 1463, in attach_volume
     guest.attach_device(conf, persistent=True, live=live)
   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py",
line 303, in attach_device
     self._domain.attachDeviceFlags(device_xml, flags=flags)
   File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 186, in
doit
     result = proxy_call(self._autowrap, f, *args, **kwargs)
   File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 144, in
proxy_call
     rv = execute(f, *args, **kwargs)
   File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 125, in
execute
     six.reraise(c, e, tb)
   File "/usr/lib/python3/dist-packages/eventlet/support/six.py", line
625, in reraise
     raise value
   File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 83, in
tworker
     rv = meth(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/libvirt.py", line 585, in
attachDeviceFlags
     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags()
failed', dom=self)
 libvirt.libvirtError: internal error: unable to execute QEMU command
'object-add': Incorrect number of padding bytes (57) found on decrypted data

This is due to a known libvirt issue detailed in the following bug report and fix:

Unable to use LUKS passphrase that is exactly 16 bytes long
https://bugzilla.redhat.com/show_bug.cgi?id=1447297

[libvirt] [PATCH] Fix padding of encrypted data
https://www.redhat.com/archives/libvir-list/2017-May/msg00030.html

Nova should log a breadcrumb when this issue is encountered directing the operator to update Libvirt to resolve this issue.

[1] http://lists.openstack.org/pipermail/openstack-dev/2018-June/131653.html

Steps to reproduce
==================
Using a version of Libvirt without the above fix simply attempt to attach and encrypted volume using the native LUKS decryption feature in Nova.

Expected result
===============
Breadcrumb line logged if this issue is hit.

Actual result
=============
Only the trace is logged.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   Current Rocky master.

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libvirt

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   N/A

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/577164

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/577164
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=78891c2305bff6e16706339a9c5eca99a84e409c
Submitter: Zuul
Branch: master

commit 78891c2305bff6e16706339a9c5eca99a84e409c
Author: Lee Yarwood <email address hidden>
Date: Wed Jun 20 18:06:13 2018 +0100

    libvirt: Log breadcrumb for known encryption bug

    The initial implementation of native LUKS support within Libvirt
    introduced a small issue when using a passphrase that is a multiple of
    16 bytes in size. This is documented in the following bug and associated
    patch posted to the Libvirt development list:

    Unable to use LUKS passphrase that is exactly 16 bytes long
    https://bugzilla.redhat.com/show_bug.cgi?id=1447297

    [libvirt] [PATCH] Fix padding of encrypted data
    https://www.redhat.com/archives/libvir-list/2017-May/msg00030.html

    This change introduces a known issue release note and logs an additional
    breadcrumb when we appear to hit this with pointers to the above.

    Closes-Bug: #1778044
    Change-Id: Id346bce6e47431988cce7001abcf29a9faf2936a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/577757

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/577757
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=875afe92086cf60a701abeb45574802feed9329e
Submitter: Zuul
Branch: stable/queens

commit 875afe92086cf60a701abeb45574802feed9329e
Author: Lee Yarwood <email address hidden>
Date: Wed Jun 20 18:06:13 2018 +0100

    libvirt: Log breadcrumb for known encryption bug

    The initial implementation of native LUKS support within Libvirt
    introduced a small issue when using a passphrase that is a multiple of
    16 bytes in size. This is documented in the following bug and associated
    patch posted to the Libvirt development list:

    Unable to use LUKS passphrase that is exactly 16 bytes long
    https://bugzilla.redhat.com/show_bug.cgi?id=1447297

    [libvirt] [PATCH] Fix padding of encrypted data
    https://www.redhat.com/archives/libvir-list/2017-May/msg00030.html

    This change introduces a known issue release note and logs an additional
    breadcrumb when we appear to hit this with pointers to the above.

    Closes-Bug: #1778044
    Change-Id: Id346bce6e47431988cce7001abcf29a9faf2936a
    (cherry picked from commit 78891c2305bff6e16706339a9c5eca99a84e409c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.6 release.