available volume from no-encrytion to luks retype failed

[yenai]

[stack@localhost devstack]$ cinder encryption-type-show luks-type
+--------------------------------------+----------+-----------------+----------+------------------+
| Volume Type ID | Provider | Cipher | Key Size | Control Location |
+--------------------------------------+----------+-----------------+----------+------------------+
| 0cfcee9d-20cb-4552-a440-d8203f8e11ad | luks | aes-xts-plain64 | 512 | front-end |
+--------------------------------------+----------+-----------------+----------+------------------+

[stack@localhost devstack]$ cinder show 9c91062c-a16d-44d4-b01d-7e45aa37797e
+--------------------------------+-----------------------------------------------+
| Property | Value |
+--------------------------------+-----------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-01-13T08:55:32.000000 |
| description | |
| encrypted | False |
| id | 9c91062c-a16d-44d4-b01d-7e45aa37797e |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | yenai-001 |
| os-vol-host-attr:host | localhost.localdomain@lvmdriver-1#lvmdriver-1 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | b984b88d70dd4cbb8f99efa4a48ebfe9 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2019-01-13T08:55:33.000000 |
| user_id | edffe81b43534dfea48effced3ac8e53 |
| volume_type | lvmdriver-1 |
+--------------------------------+-----------------------------------------------+

[stack@localhost devstack]$ cinder retype --migration-policy on-demand 9c91062c-a16d-44d4-b01d-7e45aa37797e luks-type

ERROR MSG:
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager [req-7a0380ea-987a-4803-a76b-c1edbb5a4ca8 req-f2187717-b025-4fb9-8037-f10b95d4695b admin None] Failed to copy volume 9c91062c-a16d-44d4-b01d-7e45aa37797e to bc12e2ee-cecc-4e5a-b6b0-8fc7678a66f3: ProcessExecutionError: Unexpected error while running command.
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/sdh of=/dev/disk/by-id/scsi-36001405d4d948676c98448f9d10a2217 count=1073741824 bs=1M iflag=count_bytes,direct oflag=direct conv=sparse
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: Exit code: 1
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: Stdout: u''
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: Stderr: '/bin/dd: error writing \xe2\x80\x98/dev/disk/by-id/scsi-36001405d4d948676c98448f9d10a2217\xe2\x80\x99: No space left on device\n1023+0 records in\n1022+0 records out\n1071644672 bytes (1.1 GB) copied, 3.06053 s, 350 MB/s\n'
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager Traceback (most recent call last):
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2166, in _migrate_volume_generic
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager self._copy_volume_data(ctxt, volume, new_volume, remote='dest')
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2081, in _copy_volume_data
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager {'src': src_vol['id'], 'dest': dest_vol['id']})
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager self.force_reraise()
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager six.reraise(self.type_, self.value, self.tb)
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2077, in _copy_volume_data
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager sparse=sparse_copy_volume)
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/utils.py", line 602, in copy_volume
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager sparse=sparse)
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/utils.py", line 486, in _copy_volume_with_path
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager execute(*cmd, run_as_root=True)
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 128, in execute
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager return processutils.execute(*cmd, **kwargs)
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager cmd=sanitized_cmd)
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager ProcessExecutionError: Unexpected error while running command.
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/sdh of=/dev/disk/by-id/scsi-36001405d4d948676c98448f9d10a2217 count=1073741824 bs=1M iflag=count_bytes,direct oflag=direct conv=sparse
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager Exit code: 1
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager Stdout: u''
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager Stderr: '/bin/dd: error writing \xe2\x80\x98/dev/disk/by-id/scsi-36001405d4d948676c98448f9d10a2217\xe2\x80\x99: No space left on device\n1023+0 records in\n1022+0 records out\n1071644672 bytes (1.1 GB) copied, 3.06053 s, 350 MB/s\n'
Jan 13 19:00:46 localhost.localdomain cinder-volume[46318]: ERROR cinder.volume.manager

====================================================================================================
The reason is luks-volume contains LUKS header in the device, it costs 2M (512bytes * 4096sectors).

[root@localhost ~]# cryptsetup status /dev/mapper/crypt-scsi-36001405eb821aa8e66e44ecbfec83dfa
/dev/mapper/crypt-scsi-36001405eb821aa8e66e44ecbfec83dfa is active and is in use.
  type: LUKS1
  cipher: aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device: /dev/sde
  sector size: 512
  offset: 4096 sectors
  size: 2093056 sectors
  mode: read/write

[root@localhost ~]# cryptsetup luksDump /dev/sde
LUKS header information for /dev/sde

Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 512
MK digest: 5d 6a d7 86 41 d0 f4 0a dd 55 c7 c6 ff f5 ed 63 76 e9 61 fd
MK salt: 77 c2 88 b6 a0 7e cb 18 f2 79 63 09 19 14 dc 76
                40 8b 0f d5 eb aa 0d ef ba 85 e3 58 75 48 bc df
MK iterations: 15784
UUID: 83431299-e4c5-4999-b007-973425725ace

Key Slot 0: ENABLED
        Iterations: 279470
        Salt: 8e 57 e5 c6 d2 65 d2 da 94 bb cb a7 65 e3 2c 6d
                                d1 46 7d 30 0e 1f d1 33 89 39 f9 97 30 54 aa 17
        Key material offset: 8
        AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

[yenai (yenai2008)]

When use plain, it's OK.

[root@localhost ~]# cryptsetup status "/dev/disk/by-id/scsi-3600140548df84c9a1704122a77658534"
/dev/disk/by-id/scsi-3600140548df84c9a1704122a77658534 is active and is in use.
  type: PLAIN
  cipher: aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device: /dev/sdf
  sector size: 512
  offset: 0 sectors
  size: 2097152 sectors
  mode: read/write

[Eric Harney (eharney)]

Sounds related to
https://bugs.launchpad.net/cinder/+bug/1720885

[Yeshwanth Allampati (yeshwanth-allampati)]

Hi... bug #1720885 went on well on my Devstack setup. But I was able to reproduce this one.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/682339

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/682355

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean McGinnis (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/682355
Reason: Abandoning since there are issues here that are just going to cause test failures, and needs to be proposed to master first before backporting to stein.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Yeshwanth Allampati <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/682339