Out of space when migrating encrypted volume between two LVM backends

Bug #1720885 reported by Eric Harney on 2017-10-02
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Cinder
High
Yeshwanth Allampati

Bug Description

Deployed devstack w/ Barbican.

$ cinder show d0d68a2e-7687-4d8f-a376-20d870e7708a
+--------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------+-------------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2017-10-02T21:12:10.000000 |
| description | None |
| encrypted | True |
| id | d0d68a2e-7687-4d8f-a376-20d870e7708a |
| metadata | |
| migration_status | error |
| multiattach | False |
| name | None |
| os-vol-host-attr:host | fedora25dev.localdomain@lvmdriver-1#lvmdriver-1 |
| os-vol-mig-status-attr:migstat | error |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | bb2142eaba20414d87583fe4b0d3efdf |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2017-10-02T21:22:55.000000 |
| user_id | b7ada84e4fa247ef8de596c0e66d3c91 |
| volume_image_metadata | checksum : f8ab98ff5e73ebab884d80c9dc9c7290 |
| | container_format : bare |
| | disk_format : qcow2 |
| | image_id : 43afb53b-2f4c-4d14-a3f1-e08712e9cf34 |
| | image_name : cirros-0.3.5-x86_64-disk |
| | min_disk : 0 |
| | min_ram : 0 |
| | size : 13267968 |
| volume_type | lvmdriver-1 |
+--------------------------------+-------------------------------------------------+

$ cinder migrate --force-host-copy True d0d68a2e-7687-4d8f-a376-20d870e7708a fedora25dev.localdomain@lvmdriver-2

2017-10-02 17:16:08.973 TRACE cinder.volume.manager Traceback (most recent call last):
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2094, in _migrate_volume_generic
2017-10-02 17:16:08.973 TRACE cinder.volume.manager self._copy_volume_data(ctxt, volume, new_volume, remote='dest')
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2011, in _copy_volume_data
2017-10-02 17:16:08.973 TRACE cinder.volume.manager {'src': src_vol['id'], 'dest': dest_vol['id']})
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-02 17:16:08.973 TRACE cinder.volume.manager self.force_reraise()
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-02 17:16:08.973 TRACE cinder.volume.manager six.reraise(self.type_, self.value, self.tb)
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 2006, in _copy_volume_data
2017-10-02 17:16:08.973 TRACE cinder.volume.manager sparse=sparse_copy_volume)
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/utils.py", line 585, in copy_volume
2017-10-02 17:16:08.973 TRACE cinder.volume.manager sparse=sparse)
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/volume/utils.py", line 469, in _copy_volume_with_path
2017-10-02 17:16:08.973 TRACE cinder.volume.manager execute(*cmd, run_as_root=True)
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/opt/stack/cinder/cinder/utils.py", line 123, in execute
2017-10-02 17:16:08.973 TRACE cinder.volume.manager return processutils.execute(*cmd, **kwargs)
2017-10-02 17:16:08.973 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 400, in execute
2017-10-02 17:16:08.973 TRACE cinder.volume.manager cmd=sanitized_cmd)
2017-10-02 17:16:08.973 TRACE cinder.volume.manager ProcessExecutionError: Unexpected error while running command.
2017-10-02 17:16:08.973 TRACE cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/disk/by-id/scsi-36001405dae6f7cd109c44a8a280c0ed3 of=/dev/disk/by-id/scsi-3600140557a05fbd6a5749abb7b47c58a count=1073741824 bs=1M iflag=count_bytes,direct oflag=direct conv=sparse
2017-10-02 17:16:08.973 TRACE cinder.volume.manager Exit code: 1
2017-10-02 17:16:08.973 TRACE cinder.volume.manager Stdout: u''
2017-10-02 17:16:08.973 TRACE cinder.volume.manager Stderr: u"/bin/dd: error writing '/dev/disk/by-id/scsi-3600140557a05fbd6a5749abb7b47c58a': No space left on device\n1022+1 records in\n1022+0 records out\n1071644672 bytes (1.1 GB, 1022 MiB) copied, 38.9663 s, 27.5 MB/s\n"
2017-10-02 17:16:08.973 TRACE cinder.volume.manager

Eric Harney (eharney) on 2017-10-02
description: updated
Eric Harney (eharney) on 2017-11-29
tags: added: encryption

Hi all

Is any one working on it? I am willing to take this issue.

Thanks

Sean McGinnis (sean-mcginnis) wrote :

Vishakha, if you are still able to work on this, please reassign the owner to you and proceed. Thanks!

Changed in cinder:
assignee: Eric Harney (eharney) → Mauricio Pesci da Silva (mauricio.pesci)
yenai (yenai2008) wrote :

The reason is luks-volume contains LUKS header in the device, it costs 2M (512bytes * 4096sectors).

[root@localhost ~]# cryptsetup status /dev/mapper/crypt-scsi-36001405eb821aa8e66e44ecbfec83dfa
/dev/mapper/crypt-scsi-36001405eb821aa8e66e44ecbfec83dfa is active and is in use.
  type: LUKS1
  cipher: aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device: /dev/sde
  sector size: 512
  offset: 4096 sectors
  size: 2093056 sectors
  mode: read/write

[root@localhost ~]# cryptsetup luksDump /dev/sde
LUKS header information for /dev/sde

Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 512
MK digest: 5d 6a d7 86 41 d0 f4 0a dd 55 c7 c6 ff f5 ed 63 76 e9 61 fd
MK salt: 77 c2 88 b6 a0 7e cb 18 f2 79 63 09 19 14 dc 76
                40 8b 0f d5 eb aa 0d ef ba 85 e3 58 75 48 bc df
MK iterations: 15784
UUID: 83431299-e4c5-4999-b007-973425725ace

Key Slot 0: ENABLED
        Iterations: 279470
        Salt: 8e 57 e5 c6 d2 65 d2 da 94 bb cb a7 65 e3 2c 6d
                                d1 46 7d 30 0e 1f d1 33 89 39 f9 97 30 54 aa 17
        Key material offset: 8
        AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Changed in cinder:
assignee: Mauricio Pesci da Silva (mauricio.pesci) → Yeshwanth Allampati (yeshwanth-allampati)
assignee: Yeshwanth Allampati (yeshwanth-allampati) → nobody
assignee: nobody → Yeshwanth Allampati (yeshwanth-allampati)
description: updated

If this is an Encrypted Volume, why is the volumetype lvmdriver and not LUKS?

Download full text (4.7 KiB)

I configured an LVM Backend and tried this out and this worked fine for me.

cinder show 3f83570d-b1a0-4c91-9773-2b697c1e982d

+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-08-02T08:17:09.000000 |
| description | None |
| encrypted | True |
| id | 3f83570d-b1a0-4c91-9773-2b697c1e982d |
| metadata | |
| migration_status | success |
| multiattach | False |
| name | encrypted volume |
| os-vol-host-attr:host | infra@lvmdriver-1#lvmdriver-1 |
| os-vol-mig-status-attr:migstat | success |
| os-vol-mig-status-attr:name_id | e9b415df-1ec7-4e41-a7c8-d9f4c8a39ba5 |
| os-vol-tenant-attr:tenant_id | 5cb1426e43cd4f2db6bb428a6051622a |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2019-08-02T09:56:23.000000 |
| user_id | d645bb574c394feeb2390140ed981f8c |
| volume_type | LUKS |
+--------------------------------+--------------------------------------+

I executed the command

cinder migrate --force-host-copy True 3f83570d-b1a0-4c91-9773-2b697c1e982d infra@lvmdriver-3
Request to migrate volume 3f83570d-b1a0-4c91-9773-2b697c1e982d has been accepted.

After the command.. I got

cinder show 3f83570d-b1a0-4c91-9773-2b697c1e982d
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-08-02T08:17:09.00...

Read more...

@Eric Harney .. The /etc/cinder/cinder.conf is very much needed to understand the root cause of the problem.

Analysis of lvm backend configuration is needed.

Changed in cinder:
status: New → In Progress

This worked fine for me but the Duplicate of this bug i.e.,

https://bugs.launchpad.net/cinder/+bug/1824321

got reproduced. Working on it.

Change abandoned by Sean McGinnis (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/682355
Reason: Abandoning since there are issues here that are just going to cause test failures, and needs to be proposed to master first before backporting to stein.

Changed in cinder:
status: In Progress → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers