The computing node has left the mount information after the encryption fails.

Bug #1812945 reported by yenai on 2019-01-23
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
yenai
Queens
Medium
Matt Riedemann
Rocky
Medium
Matt Riedemann
Stein
Medium
Matt Riedemann

Bug Description

[stack@localhost devstack]$ cinder type-create encrypt-NoOpEncryptor
+--------------------------------------+-----------------------+-------------+-----------+
| ID | Name | Description | Is_Public |
+--------------------------------------+-----------------------+-------------+-----------+
| 054fad16-6b6b-4426-ac9b-af63bc43d113 | encrypt-NoOpEncryptor | - | True |
+--------------------------------------+-----------------------+-------------+-----------+
[stack@localhost devstack]$ cinder encryption-type-create 054fad16-6b6b-4426-ac9b-af63bc43d113 --control-location front-end NoOpEncryptor
+--------------------------------------+---------------+--------+----------+------------------+
| Volume Type ID | Provider | Cipher | Key Size | Control Location |
+--------------------------------------+---------------+--------+----------+------------------+
| 054fad16-6b6b-4426-ac9b-af63bc43d113 | NoOpEncryptor | - | - | front-end |
+--------------------------------------+---------------+--------+----------+------------------+
[stack@localhost devstack]$ cinder create --volume-type encrypt-NoOpEncryptor --name yenai 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-01-12T01:36:36.000000 |
| description | None |
| encrypted | True |
| id | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | yenai |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 9b881cf6b92049b8a155178274662836 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | 657205033877433db3dd8dadb212d9fb |
| volume_type | encrypt-NoOpEncryptor |
+--------------------------------+--------------------------------------+
[stack@localhost libvirt]$ cinder list
+--------------------------------------+-----------+-------+------+-----------------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------+------+-----------------------+----------+--------------------------------------+
| 247bd974-3cb5-44a5-9b7d-599ca2fc9bda | available | yenai | 1 | encrypt-NoOpEncryptor | false | |
| f1c15752-c247-470b-9a5f-1f5559f51ce7 | in-use | yenai | 1 | encrypt-luks | false | 6ce2254a-0e05-4e63-874e-44d112bed7d0 |
+--------------------------------------+-----------+-------+------+-----------------------+----------+--------------------------------------+
[[stack@localhost devstack]$ nova volume-attach 6ce2254a-0e05-4e63-874e-44d112bed7d0 247bd974-3cb5-44a5-9b7d-599ca2fc9bda
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdc |
| id | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
| serverId | 6ce2254a-0e05-4e63-874e-44d112bed7d0 |
| volumeId | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
+----------+--------------------------------------+
[stack@localhost devstack]$

ERROR:
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [None req-756c266a-47e7-4173-b06f-c4a60c5dcd55 admin admin] [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] Driver failed to attach volume 247bd974-3cb5-44a5-9b7d-599ca2fc9bda at /dev/vdc: TypeError: attach_volume() got an unexpected keyword argument 'cipher'
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] Traceback (most recent call last):
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/block_device.py", line 559, in _volume_attach
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] device_type=self['device_type'], encryption=encryption)
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1460, in attach_volume
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] encryption=encryption)
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1260, in _connect_volume
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] allow_native_luks)
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1395, in _attach_encryptor
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] encryptor.attach_volume(context, **encryption)
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] TypeError: attach_volume() got an unexpected keyword argument 'cipher'
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]

[stack@localhost encryptors]$ nova show 6ce2254a-0e05-4e63-874e-44d112bed7d0
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | localhost.localdomain |
| OS-EXT-SRV-ATTR:hostname | yenai |
| OS-EXT-SRV-ATTR:hypervisor_hostname | localhost.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-m8xrsuw9 |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-01-11T22:22:00.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2019-01-11T22:21:51Z |
| description | - |
| flavor:disk | 1 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {} |
| flavor:original_name | m1.tiny |
| flavor:ram | 512 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | b3a5a7c35a7b6862a03b3bdca5defe64fef34763e1440a3cfb4aa2ad |
| host_status | UP |
| id | 6ce2254a-0e05-4e63-874e-44d112bed7d0 |
| image | cirros-0.3.6-x86_64-disk (c10be70f-38a7-4b58-a033-8c045a8c4645) |
| key_name | - |
| locked | False |
| metadata | {} |
| name | yenai |
| os-extended-volumes:volumes_attached | [{"id": "f1c15752-c247-470b-9a5f-1f5559f51ce7", "delete_on_termination": false}] |
| progress | 0 |
| public network | 2001:db8::1d2, 172.24.4.104 |
| security_groups | default |
| status | ACTIVE |
| tags | [] |
| tenant_id | 9b881cf6b92049b8a155178274662836 |
| trusted_image_certificates | - |
| updated | 2019-01-11T22:22:00Z |
| user_id | 657205033877433db3dd8dadb212d9fb |
+--------------------------------------+----------------------------------------------------------------------------------+
[stack@localhost encryptors]$ cinder show 247bd974-3cb5-44a5-9b7d-599ca2fc9bda
+--------------------------------+-----------------------------------------------+
| Property | Value |
+--------------------------------+-----------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-01-12T01:36:36.000000 |
| description | None |
| encrypted | True |
| id | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | yenai |
| os-vol-host-attr:host | localhost.localdomain@lvmdriver-1#lvmdriver-1 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 9b881cf6b92049b8a155178274662836 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2019-01-12T05:18:36.000000 |
| user_id | 657205033877433db3dd8dadb212d9fb |
| volume_type | encrypt-NoOpEncryptor |
+--------------------------------+-----------------------------------------------+
[stack@localhost encryptors]$ ls -lh /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root 9 Jan 12 13:18 ip-192.168.43.139:3260-iscsi-iqn.2010-10.org.openstack:volume-247bd974-3cb5-44a5-9b7d-599ca2fc9bda-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root 9 Jan 12 06:30 ip-192.168.43.139:3260-iscsi-iqn.2010-10.org.openstack:volume-f1c15752-c247-470b-9a5f-1f5559f51ce7-lun-0 -> ../../sdb
lrwxrwxrwx. 1 root root 9 Jan 11 15:42 pci-0000:00:07.1-ata-2.0 -> ../../sr0
lrwxrwxrwx. 1 root root 9 Jan 11 15:42 pci-0000:00:10.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx. 1 root root 10 Jan 11 15:42 pci-0000:00:10.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx. 1 root root 10 Jan 11 15:42 pci-0000:00:10.0-scsi-0:0:0:0-part2 -> ../../sda2

yenai (yenai2008) on 2019-01-23
Changed in nova:
assignee: nobody → yenai (yenai2008)

Fix proposed to branch: master
Review: https://review.openstack.org/632598

Changed in nova:
status: New → In Progress
yenai (yenai2008) on 2019-01-23
description: updated
tags: added: cinder volumes
Lee Yarwood (lyarwood) wrote :

I wonder if we also need to drop the NoOpEncryptor?

yenai (yenai2008) wrote :

I think so @Lee Yarwood (lyarwood)

Matt Riedemann (mriedem) on 2019-04-05
Changed in nova:
importance: Undecided → Medium
tags: added: libvirt
Changed in nova:
assignee: yenai (yenai2008) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem) wrote :

Feel free to backport this since it was introduced in Queens.

Changed in nova:
assignee: Matt Riedemann (mriedem) → yenai (yenai2008)

Reviewed: https://review.openstack.org/632598
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=79bcb4e21b7909f3e989d4fbcf68ac268680b865
Submitter: Zuul
Branch: master

commit 79bcb4e21b7909f3e989d4fbcf68ac268680b865
Author: yenai <email address hidden>
Date: Wed Jan 23 11:02:33 2019 +0800

    libvirt: disconnect volume when encryption fails

    The compute node has left the attachment information after the
    encryption fails. This patch fixes this by ensuring the volume is
    disconnected when an exception occurs.

    Change-Id: I9d652f182d83a2557ff6ed0b21c69a735c3f9840
    Closes-Bug: #1812945

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/650931
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d773e3b1469a76f129d2f7a4edab68d007f3f928
Submitter: Zuul
Branch: stable/stein

commit d773e3b1469a76f129d2f7a4edab68d007f3f928
Author: yenai <email address hidden>
Date: Wed Jan 23 11:02:33 2019 +0800

    libvirt: disconnect volume when encryption fails

    The compute node has left the attachment information after the
    encryption fails. This patch fixes this by ensuring the volume is
    disconnected when an exception occurs.

    Change-Id: I9d652f182d83a2557ff6ed0b21c69a735c3f9840
    Closes-Bug: #1812945
    (cherry picked from commit 79bcb4e21b7909f3e989d4fbcf68ac268680b865)

Reviewed: https://review.openstack.org/651796
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2cde44659c42ebab25259e2b05cd5c2ffe38c131
Submitter: Zuul
Branch: stable/rocky

commit 2cde44659c42ebab25259e2b05cd5c2ffe38c131
Author: yenai <email address hidden>
Date: Wed Jan 23 11:02:33 2019 +0800

    libvirt: disconnect volume when encryption fails

    The compute node has left the attachment information after the
    encryption fails. This patch fixes this by ensuring the volume is
    disconnected when an exception occurs.

    Change-Id: I9d652f182d83a2557ff6ed0b21c69a735c3f9840
    Closes-Bug: #1812945
    (cherry picked from commit 79bcb4e21b7909f3e989d4fbcf68ac268680b865)
    (cherry picked from commit d773e3b1469a76f129d2f7a4edab68d007f3f928)

Reviewed: https://review.opendev.org/653033
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ee52fb12bb42b80ace19ab898bac008076cb67e6
Submitter: Zuul
Branch: stable/queens

commit ee52fb12bb42b80ace19ab898bac008076cb67e6
Author: yenai <email address hidden>
Date: Wed Jan 23 11:02:33 2019 +0800

    libvirt: disconnect volume when encryption fails

    The compute node has left the attachment information after the
    encryption fails. This patch fixes this by ensuring the volume is
    disconnected when an exception occurs.

    Change-Id: I9d652f182d83a2557ff6ed0b21c69a735c3f9840
    Closes-Bug: #1812945
    (cherry picked from commit 79bcb4e21b7909f3e989d4fbcf68ac268680b865)
    (cherry picked from commit d773e3b1469a76f129d2f7a4edab68d007f3f928)
    (cherry picked from commit 2cde44659c42ebab25259e2b05cd5c2ffe38c131)

This issue was fixed in the openstack/nova 19.0.1 release.

This issue was fixed in the openstack/nova 18.2.1 release.

This issue was fixed in the openstack/nova 17.0.11 release.

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers