OpenStack Compute (nova)

The computing node has left the mount information after the encryption fails.

Series rocky
Bug #1812945

Bug #1812945 reported by yenai on 2019-01-23

This bug affects 1 person

	Status	Importance	Assigned to
OpenStack Compute (nova)	Fix Released	Medium	yenai
Queens	Fix Committed	Medium	Matt Riedemann
Rocky	Fix Committed	Medium	Matt Riedemann
Stein	Fix Committed	Medium	Matt Riedemann

Bug Description

[stack@localhost devstack]$ cinder type-create encrypt-NoOpEncryptor
+--------------------------------------+-----------------------+-------------+-----------+
| ID | Name | Description | Is_Public |
+--------------------------------------+-----------------------+-------------+-----------+
| 054fad16-6b6b-4426-ac9b-af63bc43d113 | encrypt-NoOpEncryptor | - | True |
+--------------------------------------+-----------------------+-------------+-----------+
[stack@localhost devstack]$ cinder encryption-type-create 054fad16-6b6b-4426-ac9b-af63bc43d113 --control-location front-end NoOpEncryptor
+--------------------------------------+---------------+--------+----------+------------------+
| Volume Type ID | Provider | Cipher | Key Size | Control Location |
+--------------------------------------+---------------+--------+----------+------------------+
| 054fad16-6b6b-4426-ac9b-af63bc43d113 | NoOpEncryptor | - | - | front-end |
+--------------------------------------+---------------+--------+----------+------------------+
[stack@localhost devstack]$ cinder create --volume-type encrypt-NoOpEncryptor --name yenai 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-01-12T01:36:36.000000 |
| description | None |
| encrypted | True |
| id | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | yenai |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 9b881cf6b92049b8a155178274662836 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | 657205033877433db3dd8dadb212d9fb |
| volume_type | encrypt-NoOpEncryptor |
+--------------------------------+--------------------------------------+
[stack@localhost libvirt]$ cinder list
+--------------------------------------+-----------+-------+------+-----------------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------+------+-----------------------+----------+--------------------------------------+
| 247bd974-3cb5-44a5-9b7d-599ca2fc9bda | available | yenai | 1 | encrypt-NoOpEncryptor | false | |
| f1c15752-c247-470b-9a5f-1f5559f51ce7 | in-use | yenai | 1 | encrypt-luks | false | 6ce2254a-0e05-4e63-874e-44d112bed7d0 |
+--------------------------------------+-----------+-------+------+-----------------------+----------+--------------------------------------+
[[stack@localhost devstack]$ nova volume-attach 6ce2254a-0e05-4e63-874e-44d112bed7d0 247bd974-3cb5-44a5-9b7d-599ca2fc9bda
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdc |
| id | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
| serverId | 6ce2254a-0e05-4e63-874e-44d112bed7d0 |
| volumeId | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
+----------+--------------------------------------+
[stack@localhost devstack]$

ERROR:
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [None req-756c266a-47e7-4173-b06f-c4a60c5dcd55 admin admin] [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] Driver failed to attach volume 247bd974-3cb5-44a5-9b7d-599ca2fc9bda at /dev/vdc: TypeError: attach_volume() got an unexpected keyword argument 'cipher'
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] Traceback (most recent call last):
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/block_device.py", line 559, in _volume_attach
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] device_type=self['device_type'], encryption=encryption)
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1460, in attach_volume
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] encryption=encryption)
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1260, in _connect_volume
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] allow_native_luks)
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1395, in _attach_encryptor
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] encryptor.attach_volume(context, **encryption)
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] TypeError: attach_volume() got an unexpected keyword argument 'cipher'
Jan 12 09:37:25 localhost.localdomain nova-compute[24418]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]

[stack@localhost encryptors]$ nova show 6ce2254a-0e05-4e63-874e-44d112bed7d0
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | localhost.localdomain |
| OS-EXT-SRV-ATTR:hostname | yenai |
| OS-EXT-SRV-ATTR:hypervisor_hostname | localhost.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-m8xrsuw9 |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-01-11T22:22:00.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2019-01-11T22:21:51Z |
| description | - |
| flavor:disk | 1 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {} |
| flavor:original_name | m1.tiny |
| flavor:ram | 512 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | b3a5a7c35a7b6862a03b3bdca5defe64fef34763e1440a3cfb4aa2ad |
| host_status | UP |
| id | 6ce2254a-0e05-4e63-874e-44d112bed7d0 |
| image | cirros-0.3.6-x86_64-disk (c10be70f-38a7-4b58-a033-8c045a8c4645) |
| key_name | - |
| locked | False |
| metadata | {} |
| name | yenai |
| os-extended-volumes:volumes_attached | [{"id": "f1c15752-c247-470b-9a5f-1f5559f51ce7", "delete_on_termination": false}] |
| progress | 0 |
| public network | 2001:db8::1d2, 172.24.4.104 |
| security_groups | default |
| status | ACTIVE |
| tags | [] |
| tenant_id | 9b881cf6b92049b8a155178274662836 |
| trusted_image_certificates | - |
| updated | 2019-01-11T22:22:00Z |
| user_id | 657205033877433db3dd8dadb212d9fb |
+--------------------------------------+----------------------------------------------------------------------------------+
[stack@localhost encryptors]$ cinder show 247bd974-3cb5-44a5-9b7d-599ca2fc9bda
+--------------------------------+-----------------------------------------------+
| Property | Value |
+--------------------------------+-----------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-01-12T01:36:36.000000 |
| description | None |
| encrypted | True |
| id | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | yenai |
| os-vol-host-attr:host | localhost.localdomain@lvmdriver-1#lvmdriver-1 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 9b881cf6b92049b8a155178274662836 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2019-01-12T05:18:36.000000 |
| user_id | 657205033877433db3dd8dadb212d9fb |
| volume_type | encrypt-NoOpEncryptor |
+--------------------------------+-----------------------------------------------+
[stack@localhost encryptors]$ ls -lh /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root 9 Jan 12 13:18 ip-192.168.43.139:3260-iscsi-iqn.2010-10.org.openstack:volume-247bd974-3cb5-44a5-9b7d-599ca2fc9bda-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root 9 Jan 12 06:30 ip-192.168.43.139:3260-iscsi-iqn.2010-10.org.openstack:volume-f1c15752-c247-470b-9a5f-1f5559f51ce7-lun-0 -> ../../sdb
lrwxrwxrwx. 1 root root 9 Jan 11 15:42 pci-0000:00:07.1-ata-2.0 -> ../../sr0
lrwxrwxrwx. 1 root root 9 Jan 11 15:42 pci-0000:00:10.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx. 1 root root 10 Jan 11 15:42 pci-0000:00:10.0-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx. 1 root root 10 Jan 11 15:42 pci-0000:00:10.0-scsi-0:0:0:0-part2 -> ../../sda2

See original description

Tags:

yenai (yenai2008) on 2019-01-23

Changed in nova:
assignee:	nobody → yenai (yenai2008)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-01-23: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/632598

Changed in nova:
status:	New → In Progress

yenai (yenai2008) on 2019-01-23

description:

updated

Takashi Natsume (natsume-takashi) on 2019-02-21

tags:

added: cinder volumes

Revision history for this message

Lee Yarwood (lyarwood) wrote on 2019-02-21:

I wonder if we also need to drop the NoOpEncryptor?

Revision history for this message

yenai (yenai2008) wrote on 2019-03-28:

I think so @Lee Yarwood (lyarwood)

Matt Riedemann (mriedem) on 2019-04-05

Changed in nova:
importance:	Undecided → Medium
tags:	added: libvirt

OpenStack Infra (hudson-openstack) on 2019-04-06

Changed in nova:
assignee:	yenai (yenai2008) → Matt Riedemann (mriedem)

Revision history for this message

Matt Riedemann (mriedem) wrote on 2019-04-06:

Feel free to backport this since it was introduced in Queens.

Changed in nova:
assignee:	Matt Riedemann (mriedem) → yenai (yenai2008)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-06: Fix merged to nova (master)

Reviewed: https://review.openstack.org/632598
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=79bcb4e21b7909f3e989d4fbcf68ac268680b865
Submitter: Zuul
Branch: master

commit 79bcb4e21b7909f3e989d4fbcf68ac268680b865
Author: yenai <email address hidden>
Date: Wed Jan 23 11:02:33 2019 +0800

libvirt: disconnect volume when encryption fails

    The compute node has left the attachment information after the
    encryption fails. This patch fixes this by ensuring the volume is
    disconnected when an exception occurs.

Change-Id: I9d652f182d83a2557ff6ed0b21c69a735c3f9840
Closes-Bug: #1812945

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-08: Fix proposed to nova (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.openstack.org/650931

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-10: Fix merged to nova (stable/stein)

Reviewed: https://review.openstack.org/650931
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d773e3b1469a76f129d2f7a4edab68d007f3f928
Submitter: Zuul
Branch: stable/stein

commit d773e3b1469a76f129d2f7a4edab68d007f3f928
Author: yenai <email address hidden>
Date: Wed Jan 23 11:02:33 2019 +0800

libvirt: disconnect volume when encryption fails

    The compute node has left the attachment information after the
    encryption fails. This patch fixes this by ensuring the volume is
    disconnected when an exception occurs.

    Change-Id: I9d652f182d83a2557ff6ed0b21c69a735c3f9840
    Closes-Bug: #1812945
    (cherry picked from commit 79bcb4e21b7909f3e989d4fbcf68ac268680b865)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-11: Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/651796

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-16: Fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/651796
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2cde44659c42ebab25259e2b05cd5c2ffe38c131
Submitter: Zuul
Branch: stable/rocky

commit 2cde44659c42ebab25259e2b05cd5c2ffe38c131
Author: yenai <email address hidden>
Date: Wed Jan 23 11:02:33 2019 +0800

libvirt: disconnect volume when encryption fails

    The compute node has left the attachment information after the
    encryption fails. This patch fixes this by ensuring the volume is
    disconnected when an exception occurs.

    Change-Id: I9d652f182d83a2557ff6ed0b21c69a735c3f9840
    Closes-Bug: #1812945
    (cherry picked from commit 79bcb4e21b7909f3e989d4fbcf68ac268680b865)
    (cherry picked from commit d773e3b1469a76f129d2f7a4edab68d007f3f928)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-16: Fix proposed to nova (stable/queens)

#10

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/653033

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-25: Fix merged to nova (stable/queens)

#11

Reviewed: https://review.opendev.org/653033
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ee52fb12bb42b80ace19ab898bac008076cb67e6
Submitter: Zuul
Branch: stable/queens

commit ee52fb12bb42b80ace19ab898bac008076cb67e6
Author: yenai <email address hidden>
Date: Wed Jan 23 11:02:33 2019 +0800

libvirt: disconnect volume when encryption fails

    The compute node has left the attachment information after the
    encryption fails. This patch fixes this by ensuring the volume is
    disconnected when an exception occurs.