catch VolumeAttachmentNotFound when attach failed

[yenai]

[stack@localhost devstack]$ cinder type-create encrypt-NoOpEncryptor
+--------------------------------------+-----------------------+-------------+-----------+
| ID | Name | Description | Is_Public |
+--------------------------------------+-----------------------+-------------+-----------+
| 054fad16-6b6b-4426-ac9b-af63bc43d113 | encrypt-NoOpEncryptor | - | True |
+--------------------------------------+-----------------------+-------------+-----------+
[stack@localhost devstack]$ cinder encryption-type-create 054fad16-6b6b-4426-ac9b-af63bc43d113 --control-location front-end NoOpEncryptor
+--------------------------------------+---------------+--------+----------+------------------+
| Volume Type ID | Provider | Cipher | Key Size | Control Location |
+--------------------------------------+---------------+--------+----------+------------------+
| 054fad16-6b6b-4426-ac9b-af63bc43d113 | NoOpEncryptor | - | - | front-end |
+--------------------------------------+---------------+--------+----------+------------------+
[stack@localhost devstack]$ cinder create --volume-type encrypt-NoOpEncryptor --name yenai 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-01-12T01:36:36.000000 |
| description | None |
| encrypted | True |
| id | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | yenai |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 9b881cf6b92049b8a155178274662836 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | 657205033877433db3dd8dadb212d9fb |
| volume_type | encrypt-NoOpEncryptor |
+--------------------------------+--------------------------------------+
[stack@localhost libvirt]$ cinder list
+--------------------------------------+-----------+-------+------+-----------------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------+------+-----------------------+----------+--------------------------------------+
| 247bd974-3cb5-44a5-9b7d-599ca2fc9bda | available | yenai | 1 | encrypt-NoOpEncryptor | false | |
| f1c15752-c247-470b-9a5f-1f5559f51ce7 | in-use | yenai | 1 | encrypt-luks | false | 6ce2254a-0e05-4e63-874e-44d112bed7d0 |
+--------------------------------------+-----------+-------+------+-----------------------+----------+--------------------------------------+
[[stack@localhost devstack]$ nova volume-attach 6ce2254a-0e05-4e63-874e-44d112bed7d0 247bd974-3cb5-44a5-9b7d-599ca2fc9bda
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdc |
| id | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
| serverId | 6ce2254a-0e05-4e63-874e-44d112bed7d0 |
| volumeId | 247bd974-3cb5-44a5-9b7d-599ca2fc9bda |
+----------+--------------------------------------+
[stack@localhost devstack]$
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [None req-c7c9bfab-75a6-41b8-a677-d00aac80b11b admin admin] [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] Driver failed to attach volume 247bd974-3cb5-44a5-9b7d-599ca2fc9bda at /dev/vdc: TypeError: attach_volume() got an unexpected keyword argument 'cipher'
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] Traceback (most recent call last):
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/block_device.py", line 559, in _volume_attach
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] device_type=self['device_type'], encryption=encryption)
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1464, in attach_volume
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] encryption=encryption)
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1264, in _connect_volume
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] vol_driver.disconnect_volume(connection_info, instance)
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] self.force_reraise()
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] six.reraise(self.type_, self.value, self.tb)
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1261, in _connect_volume
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] context, connection_info, encryption, allow_native_luks)
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1399, in _attach_encryptor
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] encryptor.attach_volume(context, **encryption)
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] TypeError: attach_volume() got an unexpected keyword argument 'cipher'
Jan 12 13:58:48 localhost.localdomain nova-compute[95555]: ERROR nova.virt.block_device [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [None req-c7c9bfab-75a6-41b8-a677-d00aac80b11b admin admin] [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] Failed to attach 247bd974-3cb5-44a5-9b7d-599ca2fc9bda at /dev/vdc: TypeError: attach_volume() got an unexpected keyword argument 'cipher'
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] Traceback (most recent call last):
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/compute/manager.py", line 5442, in _attach_volume
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] do_driver_attach=True)
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/block_device.py", line 46, in wrapped
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] ret_val = method(obj, context, *args, **kwargs)
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/block_device.py", line 643, in attach
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] virt_driver, do_driver_attach)
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/block_device.py", line 621, in _do_attach
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] do_driver_attach)
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/block_device.py", line 568, in _volume_attach
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] attachment_id)
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] self.force_reraise()
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] six.reraise(self.type_, self.value, self.tb)
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/block_device.py", line 559, in _volume_attach
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] device_type=self['device_type'], encryption=encryption)
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1464, in attach_volume
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] encryption=encryption)
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1264, in _connect_volume
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] vol_driver.disconnect_volume(connection_info, instance)
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] self.force_reraise()
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] six.reraise(self.type_, self.value, self.tb)
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1261, in _connect_volume
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] context, connection_info, encryption, allow_native_luks)
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1399, in _attach_encryptor
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] encryptor.attach_volume(context, **encryption)
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0] TypeError: attach_volume() got an unexpected keyword argument 'cipher'
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.compute.manager [instance: 6ce2254a-0e05-4e63-874e-44d112bed7d0]
Jan 12 13:58:50 localhost.localdomain nova-compute[95555]: ERROR nova.volume.cinder [None req-c7c9bfab-75a6-41b8-a677-d00aac80b11b admin admin] Delete attachment failed for attachment 0a85e2d1-4e33-4f9e-ac87-f5c8f0aaec78. Error: Volume attachment could not be found with filter: attachment_id = 0a85e2d1-4e33-4f9e-ac87-f5c8f0aaec78. (HTTP 404) (Request-ID: req-97e95f31-4dcb-4714-bcea-0c80508e2b08) Code: 404: NotFound: Volume attachment could not be found with filter: attachment_id = 0a85e2d1-4e33-4f9e-ac87-f5c8f0aaec78. (HTTP 404) (Request-ID: req-97e95f31-4dcb-4714-bcea-0c80508e2b08)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/632657

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/632657
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=595866eb62db56fbd4f965619031ac902f99441a
Submitter: Zuul
Branch: master

commit 595866eb62db56fbd4f965619031ac902f99441a
Author: yenai <email address hidden>
Date: Wed Jan 23 13:54:47 2019 +0800

    Ignore VolumeAttachmentNotFound exception in compute.manager

    DriverVolumeBlockDevice will delete volume attachment when attach
    fails, codes link:
    https://github.com/openstack/nova/blob/907c7d2cf/nova/virt/block_device.py#L561-L568
    However, nova.compute.manager will delete it again and it will raise
    VolumeAttachmentNotFound exception. This outputs an incorrect error
    log and this exception should be ignored.

    Change-Id: I939c09e5b0efb3b17a9855af227e6d60c64d23e2
    Closes-Bug: #1812969

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/636954

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/636954
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1a684f6fbef941de88556ec023860ab532938d88
Submitter: Zuul
Branch: stable/rocky

commit 1a684f6fbef941de88556ec023860ab532938d88
Author: yenai <email address hidden>
Date: Wed Jan 23 13:54:47 2019 +0800

    Ignore VolumeAttachmentNotFound exception in compute.manager

    DriverVolumeBlockDevice will delete volume attachment when attach
    fails, codes link:
    https://github.com/openstack/nova/blob/907c7d2cf/nova/virt/block_device.py#L561-L568
    However, nova.compute.manager will delete it again and it will raise
    VolumeAttachmentNotFound exception. This outputs an incorrect error
    log and this exception should be ignored.

    Change-Id: I939c09e5b0efb3b17a9855af227e6d60c64d23e2
    Closes-Bug: #1812969
    (cherry picked from commit 595866eb62db56fbd4f965619031ac902f99441a)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/637391

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637391
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4a7575bf1445fc111ab7d55d9fb9a11176b2dd9d
Submitter: Zuul
Branch: stable/queens

commit 4a7575bf1445fc111ab7d55d9fb9a11176b2dd9d
Author: yenai <email address hidden>
Date: Wed Jan 23 13:54:47 2019 +0800

    Ignore VolumeAttachmentNotFound exception in compute.manager

    DriverVolumeBlockDevice will delete volume attachment when attach
    fails, codes link:
    https://github.com/openstack/nova/blob/907c7d2cf/nova/virt/block_device.py#L561-L568
    However, nova.compute.manager will delete it again and it will raise
    VolumeAttachmentNotFound exception. This outputs an incorrect error
    log and this exception should be ignored.

    NOTE(mriedem): There are a couple of Queens-only changes to the
    unit test due to I4436f5c1cc819c55ca9186bda0362bb74555c95a and
    Id587967ea4f9980c292492e2f659bf55fb037b28 not being in stable/queens.

    Change-Id: I939c09e5b0efb3b17a9855af227e6d60c64d23e2
    Closes-Bug: #1812969
    (cherry picked from commit 595866eb62db56fbd4f965619031ac902f99441a)
    (cherry picked from commit 1a684f6fbef941de88556ec023860ab532938d88)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.11 release.