Nova will not detect when extending volume online fails

Bug #1843008 reported by hulina
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Low
hulina

Bug Description

Description
===========
Extending volume in-use may fail, in that case, CLI "nova instance-action-list" should show ERROR in the coulume of message.However , when we catch exceptions in /nova/compute/manager.py
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8659-L8677
nova extends volume will fail ,however CLI "nova instance-action vmid req-id" returns success,that's wrong
===========
1.create volume-backend VM
compute]# nova boot --flavor 1 --nic net-id=27105f4f-5847-4538-862d-6565ad002c9e --boot-volume 53bc2318-ae6c-42b5-ad3b-f197f405cf1e VM
+--------------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | vm |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-1jrun74v |
| OS-EXT-SRV-ATTR:root_device_name | - |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | TbPSQRBic9NQ |
| config_drive | |
| created | 2019-09-06T06:41:55Z |
| description | - |
| flavor:disk | 1 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {} |
| flavor:original_name | m1.tiny |
| flavor:ram | 512 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | |
| host_status | |
| id | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| locked | False |
| locked_reason | - |
| metadata | {} |
| name | VM |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| server_groups | [] |
| status | BUILD |
| tags | [] |
| tenant_id | 92cd1d753cf54f3496fda649212cc37b |
| trusted_image_certificates | - |
| updated | 2019-09-06T06:41:55Z |
| user_id | 85c12fb6eada44bc879da10ea394832b |
+--------------------------------------+-------------------------------------------------+
2.Setting specific scenarios to trigger exceptions in
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8659-L8677
for example :insert 'extended_volume_id=1' in
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8656-L8658 ,
pass objects.BlockDeviceMapping.get_by_volume_and_instance invalid extended_volume_id
3. extend instance's volume
root@controller compute]# cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 53bc2318-ae6c-42b5-ad3b-f197f405cf1e | in-use | - | 2 | lvmdriver-1 | true | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
[root@controller compute]# cinder extend 53bc2318-ae6c-42b5-ad3b-f197f405cf1e 3
[root@controller compute]# cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 53bc2318-ae6c-42b5-ad3b-f197f405cf1e | in-use | - | 3 | lvmdriver-1 | true | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
4.volume extend successfully in cinder , check extending result in nova, it shows success,In fact,extending volume in nova failed
[root@controller compute]# nova instance-action-list fe3c831c-95af-4f0b-b32a-31fb1e425462
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
| Action | Request_ID | Message | Start_Time | Updated_At |
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
| create | req-c3220c13-1de3-435d-a765-c1e931025974 | - | 2019-09-06T06:41:54.000000 | 2019-09-06T06:42:02.000000 |
| extend_volume | req-9c233e67-0105-4a76-86aa-36fdd59593dd | - | 2019-09-06T06:42:42.000000 | 2019-09-06T06:42:42.000000 |
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
[root@controller compute]# nova instance-action fe3c831c-95af-4f0b-b32a-31fb1e425462 req-9c233e67-0105-4a76-86aa-36fdd59593dd
+---------------+---------------------------------------------------------------------------+
| Property | Value |
+---------------+---------------------------------------------------------------------------+
| action | extend_volume |
| events | [{u'event': u'compute_extend_volume', |
| | u'finish_time': u'2019-09-06T06:42:42.000000', |
| | u'host': u'controller', |
| | u'hostId': u'b7bbc1111bdc01fbf6ea645b2942c2a201d5211cf997f93543ab57a9', |
| | u'result': u'Success', |
| | u'start_time': u'2019-09-06T06:42:42.000000', |
| | u'traceback': None}] |
| instance_uuid | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
| message | - |
| project_id | ade79e9fbd42422baeefd9e5d4f8a66e |
| request_id | req-9c233e67-0105-4a76-86aa-36fdd59593dd |
| start_time | 2019-09-06T06:42:42.000000 |
| updated_at | 2019-09-06T06:42:42.000000 |
| user_id | 647971cd085b4489af7ccc0126bf8c93 |
+---------------+---------------------------------------------------------------------------+
[root@controller compute]# journalctl -a --unit <email address hidden> |grep req-9c233e67-0105-4a76-86aa-36fdd59593dd
Logs:
Sep 06 14:42:42 controller nova-compute[13260]: DEBUG nova.compute.manager [req-83c716d0-6be3-4034-9ed4-9d416d74358b req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance: fe3c831c-95af-4f0b-b32a-31fb1e425462] Received event volume-extended-53bc2318-ae6c-42b5-ad3b-f197f405cf1e {{(pid=13260) external_instance_event /opt/stack/nova/nova/compute/manager.py:8799}}
Sep 06 14:42:42 controller nova-compute[13260]: DEBUG nova.compute.manager [req-83c716d0-6be3-4034-9ed4-9d416d74358b req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance: fe3c831c-95af-4f0b-b32a-31fb1e425462] Handling volume-extended event for volume 53bc2318-ae6c-42b5-ad3b-f197f405cf1e {{(pid=13260) extend_volume /opt/stack/nova/nova/compute/manager.py:8644}}
Sep 06 14:42:42 controller nova-compute[13260]: WARNING nova.compute.manager [req-83c716d0-6be3-4034-9ed4-9d416d74358b req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance: fe3c831c-95af-4f0b-b32a-31fb1e425462] Extend volume failed, volume 1 is not attached to instance.: VolumeBDMNotFound_Remote: No volume Block Device Mapping with id 1.

hulina (sharonhu)
Changed in nova:
assignee: nobody → hulina (sharonhu)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/680648

Matt Riedemann (mriedem)
tags: added: compute volumes
Changed in nova:
status: New → In Progress
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.