Cinder

attach volume fail with linstor driver

Bug #1914904 reported by FredericWU on 2021-02-07

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	New	Medium	Unassigned

Bug Description

Attach volume test failed（LINSTOR+iSCSI protocol）

Environment：

1.ARM64

2.root@openstcontroller:~# uname -a
Linux openstcontroller 4.15.0-130-generic #134-Ubuntu SMP Tue Jan 5 20:43:12 UTC 2021

3.root@openstcontroller:~# cat /proc/version
Linux version 4.15.0-130-generic (buildd@bos02-arm64-033) (gcc version 7.5.0 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)) #134-Ubuntu SMP Tue Jan 5 20:43:12 UTC 2021

4.openstack version：Stein

5.LINSTOR version:linstor 1.6.1; GIT-hash: df9121a4e86502351bd7373446dc0f9233ab3bb5

6.DRBD version：DRBDADM_BUILDTAG=GIT-hash:\ baaca8a080dc54652f57da4bafb2dce51dfe9f68\ build\ by\ buildd@bos02-arm64-030,\ 2020-09-28\ 08:05:14DRBDADM_API_VERSION=2DRBD_KERNEL_VERSION_CODE=0x090019DRBD_KERNEL_VERSION=9.0.25DRBDADM_VERSION_CODE=0x090f00DRBDADM_VERSION=9.15.0

Architecture diagram：Please view the first comment picture

Cinder configuration （/etc/cinder/cinder.conf ）
root@node88:~# cat /etc/cinder/cinder.conf
[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
state_path = /var/lib/cinder
lock_path = /var/lock/cinder
volumes_dir = /var/lib/cinder/volumes

transport_url = rabbit://openstack:password@openstcontroller
auth_strategy = keystone
my_ip = 192.168.7.88
enabled_backends = lvm,linstor,linstor_iscsi
glance_api_servers = http://openstcontroller:9292

[database]
connection = mysql+pymysql://cinder:password@openstcontroller/cinder
#connection = sqlite:////var/lib/cinder/cinder.sqlite

[keystone_authtoken]
www_authenticate_uri = http://openstcontroller:5000
auth_url = http://openstcontroller:5000
memcached_servers = openstcontroller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = password

[linstor_iscsi]
volume_backend_name = linstor_iscsi
volume_driver = cinder.volume.drivers.linstordrv.LinstorIscsiDriver
image_volume_cache_enabled = True
linstor_default_volume_group_name=drbdpool
linstor_default_uri=linstor://192.168.7.43
linstor_default_storage_pool_name=DfltStorPool
linstor_default_resource_size=1
linstor_volume_downsize_factor=4096
linstor_controller_diskless=False
iscsi_helper=tgtadm

[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
target_protocol = iscsi
target_helper = tgtadm

[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

Check status on OpenStack controller node

openstcontroller@openstcontroller:~$ cinder service-list
+------------------+----------------------+------+----------+-------+----------------------------+---------+-----------------+---------------+
| Binary | Host | Zone | Status | State | Updated_at | Cluster | Disabled Reason | Backend State |
+------------------+----------------------+------+----------+-------+----------------------------+---------+-----------------+---------------+
| cinder-scheduler | openstcontroller | nova | enabled | up | 2021-02-05T06:19:33.000000 | - | - | |
| cinder-volume | node88@linstor_iscsi | nova | enabled | up | 2021-02-05T06:19:26.000000 | - | - | up |
| cinder-volume | node88@lvm | nova | enabled | up | 2021-02-05T06:19:26.000000 | - | - | up |
| cinder-volume | openstcontroller@lvm | nova | disabled | down | 2021-01-25T07:27:39.000000 | - | - | - |
+------------------+----------------------+------+----------+-------+----------------------------+---------+-----------------+---------------+

Create volume successful！

Check status on cinder node(node88)

root@node88:~# linstor r l
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ CV_89a5d0ed-990e-4976-aa21-03162761c849 ┊ linstora ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2021-02-05 01:20:41 ┊
┊ CV_89a5d0ed-990e-4976-aa21-03162761c849 ┊ linstorb ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2021-02-05 01:20:46 ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

root@node88:~# drbdadm status
CV_89a5d0ed-990e-4976-aa21-03162761c849 role:Secondary
  disk:Diskless
  node43 role:Secondary
    peer-disk:UpToDate
  ubuntu role:Secondary
    peer-disk:UpToDate

Attach volume to virtual machine failed

openstcontroller@openstcontroller:~$ openstack server add volume ubuntu linstoriscsi8g
openstcontrolle@openstcontroller:~$

Volume not in use，attach failed！

Cinder log:
root@node88:~# cat /var/log/cinder/cinder-volume.log
2021-02-05 01:20:38.457 35844 INFO cinder.volume.flows.manager.create_volume [req-fee6b110-8e6e-499d-9b93-ada17413ba39 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Volume 89a5d0ed-990e-4976-aa21-03162761c849: being created as raw with specification: {'status': 'creating', 'volume_name': 'volume-89a5d0ed-990e-4976-aa21-03162761c849', 'volume_size': 8}
2021-02-05 01:20:47.028 35844 INFO cinder.volume.flows.manager.create_volume [req-fee6b110-8e6e-499d-9b93-ada17413ba39 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Volume volume-89a5d0ed-990e-4976-aa21-03162761c849 (89a5d0ed-990e-4976-aa21-03162761c849): created successfully
2021-02-05 01:20:47.041 35844 INFO cinder.volume.manager [req-fee6b110-8e6e-499d-9b93-ada17413ba39 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Created volume successfully.
2021-02-05 01:23:09.817 35844 WARNING cinder.volume.targets.tgt [req-43f970fe-c691-4303-829b-33050cc189b0 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Attempting recreate of backing lun...
2021-02-05 01:23:19.837 35844 ERROR cinder.volume.targets.tgt [req-43f970fe-c691-4303-829b-33050cc189b0 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Failed recovery attempt to create iscsi backing lun for Volume ID:iqn.2010-10.org.openstack:volume-89a5d0ed-990e-4976-aa21-03162761c849: Unexpected error while running command.
Command: tgtadm --lld iscsi --op new --mode logicalunit --tid 7 --lun 1 -b None
Exit code: 22
Stdout: ''
Stderr: 'tgtadm: invalid request\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager [req-43f970fe-c691-4303-829b-33050cc189b0 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Create export for volume failed (Failed to create iscsi target for volume volume-89a5d0ed-990e-4976-aa21-03162761c849.).: cinder.exception.ISCSITargetCreateFailed: Failed to create iscsi target for volume volume-89a5d0ed-990e-4976-aa21-03162761c849.
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager Traceback (most recent call last):
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 4445, in _connection_create
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager volume, connector)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/drivers/linstordrv.py", line 1051, in create_export
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager volume_path)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/targets/iscsi.py", line 210, in create_export
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager **portals_config)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/utils.py", line 818, in _wrapper
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/utils.py", line 818, in _wrapper
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager return r.call(f, *args, **kwargs)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager return attempt.get(self._wrap_exception)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager six.reraise(self.value[0], self.value[1], self.value[2])
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager raise value
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/targets/tgt.py", line 224, in create_iscsi_target
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager raise exception.ISCSITargetCreateFailed(volume_id=vol_id)
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager cinder.exception.ISCSITargetCreateFailed: Failed to create iscsi target for volume volume-89a5d0ed-990e-4976-aa21-03162761c849.
2021-02-05 01:23:19.928 35844 ERROR cinder.volume.manager
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server [req-43f970fe-c691-4303-829b-33050cc189b0 2a19dea7fcb848eba9d25af19fa038e0 f27b37f7b0954575828bf6157ab11ce6 - default default] Exception during message handling: cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Create export for volume failed (Failed to create iscsi target for volume volume-89a5d0ed-990e-4976-aa21-03162761c849.).
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 4445, in _connection_create
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server volume, connector)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/drivers/linstordrv.py", line 1051, in create_export
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server volume_path)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/targets/iscsi.py", line 210, in create_export
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server **portals_config)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/utils.py", line 818, in _wrapper
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server return attempt.get(self._wrap_exception)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2])
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server raise value
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/targets/tgt.py", line 224, in create_iscsi_target
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server raise exception.ISCSITargetCreateFailed(volume_id=vol_id)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server cinder.exception.ISCSITargetCreateFailed: Failed to create iscsi target for volume volume-89a5d0ed-990e-4976-aa21-03162761c849.
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 4517, in attachment_update
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server connector)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server connector)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 4449, in _connection_create
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=err_msg)
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Create export for volume failed (Failed to create iscsi target for volume volume-89a5d0ed-990e-4976-aa21-03162761c849.).
2021-02-05 01:23:19.930 35844 ERROR oslo_messaging.rpc.server
root@node88:~#

if anyone need more information,please let me know.

See original description

Tags:

Revision history for this message

FredericWU (frederic22) wrote on 2021-02-07:

Screen Shot 2021-02-07 at 10.45.06 AM.png Edit (45.2 KiB, image/png)

Architecture diagram on my environment

description:

updated

FredericWU (frederic22) on 2021-02-07

description:

updated

Sofia Enriquez (lsofia-enriquez) on 2021-02-10

Changed in cinder:
importance:	Undecided → Medium
tags:	added: stein

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Screen Shot 2021-02-07 at 10.45.06 AM.png Edit

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.