kolla-ansible

[DELL Unity] Image volume creation fail in Unity

Bug #2019943 reported by Jaeshik Ko on 2023-05-17

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	kolla-ansible	New	Undecided	Unassigned

Bug Description

We are now using 2 Unity Box and trying to create image volume from unity.
Unity-1 is operating normally but if we create image volume using Unity-2 error occur as below.
The volume is created normally in Unity-2, but cannot attach to the controller node.
Can I get some advice?

Openstack Version : Zed(kolla ansible)
Unity Cinder
cat /var/lib/docker/overlay2/7663952c08a841bc713f7439515009a7c10b0bc92410124d50de5a39b432ae7c/diff/cinder-base-source/cinder-21.1.1.dev2/cinder/volume/drivers/dell_emc/unity/driver.py | grep VERSION
VERSION = '07.02.00'
stats['driver_version'] = self.VERSION.

2023-05-08 08:52:12.043 121 ERROR cinder.volume.driver [None req-53dbc5dc-3b4e-481a-8b55-85cb882d53e6 6bbc0a2aa3c04b39ae780c19e9308e44 c2f467630b56446a98b5741bd8d4c43c - - - -] Unable to fetch connection information from backend: multiple UnityHost with name ay-openstackctrl-02 found.: storops.exception.UnityNameNotUniqueError: multiple UnityHost with name ay-openstackctrl-02 found.
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils [None req-53dbc5dc-3b4e-481a-8b55-85cb882d53e6 6bbc0a2aa3c04b39ae780c19e9308e44 c2f467630b56446a98b5741bd8d4c43c - - - -] Failed to copy image 935b4940-87f0-442f-bba7-f9d5034b061c to volume: 35a74276-2f64-45d1-b1be-348993d32f5d: cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: multiple UnityHost with name ay-openstackctrl-02 found.
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils Traceback (most recent call last):
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/driver.py", line 1046, in _attach_volume
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils conn = self.initialize_connection(volume, properties)
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/unity/driver.py", line 219, in initialize_connection
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils conn_info = self.adapter.initialize_connection(volume, connector)
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/volume_utils.py", line 1487, in trace_logging_wrapper
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils return f(*args, **kwargs)
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/unity/adapter.py", line 537, in initialize_connection
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils return self._initialize_connection(lun, connector, volume.id [volume.id])
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/unity/adapter.py", line 520, in _initialize_connection
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils host, hlu = self._create_host_and_attach(connector['host'],
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/unity/adapter.py", line 516, in _create_host_and_attach
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils return _lock_helper('{unity}-{host}'.format(unity=self.client.host,
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/unity/adapter.py", line 509, in _lock_helper
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils host = self.client.create_host(host_name)
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/decorator.py", line 232, in fun
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils return caller(func, *(extras + args), **kw)
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/coordination.py", line 200, in _synchronized
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils return f(*a, **k)
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/unity/client.py", line 258, in create_host
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils return self.create_host_wo_lock(name)
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/dell_emc/unity/client.py", line 264, in create_host_wo_lock
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils host = self.system.get_host(name=name)
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/cinder/.local/lib/python3.9/site-packages/storops/unity/resource/system.py", line 343, in get_host
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils ret = self._get_unity_rsc(UnityHostList, _id=_id, name=name,
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/cinder/.local/lib/python3.9/site-packages/storops/unity/resource/__init__.py", line 200, in _get_unity_rsc
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils raise UnityNameNotUniqueError(
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils storops.exception.UnityNameNotUniqueError: multiple UnityHost with name ay-openstackctrl-02 found.
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils During handling of the above exception, another exception occurred:
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils Traceback (most recent call last):
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/volume_utils.py", line 1209, in copy_image_to_volume
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils driver.copy_image_to_volume(
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/driver.py", line 855, in copy_image_to_volume
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils self._copy_image_data_to_volume(
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/driver.py", line 877, in _copy_image_data_to_volume
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils attach_info, volume = self._attach_volume(context, volume, properties)
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/driver.py", line 1059, in _attach_volume
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils raise exception.VolumeBackendAPIException(data=err_msg)
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: multiple UnityHost with name ay-openstackctrl-02 found.
2023-05-08 08:52:12.044 121 ERROR cinder.volume.volume_utils

[root@ay-openstackdeploy-01 ~]# cinder service-list
+------------------+-----------------------------+------+---------+-------+----------------------------+---------+-----------------+---------------+
| Binary | Host | Zone | Status | State | Updated_at | Cluster | Disabled Reason | Backend State |
+------------------+-----------------------------+------+---------+-------+----------------------------+---------+-----------------+---------------+
| cinder-backup | ay-openstackctrl-01 | nova | enabled | up | 2023-05-10T04:03:26.000000 | - | - | |
| cinder-backup | ay-openstackctrl-02 | nova | enabled | up | 2023-05-10T04:03:26.000000 | - | - | |
| cinder-backup | ay-openstackctrl-03 | nova | enabled | up | 2023-05-10T04:03:29.000000 | - | - | |
| cinder-scheduler | ay-openstackctrl-01 | nova | enabled | up | 2023-05-10T04:03:32.000000 | - | - | |
| cinder-scheduler | ay-openstackctrl-02 | nova | enabled | up | 2023-05-10T04:03:23.000000 | - | - | |
| cinder-scheduler | ay-openstackctrl-03 | nova | enabled | up | 2023-05-10T04:03:26.000000 | - | - | |
| cinder-volume | ay-openstackctrl-01@rbd-1 | nova | enabled | up | 2023-05-10T04:03:24.000000 | - | - | up |
| cinder-volume | ay-openstackctrl-01@unity-1 | nova | enabled | up | 2023-05-10T04:03:30.000000 | - | - | up |
| cinder-volume | ay-openstackctrl-01@unity-2 | nova | enabled | up | 2023-05-10T04:03:31.000000 | - | - | up |
| cinder-volume | ay-openstackctrl-02@rbd-1 | nova | enabled | up | 2023-05-10T04:03:27.000000 | - | - | up |
| cinder-volume | ay-openstackctrl-02@unity-1 | nova | enabled | up | 2023-05-10T04:03:25.000000 | - | - | up |
| cinder-volume | ay-openstackctrl-02@unity-2 | nova | enabled | up | 2023-05-10T04:03:25.000000 | - | - | up |
| cinder-volume | ay-openstackctrl-03@rbd-1 | nova | enabled | up | 2023-05-10T04:03:29.000000 | - | - | up |
| cinder-volume | ay-openstackctrl-03@unity-1 | nova | enabled | up | 2023-05-10T04:03:27.000000 | - | - | up |
| cinder-volume | ay-openstackctrl-03@unity-2 | nova | enabled | up | 2023-05-10T04:03:27.000000 | - | - | up |
+------------------+-----------------------------+------+---------+-------+----------------------------+---------+-----------------+---------------+

Tags:

Revision history for this message

Sofia Enriquez (lsofia-enriquez) wrote on 2023-05-17:

Hello Jaeshik Ko,

I'll bring up this bug in today's cinder bug meeting. I suspect the issue might be related to having duplicate ay-openstackctrl-02 with different Unity versions, and you'll need to define both separately. Could you please provide the cinder.conf file?

Thanks

summary:	- Image volume creation fail in Unity + [DELL Unity] Image volume creation fail in Unity
tags:	added: dell drivers emc kolla unity

Sofia Enriquez (lsofia-enriquez) on 2023-05-17

Changed in cinder:
importance:	Undecided → Low
status:	New → Incomplete

Revision history for this message

Sofia Enriquez (lsofia-enriquez) wrote on 2023-05-17:

Moving bug to Kolla after discussion https://meetings.opendev.org/meetings/cinder_bs/2023/cinder_bs.2023-05-17-15.04.log.html#l-28

no longer affects:

cinder

Revision history for this message

Jaeshik Ko (kocap77) wrote on 2023-05-17:

Download full text (3.7 KiB)

Hello Sofia Enriquez, Thank you for you help
Below is the cinder.conf we are using.
Unity-1 backend works normally, and an error occurs when using Unity-2 backend.

[DEFAULT]
debug = False
log_dir = /var/log/kolla/cinder
use_forwarded_for = true
use_stderr = False
my_ip = 10.8.6.31
volume_name_template = volume-%s
glance_api_servers = http://10.8.6.34:9292 [10.8.6.34]
glance_num_retries = 3
glance_ca_certificates_file =
enabled_backends = rbd-1,unity-1,unity-2
default_volume_type = unity-1
api_paste_config = /etc/cinder/api-paste.ini
auth_strategy = keystone
transport_url = rabbit://openstack:uW9Ajepp24xdDAuuS2XLZFHc2hACUOsyzLuX5Ibp@10.8.6.31:5672 [10.8.6.31],openstack:uW9Ajepp24xdDAuuS2XLZFHc2hACUOsyzLuX5Ibp@10.8.6.32:5672 [10.8.6.32],openstack:uW9Ajepp24xdDAuuS2XLZFHc2hACUOsyzLuX5Ibp@10.8.6.33:5672// [10.8.6.33]
rpc_response_timeout = 600
sf_volume_create_timeout = 600
enable_force_upload = True
cinder_internal_tenant_project_id = c2f467630b56446a98b5741bd8d4c43c
cinder_internal_tenant_user_id = 6bbc0a2aa3c04b39ae780c19e9308e44

[oslo_messaging_notifications]
transport_url = rabbit://openstack:uW9Ajepp24xdDAuuS2XLZFHc2hACUOsyzLuX5Ibp@10.8.6.31:5672 [10.8.6.31],openstack:uW9Ajepp24xdDAuuS2XLZFHc2hACUOsyzLuX5Ibp@10.8.6.32:5672 [10.8.6.32],openstack:uW9Ajepp24xdDAuuS2XLZFHc2hACUOsyzLuX5Ibp@10.8.6.33:5672// [10.8.6.33]
driver = messagingv2
topics = notifications

[oslo_messaging_rabbit]
heartbeat_in_pthread = False

[oslo_middleware]
enable_proxy_headers_parsing = True

[nova]
interface = internal
auth_url = http://10.8.6.34:5000 [10.8.6.34]
auth_type = password
project_domain_id = default
user_domain_id = default
region_name = RegionOne
project_name = service
username = nova
password = HJbEXHAeKJf7RV94wyMqPiIxrQwdpLG4DGXpjtCr
cafile =

[database]
connection = mysql+pymysql://cinder:jMMyTCaLq9pac4VZxeqtVa1UD16QEErp1vGuSCNh@10.8.6.34:3306/cinder [10.8.6.34]
connection_recycle_time = 10
max_pool_size = 1
max_retries = -1

[keystone_authtoken]
service_type = volume
www_authenticate_uri = http://10.8.6.34:5000 [10.8.6.34]
auth_url = http://10.8.6.34:5000 [10.8.6.34]
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = Dyc0c1rMNGZM3Q2c67YtDfaIavZeSORxvMLVwNDu
cafile =
region_name = RegionOne
memcache_security_strategy = ENCRYPT
memcache_secret_key = vrfWidmR3y8VNIJopg9HZxWxWl4PWalxAjVgUJlC
memcached_servers = 10.8.6.31:11211 [10.8.6.31],10.8.6.32:11211 [10.8.6.32],10.8.6.33:11211 [10.8.6.33]

[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

[rbd-1]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = rbd-1
rbd_pool = cinder-volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rados_connect_timeout = 5
rbd_user = cinder
rbd_secret_uuid = f2937a79-6827-4924-ac32-5155706013ad
report_discard_supported = True

[unity-1]
# Storage protocol
storage_protocol = FC
# Unisphere IP
san_ip = 172.19.254.56
# Unisphere username and password
san_login = admin
san_password = P@ssw0rd
# Volume driver name
volume_driver = cinder.volume.drivers.dell_emc.unity.Driver
# backend's name
volume_backend_name = Storage_FC_01
image_volume_cache_enabled = True

[unity-2]
# Storage protocol
s...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.