I've installed OpenStack Stein on a number of Ubuntu 18.04 nodes. However, trying to integrate netapp, I get a funny error in the cinder-volume log of the Storage node:
/var/log/cinder/volume.log
---
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager [req-c5604973-7aa4-4faa-b818-739dfc23659f - - - - -] Failed to initialize driver.: File "<string>", line 24
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager Traceback (most recent call last):
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 468, in _init_host
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager self.driver.do_setup(ctxt)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/drivers/netapp/dataontap/iscsi_cmode.py", line 46, in do_setup
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager self.library.do_setup(context)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/utils.py", line 849, in trace_method_logging_wrapper
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager return f(*args, **kwargs)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/utils.py", line 849, in trace_method_logging_wrapper
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager return f(*args, **kwargs)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/drivers/netapp/dataontap/block_cmode.py", line 74, in do_setup
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager self.failed_over_backend_name or self.backend_name)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/drivers/netapp/dataontap/utils/utils.py", line 77, in get_client_for_backend
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager api_trace_pattern=config.netapp_api_trace_pattern)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/utils.py", line 849, in trace_method_logging_wrapper
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager return f(*args, **kwargs)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/utils.py", line 849, in trace_method_logging_wrapper
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager return f(*args, **kwargs)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/drivers/netapp/dataontap/client/client_cmode.py", line 51, in __init__
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager (major, minor) = self.get_ontapi_version(cached=False)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/utils.py", line 849, in trace_method_logging_wrapper
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager return f(*args, **kwargs)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/drivers/netapp/dataontap/client/client_base.py", line 72, in get_ontapi_version
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager res = self.connection.invoke_successfully(ontapi_version, False)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 215, in invoke_successfully
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager result = self.send_http_request(na_element, enable_tunneling)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/utils.py", line 861, in trace_api_logging_wrapper
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager return f(*args, **kwargs)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 203, in send_http_request
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager response_element = self._get_result(response_xml)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 281, in _get_result
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager processed_response = self._parse_response(response)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 276, in _parse_response
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager xml = etree.XML(response)
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "src/lxml/etree.pyx", line 3192, in lxml.etree.XML
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "src/lxml/parser.pxi", line 1876, in lxml.etree._parseMemoryDocument
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "src/lxml/parser.pxi", line 1764, in lxml.etree._parseDoc
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "src/lxml/parser.pxi", line 1126, in lxml.etree._BaseParser._parseDoc
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "src/lxml/parser.pxi", line 600, in lxml.etree._ParserContext._handleParseResultDoc
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "src/lxml/parser.pxi", line 710, in lxml.etree._handleParseResult
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "src/lxml/parser.pxi", line 639, in lxml.etree._raiseParseError
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager File "<string>", line 24
2020-05-11 06:25:14.195 8917 ERROR cinder.volume.manager lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: meta line 12 and head, line 24, column 8
---
I've decided to try the same config on a CentOS 7 Storage node, running the Train version of cinder, but the error is the same:
---
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager [req-c2f34046-a544-4b8e-89f5-56324706bbe3 - - - - -] Failed to initialize driver.: XMLSyntaxError: Opening and ending tag mismatch: meta line 12 and head, line 24, column 8
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager Traceback (most recent call last):
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 467, in _init_host
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager self.driver.do_setup(ctxt)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/iscsi_cmode.py", line 46, in do_setup
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager self.library.do_setup(context)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 727, in trace_method_logging_wrapper
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager return f(*args, **kwargs)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 727, in trace_method_logging_wrapper
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager return f(*args, **kwargs)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/block_cmode.py", line 74, in do_setup
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager self.failed_over_backend_name or self.backend_name)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/utils/utils.py", line 77, in get_client_for_backend
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager api_trace_pattern=config.netapp_api_trace_pattern)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 727, in trace_method_logging_wrapper
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager return f(*args, **kwargs)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 727, in trace_method_logging_wrapper
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager return f(*args, **kwargs)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_cmode.py", line 50, in __init__
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager (major, minor) = self.get_ontapi_version(cached=False)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 727, in trace_method_logging_wrapper
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager return f(*args, **kwargs)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_base.py", line 70, in get_ontapi_version
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager res = self.connection.invoke_successfully(ontapi_version, False)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 213, in invoke_successfully
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager result = self.send_http_request(na_element, enable_tunneling)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 739, in trace_api_logging_wrapper
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager return f(*args, **kwargs)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 201, in send_http_request
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager response_element = self._get_result(response_xml)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 279, in _get_result
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager processed_response = self._parse_response(response)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 274, in _parse_response
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager xml = etree.XML(response)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "lxml.etree.pyx", line 2973, in lxml.etree.XML (src/lxml/lxml.etree.c:62835)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "parser.pxi", line 1617, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:93194)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "parser.pxi", line 1495, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:92003)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "parser.pxi", line 1011, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:88660)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "parser.pxi", line 577, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:84385)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "parser.pxi", line 676, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:85488)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager File "parser.pxi", line 616, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:84811)
2020-05-11 08:53:59.355 6657 ERROR cinder.volume.manager XMLSyntaxError: Opening and ending tag mismatch: meta line 12 and head, line 24, column 8
---
Here's the cinder config on the storage nodes (the file is not identical on both servers, but the diff is only in that, that one of them is exposing an LVM backend and the other one is not):
/etc/cinder/cinder.conf
---
[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
state_path = /var/lib/cinder
lock_path = /var/lock/cinder
volumes_dir = /var/lib/cinder/volumes
transport_url = rabbit://openstack:password@controller
auth_strategy = keystone
enabled_backends = lvm,ontap-iscsi
glance_api_servers = http://controller:9292
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[keystone_authtoken]
www_authenticate_uri = http://controller:5000
auth_url = http://controller:5000
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = password
[database]
connection = mysql+pymysql://cinder:password@controller/cinder
[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
target_protocol = iscsi
target_helper = tgtadm
[ontap-iscsi]
volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver
netapp_login = user
netapp_password = password
volume_backend_name = ontap-iscsi
netapp_server_hostname = netapp
netapp_server_port = 443
netapp_transport_type = https
netapp_vserver = svm-name
netapp_storage_protocol = iscsi
netapp_storage_family = ontap_cluster
---
Any and all help will be highly appreciated.
Cheers,
Alex
Turned out this is what happens when the netapp has 2fa turned on. Once 2fa was disabled all worked as expected