Manila 8.0.0 stein GPFS Driver index error

[Iban Cabrillo]

Dear,
    I have some troubles deploying a Manila stein version using the GPFS driver over Ubuntu 18.04.
    Manila api and scheduler seem to be starting fine, the problem is with the manila-share service.

    manila:~ # dpkg -l | grep manila
ii manila-api 1:8.0.0-0ubuntu1~cloud0 all OpenStack shared file system as a service - API server
ii manila-common 1:8.0.0-0ubuntu1~cloud0 all OpenStack shared file system as a service - common files
ii manila-scheduler 1:8.0.0-0ubuntu1~cloud0 all OpenStack shared file system as a service - Scheduler server
ii manila-share 1:8.0.0-0ubuntu1~cloud0 all OpenStack shared file system as a service - Share server
ii python3-manila 1:8.0.0-0ubuntu1~cloud0 all OpenStack shared file system as a service - Python 3 libs
ii python3-manilaclient 1.27.0-0ubuntu1~cloud0 all OpenStack shared file system as a service - Python 3.x client

gpfs version on storage backend is 4.2.3-13

the configuration related to GPFS in the manila.conf is :

share_driver = manila.share.drivers.ibm.gpfs.GPFSShareDriver
gpfs_share_export_ip = xx.xx.xx.xx
gpfs_mount_point_base = /gpfs/ces
gpfs_nfs_server_type = CES
#gpfs_nfs_server_list = <None>
is_gpfs_node = false
gpfs_ssh_port = 22
gpfs_ssh_login = XXXX
#gpfs_ssh_password = <None>
gpfs_ssh_private_key = /var/lib/manila/.ssh/id_rsa
gpfs_share_helpers = CES=manila.share.drivers.ibm.gpfs.CESHelper

GPFS backend is running on other GPFS node. Ssh connection seems to be OK, but the function _get_gpfs_state() seems not to get the correct value:

019-06-06 11:56:15.345 4191 DEBUG manila.share.manager [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Start initialization of driver: '<email address hidden>' _driver_setup /usr/lib/python3/dist-packages/manila/share/manager.py:308
2019-06-06 11:56:15.347 4191 DEBUG manila.utils [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] ssh.connect: ip: XXXXX port: 22, username: user, password: None, key_filename: /var/lib/manila/.ssh/id_rsa, look_for_keys: False, timeout: 60, banner_timeout: 60 create /usr/lib/python3/dist-packages/manila/utils.py:138
2019-06-06 11:56:15.349 4191 DEBUG paramiko.transport [-] starting thread (client mode): 0x2b75dd30 _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.349 4191 DEBUG paramiko.transport [-] Local version/idstring: SSH-2.0-paramiko_2.0.0 _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.357 4191 DEBUG paramiko.transport [-] Remote version/idstring: SSH-2.0-OpenSSH_7.4 _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.357 4191 INFO paramiko.transport [-] Connected (version 2.0, client OpenSSH_7.4)
2019-06-06 11:56:15.360 4191 DEBUG paramiko.transport [-] kex algos:['curve25519-sha256', '<email address hidden>', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group16-sha512', 'diffie-hellman-group18-sha512', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha256', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'rsa-sha2-512', 'rsa-sha2-256', 'ecdsa-sha2-nistp256', 'ssh-ed25519'] client encrypt:['<email address hidden>', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', '<email address hidden>', '<email address hidden>', 'aes128-cbc', 'aes192-cbc', 'aes256-cbc', 'blowfish-cbc', 'cast128-cbc', '3des-cbc'] server encrypt:['<email address hidden>', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', '<email address hidden>', '<email address hidden>', 'aes128-cbc', 'aes192-cbc', 'aes256-cbc', 'blowfish-cbc', 'cast128-cbc', '3des-cbc'] client mac:['<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] server mac:['<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] client compress:['none', '<email address hidden>'] server compress:['none', '<email address hidden>'] client lang:[''] server lang:[''] kex follows?False _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.361 4191 DEBUG paramiko.transport [-] Kex agreed: diffie-hellman-group-exchange-sha256 _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.361 4191 DEBUG paramiko.transport [-] Cipher agreed: aes128-ctr _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.362 4191 DEBUG paramiko.transport [-] MAC agreed: hmac-sha2-256 _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.362 4191 DEBUG paramiko.transport [-] Compression agreed: none _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.368 4191 DEBUG paramiko.transport [-] Got server p (2048 bits) _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.465 4191 DEBUG paramiko.transport [-] kex engine KexGexSHA256 specified hash_algo <built-in function openssl_sha256> _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.466 4191 DEBUG paramiko.transport [-] Switch to new keys ... _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.467 4191 DEBUG paramiko.transport [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Adding ssh-rsa host key for 10.10.4.13: b'5683629c7943d6dbc63e3978a06281f0' _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.469 4191 DEBUG paramiko.transport [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Trying key b'483cd0e64b4cc8163471c2efdd1ae825' from /var/lib/manila/.ssh/id_rsa _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.505 4191 DEBUG paramiko.transport [-] userauth is OK _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.514 4191 INFO paramiko.transport [-] Authentication (publickey) successful!
2019-06-06 11:56:15.514 4191 DEBUG manila.share.drivers.ibm.gpfs [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Running cmd (SSH): /usr/lpp/mmfs/bin/mmgetstate -Y _gpfs_ssh_execute /usr/lib/python3/
2019-06-06 11:56:15.515 4191 DEBUG paramiko.transport [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] [chan 0] Max packet in: 32768 bytes _log /usr/lib/python3/dist-packages/paramiko/channel.py:1089
2019-06-06 11:56:15.516 4191 DEBUG paramiko.transport [-] Received global request "<email address hidden>" _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.517 4191 DEBUG paramiko.transport [-] Rejecting "<email address hidden>" global request from server. _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.557 4191 DEBUG paramiko.transport [-] [chan 0] Max packet out: 32768 bytes _log /usr/lib/python3/dist-packages/paramiko/channel.py:1089
2019-06-06 11:56:15.557 4191 DEBUG paramiko.transport [-] Secsh channel 0 opened. _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.559 4191 DEBUG paramiko.transport [-] [chan 0] Sesch channel 0 request ok _log /usr/lib/python3/dist-packages/paramiko/channel.py:1089
2019-06-06 11:56:15.990 4191 DEBUG paramiko.transport [-] [chan 0] EOF received (0) _log /usr/lib/python3/dist-packages/paramiko/channel.py:1089
2019-06-06 11:56:15.991 4191 DEBUG paramiko.transport [-] [chan 0] EOF sent (0) _log /usr/lib/python3/dist-packages/paramiko/channel.py:1089
2019-06-06 11:56:15.992 4191 DEBUG manila.share.drivers.ibm.gpfs [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Result was 0 _gpfs_ssh_execute /usr/lib/python3/dist-packages/manila/share/drivers/ibm/gpfs.py:232
2019-06-06 11:56:15.993 4191 ERROR manila.share.drivers.ibm.gpfs [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Failed to check GPFS state. Error: list index out of range.: IndexError: list index out of range
2019-06-06 11:56:15.994 4191 ERROR manila.share.manager [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Error encountered during initialization of driver <email address hidden>: manila.exception.GPFSException: Failed to check GPFS state. Error: list index out of range.
2019-06-06 11:56:15.994 4191 ERROR manila.share.manager Traceback (most recent call last):
2019-06-06 11:56:15.994 4191 ERROR manila.share.manager File "/usr/lib/python3/dist-packages/manila/share/drivers/ibm/gpfs.py", line 255, in _check_gpfs_state
2019-06-06 11:56:15.994 4191 ERROR manila.share.manager gpfs_state = lines[1].split(':')[state_token]

Ruuning manualy "/usr/lpp/mmfs/bin/mmgetstate -Y" over ssh works:

manila:~ # ssh XXXX@XXXXX -i /var/lib/manila/.ssh/id_rsa "/usr/lpp/mmfs/bin/mmgetstate -Y"
mmgetstate::HEADER:version:reserved:reserved:nodeName:nodeNumber:state:quorum:nodesUp:totalNodes:remarks:cnfsState:
mmgetstate::0:1:::gpfs:512:active:3:5:255:quorum nodeundefined):

and the parsing by hand the lines (gpfs_state = lines[1].split(':')[state_token] ) returns the correct value (active) while the command result for manila-share, seems to be 0 :
-----Result was 0 _gpfs_ssh_execute-----

regards, I

[Aida Palacio Hoz (aidaph)]

Hello,

Iban and I did a complete debug of manila-share code and the issue is fixed now.

For this error:

2019-06-06 11:56:15.993 4191 ERROR manila.share.drivers.ibm.gpfs [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Failed to check GPFS state. Error: list index out of range.: IndexError: list index out of range

We guess the problem is with python version. In Stein release Openstack services use python3 and this code runs properly in python2. In manila.share.drivers.ibm.gpfsgpfs module, the function _check_gpfs_state fails in lines = out.splitlines(). We set this line to lines = out.split('\\n') and it runs ok now. Same for methods _is_dir where line output.strip() == 'directory' was replaced by output.lstrip('b\'').replace('\\n\'','') == 'directory' and in _get_available_capacity where the line out = out.splitlines()[1] was set by out = out.split('\\n')[1]. The problem is the "\n" due to a misunderstanding of character '\'.

This solution is just a temporal fix because we understood that it was not the better solution.

Cheers,
Aida

[Tom Barron (tpb)]

Aida, are you saying the current code that we ship is OK, or that it needs changes like those that you describe?

[Iban Cabrillo (cabrillo)]

Hi Tom,
  We had to modify the default code at gpfs.py module and change all the lines with the code:

  lines = out.splitlines()

   by this one to get the correct value

  lines = out.split('\\n')

  Now manila-share service is working as spected.
Regrads, I

[Tom Barron (tpb)]

Iban and Aida, do you folks want to propose a fix for this bug then? I promise to see that it is reviewed.

[Vida Haririan (vhariria)]

Additional comments http://eavesdrop.openstack.org/meetings/manila/2020/manila.2020-05-28-15.00.log.html

[Vida Haririan (vhariria)]

Hi,
We believe special hardware is required to follow up. Please let us know if anyone in your team is working on a fix.

Additional comment are at https://meetings.opendev.org/meetings/manila/2023/manila.2023-06-01-15.00.log.html