Comment 0 for bug 1831983

Dear,
    I have some troubles deploying a Manila stein version using the GPFS driver over Ubuntu 18.04.
    Manila api and scheduler seem to be starting fine, the problem is with the manila-share service.

    manila:~ # dpkg -l | grep manila
ii manila-api 1:8.0.0-0ubuntu1~cloud0 all OpenStack shared file system as a service - API server
ii manila-common 1:8.0.0-0ubuntu1~cloud0 all OpenStack shared file system as a service - common files
ii manila-scheduler 1:8.0.0-0ubuntu1~cloud0 all OpenStack shared file system as a service - Scheduler server
ii manila-share 1:8.0.0-0ubuntu1~cloud0 all OpenStack shared file system as a service - Share server
ii python3-manila 1:8.0.0-0ubuntu1~cloud0 all OpenStack shared file system as a service - Python 3 libs
ii python3-manilaclient 1.27.0-0ubuntu1~cloud0 all OpenStack shared file system as a service - Python 3.x client

gpfs version on storage backend is 4.2.3-13

the configuration related to GPFS in the manila.conf is :

share_driver = manila.share.drivers.ibm.gpfs.GPFSShareDriver
gpfs_share_export_ip = xx.xx.xx.xx
gpfs_mount_point_base = /gpfs/ces
gpfs_nfs_server_type = CES
#gpfs_nfs_server_list = <None>
is_gpfs_node = false
gpfs_ssh_port = 22
gpfs_ssh_login = XXXX
#gpfs_ssh_password = <None>
gpfs_ssh_private_key = /var/lib/manila/.ssh/id_rsa
gpfs_share_helpers = CES=manila.share.drivers.ibm.gpfs.CESHelper

GPFS backend is running on other GPFS node. Ssh connection seems to be OK, but the function _get_gpfs_state() seems not to get the correct value:

019-06-06 11:56:15.345 4191 DEBUG manila.share.manager [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Start initialization of driver: '<email address hidden>' _driver_setup /usr/lib/python3/dist-packages/manila/share/manager.py:308
2019-06-06 11:56:15.347 4191 DEBUG manila.utils [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] ssh.connect: ip: XXXXX port: 22, username: user, password: None, key_filename: /var/lib/manila/.ssh/id_rsa, look_for_keys: False, timeout: 60, banner_timeout: 60 create /usr/lib/python3/dist-packages/manila/utils.py:138
2019-06-06 11:56:15.349 4191 DEBUG paramiko.transport [-] starting thread (client mode): 0x2b75dd30 _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.349 4191 DEBUG paramiko.transport [-] Local version/idstring: SSH-2.0-paramiko_2.0.0 _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.357 4191 DEBUG paramiko.transport [-] Remote version/idstring: SSH-2.0-OpenSSH_7.4 _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.357 4191 INFO paramiko.transport [-] Connected (version 2.0, client OpenSSH_7.4)
2019-06-06 11:56:15.360 4191 DEBUG paramiko.transport [-] kex algos:['curve25519-sha256', '<email address hidden>', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group16-sha512', 'diffie-hellman-group18-sha512', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha256', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'rsa-sha2-512', 'rsa-sha2-256', 'ecdsa-sha2-nistp256', 'ssh-ed25519'] client encrypt:['<email address hidden>', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', '<email address hidden>', '<email address hidden>', 'aes128-cbc', 'aes192-cbc', 'aes256-cbc', 'blowfish-cbc', 'cast128-cbc', '3des-cbc'] server encrypt:['<email address hidden>', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', '<email address hidden>', '<email address hidden>', 'aes128-cbc', 'aes192-cbc', 'aes256-cbc', 'blowfish-cbc', 'cast128-cbc', '3des-cbc'] client mac:['<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] server mac:['<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] client compress:['none', '<email address hidden>'] server compress:['none', '<email address hidden>'] client lang:[''] server lang:[''] kex follows?False _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.361 4191 DEBUG paramiko.transport [-] Kex agreed: diffie-hellman-group-exchange-sha256 _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.361 4191 DEBUG paramiko.transport [-] Cipher agreed: aes128-ctr _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.362 4191 DEBUG paramiko.transport [-] MAC agreed: hmac-sha2-256 _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.362 4191 DEBUG paramiko.transport [-] Compression agreed: none _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.368 4191 DEBUG paramiko.transport [-] Got server p (2048 bits) _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.465 4191 DEBUG paramiko.transport [-] kex engine KexGexSHA256 specified hash_algo <built-in function openssl_sha256> _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.466 4191 DEBUG paramiko.transport [-] Switch to new keys ... _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.467 4191 DEBUG paramiko.transport [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Adding ssh-rsa host key for 10.10.4.13: b'5683629c7943d6dbc63e3978a06281f0' _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.469 4191 DEBUG paramiko.transport [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Trying key b'483cd0e64b4cc8163471c2efdd1ae825' from /var/lib/manila/.ssh/id_rsa _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.505 4191 DEBUG paramiko.transport [-] userauth is OK _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.514 4191 INFO paramiko.transport [-] Authentication (publickey) successful!
2019-06-06 11:56:15.514 4191 DEBUG manila.share.drivers.ibm.gpfs [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Running cmd (SSH): /usr/lpp/mmfs/bin/mmgetstate -Y _gpfs_ssh_execute /usr/lib/python3/
2019-06-06 11:56:15.515 4191 DEBUG paramiko.transport [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] [chan 0] Max packet in: 32768 bytes _log /usr/lib/python3/dist-packages/paramiko/channel.py:1089
2019-06-06 11:56:15.516 4191 DEBUG paramiko.transport [-] Received global request "<email address hidden>" _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.517 4191 DEBUG paramiko.transport [-] Rejecting "<email address hidden>" global request from server. _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.557 4191 DEBUG paramiko.transport [-] [chan 0] Max packet out: 32768 bytes _log /usr/lib/python3/dist-packages/paramiko/channel.py:1089
2019-06-06 11:56:15.557 4191 DEBUG paramiko.transport [-] Secsh channel 0 opened. _log /usr/lib/python3/dist-packages/paramiko/transport.py:1547
2019-06-06 11:56:15.559 4191 DEBUG paramiko.transport [-] [chan 0] Sesch channel 0 request ok _log /usr/lib/python3/dist-packages/paramiko/channel.py:1089
2019-06-06 11:56:15.990 4191 DEBUG paramiko.transport [-] [chan 0] EOF received (0) _log /usr/lib/python3/dist-packages/paramiko/channel.py:1089
2019-06-06 11:56:15.991 4191 DEBUG paramiko.transport [-] [chan 0] EOF sent (0) _log /usr/lib/python3/dist-packages/paramiko/channel.py:1089
2019-06-06 11:56:15.992 4191 DEBUG manila.share.drivers.ibm.gpfs [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Result was 0 _gpfs_ssh_execute /usr/lib/python3/dist-packages/manila/share/drivers/ibm/gpfs.py:232
2019-06-06 11:56:15.993 4191 ERROR manila.share.drivers.ibm.gpfs [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Failed to check GPFS state. Error: list index out of range.: IndexError: list index out of range
2019-06-06 11:56:15.994 4191 ERROR manila.share.manager [req-79f86b63-595a-4eb9-8dad-b712a7c45b8e - - - - -] Error encountered during initialization of driver <email address hidden>: manila.exception.GPFSException: Failed to check GPFS state. Error: list index out of range.
2019-06-06 11:56:15.994 4191 ERROR manila.share.manager Traceback (most recent call last):
2019-06-06 11:56:15.994 4191 ERROR manila.share.manager File "/usr/lib/python3/dist-packages/manila/share/drivers/ibm/gpfs.py", line 255, in _check_gpfs_state
2019-06-06 11:56:15.994 4191 ERROR manila.share.manager gpfs_state = lines[1].split(':')[state_token]

Ruuning manualy "/usr/lpp/mmfs/bin/mmgetstate -Y" over ssh works:

manila:~ # ssh XXXX@XXXXX -i /var/lib/manila/.ssh/id_rsa "/usr/lpp/mmfs/bin/mmgetstate -Y"
mmgetstate::HEADER:version:reserved:reserved:nodeName:nodeNumber:state:quorum:nodesUp:totalNodes:remarks:cnfsState:
mmgetstate::0:1:::gpfs:512:active:3:5:255:quorum nodeundefined):

and the parsing by hand the lines (gpfs_state = lines[1].split(':')[state_token] ) returns the correct value (active) while the command result for manila-share, seems to be 0 :
-----Result was 0 _gpfs_ssh_execute-----

regards, I