Comment 7 for bug 1786059

Revision history for this message
Goutham Pacha Ravi (gouthamr) wrote :

Revisiting this bug since we had another reporter with the same error message. Drilling deeper, here are some troubleshooting steps we took:

- The manila-share manager process is usually run with an unprivileged user (commonly "manila")
- Check whether the user has permissions to access the private key file specified in path_to_private_key
- Check whether the key itself has the right permissions - a properly configured key must look like:

        -rw------- 1 manila manila 1679 Nov 1 22:34 /var/lib/manila/.ssh/id_rsa

- Check the version of OpenSSH running on the controller node (`ssh -V`) and the version of paramiko installed

        $ python
        >> import paramiko
        >> paramiko.__version__

Ensure you're running the latest version for your distribution

- The error:

        ERROR oslo_messaging.rpc.server SSHException: Check whether private key or password are correctly set. Error connecting via ssh: Signature verification (ssh-rsa) failed.

suggests that there was certainly network connectivity, but ssh-handshake failed. You can try to debug this further by doing what the driver does:

1) Log into as the 'manila' user on your controller node (You may have to "chsh manila /bin/bash" to allow the service user permissions to use a shell on login. Be sure to revert this when you're done debugging)
2) Launch python on the controller node and run these commands:

        SERVICE_INSTANCE is the IP address of the share server that manila is unable to connect to.
        SERVICE_PORT is 22
        USERNAME = 'manila'
        PRIVATE_KEY_PATH = your config opt value for [backend]/path_to_private_key

        >>> import paramiko
        >>> paramiko.__version__
        >>> from manila import utils
        >>> paramiko.common.logging.basicConfig(level=paramiko.common.DEBUG)
        >>> mysshpool = utils.SSHPool(SERVICE_INSTANCE, SERVICE_PORT, None, USERNAME, None, privatekey=PRIVATE_KEY_PATH)
        >>> client = mysshpool.create()
       <<<SNIP DEBUG LOG>>>
       >>> stdin, stdout, stderr = client.exec_command('echo hi')

Note the debug log, and note any errors. If no errors, the output should be like this:

        >>> stdout.readlines()
        [u'hi\n']

If there are paramiko errors, please ensure you upgrade paramiko and try again. Sometimes, the distro-packaged paramiko may have a bug or two that won't be present in paramiko from PyPi, so it may be prudent to uninstall the distro paramiko (ex: python-paramiko) and install from PyPi (pip install paramiko)

If there are no paramiko errors, and no network issues, please let us know.