We have a bug using Generic driver where SSH key is not imported to Nova instance, and Manila cannot connect to it. Here is error:
2015-08-28 16:10:03.134 DEBUG manila.share.drivers.service_instance [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Server 10.254.0.56 is not available via SSH. Waiting... from (pid=20228) _test_server_connection /opt/stack/manila/manila/share/drivers/service_instance.py:594
2015-08-28 16:10:08.135 DEBUG manila.share.drivers.service_instance [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Checking server availability. from (pid=20228) _check_server_availability /opt/stack/manila/manila/share/drivers/service_instance.py:578
2015-08-28 16:10:08.150 DEBUG manila.share.drivers.service_instance [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Server 10.254.0.56 is available via SSH. from (pid=20228) _test_server_connection /opt/stack/manila/manila/share/drivers/service_instance.py:589
2015-08-28 16:10:08.164 DEBUG paramiko.transport [-] starting thread (client mode): 0xe478de90L from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:09.58 DEBUG oslo_service.loopingcall [req-fc8721cd-a2e0-4316-a18a-2edcb5512543 None None] Fixed interval looping call 'manila.service.Service.report_state' sleeping for 9.44 seconds from (pid=20228) _run_loop /usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py:121
2015-08-28 16:10:09.749 INFO paramiko.transport [-] Connected (version 2.0, client OpenSSH_6.6.1p1)
2015-08-28 16:10:09.941 DEBUG paramiko.transport [-] kex algos:[<email address hidden>', u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group-exchange-sha1', u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1'] server key:[u'ssh-rsa', u'ssh-dss', u'ecdsa-sha2-nistp256', u'ssh-ed25519'] client encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'arcfour256', u'arcfour128', <email address hidden>', <email address hidden>', <email address hidden>', u'aes128-cbc', u'3des-cbc', u'blowfish-cbc', u'cast128-cbc', u'aes192-cbc', u'aes256-cbc', u'arcfour', <email address hidden>'] server encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'arcfour256', u'arcfour128', <email address hidden>', <email address hidden>', <email address hidden>', u'aes128-cbc', u'3des-cbc', u'blowfish-cbc', u'cast128-cbc', u'aes192-cbc', u'aes256-cbc', u'arcfour', <email address hidden>'] client mac:[<email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', u'hmac-md5', u'hmac-sha1', <email address hidden>', <email address hidden>', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-ripemd160', <email address hidden>', u'hmac-sha1-96', u'hmac-md5-96'] server mac:[<email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', u'hmac-md5', u'hmac-sha1', <email address hidden>', <email address hidden>', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-ripemd160', <email address hidden>', u'hmac-sha1-96', u'hmac-md5-96'] client compress:[u'none', <email address hidden>'] server compress:[u'none', <email address hidden>'] client lang:[u''] server lang:[u''] kex follows?False from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:09.943 DEBUG paramiko.transport [-] Ciphers agreed: local=aes128-ctr, remote=aes128-ctr from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:09.944 DEBUG paramiko.transport [-] using kex diffie-hellman-group14-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:11.822 DEBUG paramiko.transport [-] Switch to new keys ... from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:11.823 DEBUG paramiko.transport [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Adding ssh-rsa host key for 10.254.0.56: 507393874640d1fae157871882d48ec0 from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:11.825 DEBUG paramiko.transport [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Trying key cc751c207f5e2649d7d18b9d5cb6d4d3 from /home/stack/.ssh/id_rsa from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:11.850 DEBUG paramiko.transport [-] userauth is OK from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:12.439 INFO paramiko.transport [-] Authentication (publickey) failed.
2015-08-28 16:10:12.452 ERROR manila.utils [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Check whether private key or password are correctly set. Error connecting via ssh: not a valid EC private key file
2015-08-28 16:10:12.452 WARNING manila.share.manager [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Server Information in exception can not be written to db : Invalid server_details 'None'
2015-08-28 16:10:12.523 DEBUG oslo_concurrency.lockutils [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Lock "share_manager_764f7728-0c6d-4f9b-8a59-48d4d079e632" released by "manila.share.manager._provide_share_server_for_share" :: held 205.643s from (pid=20228) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:265
2015-08-28 16:10:12.523 ERROR manila.share.manager [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Failed to get share server for share instance creation.
2015-08-28 16:10:12.542 ERROR oslo_messaging.rpc.dispatcher [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Exception during message handling: Check whether private key or password are correctly set. Error connecting via ssh: not a valid EC private key file
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 404, in create_share_instance
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher {'status': constants.STATUS_ERROR}
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 395, in create_share_instance
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher snapshot=snapshot_ref
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 344, in _provide_share_server_for_share
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher return _provide_share_server_for_share()
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 336, in _provide_share_server_for_share
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher context, compatible_share_server)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 877, in _setup_server
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher self.driver.deallocate_network(context, share_server['id'])
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 828, in _setup_server
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher network_info, metadata=metadata)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/driver.py", line 362, in setup_server
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher return self._setup_server(*args, **kwargs)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/generic.py", line 796, in _setup_server
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher helper.init_helper(server)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/generic.py", line 1019, in init_helper
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher self._ssh_exec(server, ['sudo', 'exportfs'])
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/generic.py", line 160, in _ssh_exec
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher ssh = ssh_pool.create()
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/utils.py", line 117, in create
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher raise exception.SSHException(msg)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher SSHException: Check whether private key or password are correctly set. Error connecting via ssh: not a valid EC private key file
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher
Once such error appears, all share servers fail with this error. Reproducing on a lab shown that SSH key is absent on machine.
Because of this bug our Tempst CI jobs fail from time to time.
Setting importance as "critical" because it is blocker for merging gerrit changes. As this job votable.