OpenStack Shared File Systems Service (Manila)

Service instance does not get ssh key with Generic driver

Bug #1480870 reported by Valeriy Ponomaryov on 2015-08-03

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Shared File Systems Service (Manila)	Won't Fix	Low	Unassigned

Bug Description

We have a bug using Generic driver where SSH key is not imported to Nova instance, and Manila cannot connect to it. Here is error:

2015-08-28 16:10:03.134 DEBUG manila.share.drivers.service_instance [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Server 10.254.0.56 is not available via SSH. Waiting... from (pid=20228) _test_server_connection /opt/stack/manila/manila/share/drivers/service_instance.py:594
2015-08-28 16:10:08.135 DEBUG manila.share.drivers.service_instance [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Checking server availability. from (pid=20228) _check_server_availability /opt/stack/manila/manila/share/drivers/service_instance.py:578
2015-08-28 16:10:08.150 DEBUG manila.share.drivers.service_instance [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Server 10.254.0.56 is available via SSH. from (pid=20228) _test_server_connection /opt/stack/manila/manila/share/drivers/service_instance.py:589
2015-08-28 16:10:08.164 DEBUG paramiko.transport [-] starting thread (client mode): 0xe478de90L from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:09.58 DEBUG oslo_service.loopingcall [req-fc8721cd-a2e0-4316-a18a-2edcb5512543 None None] Fixed interval looping call 'manila.service.Service.report_state' sleeping for 9.44 seconds from (pid=20228) _run_loop /usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py:121
2015-08-28 16:10:09.749 INFO paramiko.transport [-] Connected (version 2.0, client OpenSSH_6.6.1p1)
2015-08-28 16:10:09.941 DEBUG paramiko.transport [-] kex algos:[<email address hidden>', u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group-exchange-sha1', u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1'] server key:[u'ssh-rsa', u'ssh-dss', u'ecdsa-sha2-nistp256', u'ssh-ed25519'] client encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'arcfour256', u'arcfour128', <email address hidden>', <email address hidden>', <email address hidden>', u'aes128-cbc', u'3des-cbc', u'blowfish-cbc', u'cast128-cbc', u'aes192-cbc', u'aes256-cbc', u'arcfour', <email address hidden>'] server encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'arcfour256', u'arcfour128', <email address hidden>', <email address hidden>', <email address hidden>', u'aes128-cbc', u'3des-cbc', u'blowfish-cbc', u'cast128-cbc', u'aes192-cbc', u'aes256-cbc', u'arcfour', <email address hidden>'] client mac:[<email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', u'hmac-md5', u'hmac-sha1', <email address hidden>', <email address hidden>', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-ripemd160', <email address hidden>', u'hmac-sha1-96', u'hmac-md5-96'] server mac:[<email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', <email address hidden>', u'hmac-md5', u'hmac-sha1', <email address hidden>', <email address hidden>', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-ripemd160', <email address hidden>', u'hmac-sha1-96', u'hmac-md5-96'] client compress:[u'none', <email address hidden>'] server compress:[u'none', <email address hidden>'] client lang:[u''] server lang:[u''] kex follows?False from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:09.943 DEBUG paramiko.transport [-] Ciphers agreed: local=aes128-ctr, remote=aes128-ctr from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:09.944 DEBUG paramiko.transport [-] using kex diffie-hellman-group14-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:11.822 DEBUG paramiko.transport [-] Switch to new keys ... from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:11.823 DEBUG paramiko.transport [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Adding ssh-rsa host key for 10.254.0.56: 507393874640d1fae157871882d48ec0 from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:11.825 DEBUG paramiko.transport [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Trying key cc751c207f5e2649d7d18b9d5cb6d4d3 from /home/stack/.ssh/id_rsa from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:11.850 DEBUG paramiko.transport [-] userauth is OK from (pid=20228) _log /usr/local/lib/python2.7/dist-packages/paramiko/transport.py:1430
2015-08-28 16:10:12.439 INFO paramiko.transport [-] Authentication (publickey) failed.
2015-08-28 16:10:12.452 ERROR manila.utils [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Check whether private key or password are correctly set. Error connecting via ssh: not a valid EC private key file
2015-08-28 16:10:12.452 WARNING manila.share.manager [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Server Information in exception can not be written to db : Invalid server_details 'None'
2015-08-28 16:10:12.523 DEBUG oslo_concurrency.lockutils [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Lock "share_manager_764f7728-0c6d-4f9b-8a59-48d4d079e632" released by "manila.share.manager._provide_share_server_for_share" :: held 205.643s from (pid=20228) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:265
2015-08-28 16:10:12.523 ERROR manila.share.manager [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Failed to get share server for share instance creation.
2015-08-28 16:10:12.542 ERROR oslo_messaging.rpc.dispatcher [req-cf4eb97b-878f-4ba4-8063-c8deeb061561 54aefb098bfb4ed5b3c1c587ef59d69a 9b10fef48f3b4a12ac6170ce47e7aeeb] Exception during message handling: Check whether private key or password are correctly set. Error connecting via ssh: not a valid EC private key file
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 404, in create_share_instance
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher {'status': constants.STATUS_ERROR}
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 395, in create_share_instance
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher snapshot=snapshot_ref
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 344, in _provide_share_server_for_share
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher return _provide_share_server_for_share()
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 336, in _provide_share_server_for_share
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher context, compatible_share_server)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 877, in _setup_server
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher self.driver.deallocate_network(context, share_server['id'])
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 828, in _setup_server
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher network_info, metadata=metadata)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/driver.py", line 362, in setup_server
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher return self._setup_server(*args, **kwargs)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/generic.py", line 796, in _setup_server
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher helper.init_helper(server)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/generic.py", line 1019, in init_helper
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher self._ssh_exec(server, ['sudo', 'exportfs'])
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/generic.py", line 160, in _ssh_exec
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher ssh = ssh_pool.create()
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/utils.py", line 117, in create
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher raise exception.SSHException(msg)
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher SSHException: Check whether private key or password are correctly set. Error connecting via ssh: not a valid EC private key file
2015-08-28 16:10:12.542 TRACE oslo_messaging.rpc.dispatcher

Once such error appears, all share servers fail with this error. Reproducing on a lab shown that SSH key is absent on machine.

Because of this bug our Tempst CI jobs fail from time to time.

See original description

Tags:

Revision history for this message

Valeriy Ponomaryov (vponomaryov) wrote on 2015-08-03:

Setting importance as "critical" because it is blocker for merging gerrit changes. As this job votable.

Changed in manila:
importance:	Undecided → Critical
milestone:	none → liberty-3
tags:	added: ci tempest

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-04: Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.openstack.org/209215

Changed in manila:
assignee:	nobody → Valeriy Ponomaryov (vponomaryov)
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-05: Change abandoned on manila (master)

Change abandoned by Valeriy Ponomaryov (<email address hidden>) on branch: master
Review: https://review.openstack.org/209215
Reason: Does not make influence. Retry is not needed as bug appears anytime after first occurrence.

Revision history for this message

Valeriy Ponomaryov (vponomaryov) wrote on 2015-08-05: Re: tempest scenario job is unstable

This bug was reproduced with Rally scenarios. see:

http://logs.openstack.org/29/188029/11/experimental/gate-rally-dsvm-manila-multibackend/cdeda30/logs/screen-m-shr.txt.gz?level=TRACE

Main feature of scenario - create shares concurrently with separate share networks.

Revision history for this message

Valeriy Ponomaryov (vponomaryov) wrote on 2015-08-07:

Setting as "high" since it is not blocker anymore.

Changed in manila:
importance:	Critical → High

Valeriy Ponomaryov (vponomaryov) on 2015-08-28

summary:	- tempest scenario job is unstable + Service instance does not get ssh key with Generic driver
description:	updated

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-09-02: Related fix proposed to manila (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/219788

Thierry Carrez (ttx) on 2015-09-03

Changed in manila:
milestone:	liberty-3 → liberty-rc1

Valeriy Ponomaryov (vponomaryov) on 2015-09-04

Changed in manila:
importance:	High → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-09-04: Related fix merged to manila (master)

Reviewed: https://review.openstack.org/219788
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=6c44bff51610ddaeb661c3124a12c80f766c693a
Submitter: Jenkins
Branch: master

commit 6c44bff51610ddaeb661c3124a12c80f766c693a
Author: Valeriy Ponomaryov <email address hidden>
Date: Wed Sep 2 19:55:13 2015 +0300

Add possibility to setup password for generic driver

Make it possible to setup password for generic driver service image and
scenario tests. Also allow to just disable its usage not unsetting the password

It is useful for testing purpose when we have some problems with SSH keys
injection to Nova VMs.

Change-Id: I26dd32b1de8cceeaa6dc674092efec683df71889
Related-Bug: #1480870

Valeriy Ponomaryov (vponomaryov) on 2015-09-08

Changed in manila:
milestone:	liberty-rc1 → none
status:	In Progress → Confirmed
assignee:	Valeriy Ponomaryov (vponomaryov) → nobody

Ben Swartzlander (bswartz) on 2015-09-08

Changed in manila:
milestone:	none → liberty-rc1

Revision history for this message

Ben Swartzlander (bswartz) wrote on 2015-09-15:

Related? https://bugs.launchpad.net/neutron/+bug/1453350

Ben Swartzlander (bswartz) on 2015-09-17

Changed in manila:
milestone:	liberty-rc1 → mitaka-1

Ben Swartzlander (bswartz) on 2015-12-22

Changed in manila:
milestone:	mitaka-1 → mitaka-2

Valeriy Ponomaryov (vponomaryov) on 2016-01-21

Changed in manila:
importance:	Medium → Low

Ben Swartzlander (bswartz) on 2016-01-21

Changed in manila:
milestone:	mitaka-2 → mitaka-3

Ben Swartzlander (bswartz) on 2016-03-03

Changed in manila:
milestone:	mitaka-3 → none

Tom Barron (tpb) on 2018-06-19

tags:

added: driver generic

Revision history for this message

Jason Grosso (jgrosso) wrote on 2019-03-18:

Any plans to fix this, and is it still happening?

Revision history for this message

Jason Grosso (jgrosso) wrote on 2019-04-16:

#10

If this bug is still occurring please re-open or create a new defect please.

Changed in manila:
status:	Confirmed → Won't Fix

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.