OpenStack Shared File Systems Service (Manila)

Generic driver: stuck at ssh connect on check share-sever availability

Bug #1528203 reported by Shaohui Wang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Undecided
Shaohui Wang

Bug Description

...skipping...
ackages/manila/share/drivers/service_instance.py:633
2015-12-14 20:29:12.562 13695 DEBUG manila.share.drivers.service_instance [req-d02105f7-a583-4580-8520-7242b3161264 ] Waiting for instance 0cdaefa9-5d6d-45c1-bd31-f7fb87fed1a6 to be active. Current status: BUILD. wait_for_instance_to_be_active /usr/lib/python2.7/dist-p
ackages/manila/share/drivers/service_instance.py:633
2015-12-14 20:29:14.078 13695 DEBUG manila.share.drivers.service_instance [req-d02105f7-a583-4580-8520-7242b3161264 ] Waiting for instance 0cdaefa9-5d6d-45c1-bd31-f7fb87fed1a6 to be active. Current status: BUILD. wait_for_instance_to_be_active /usr/lib/python2.7/dist-p
ackages/manila/share/drivers/service_instance.py:633
2015-12-14 20:29:15.556 13695 DEBUG oslo_concurrency.lockutils [req-d02105f7-a583-4580-8520-7242b3161264 ] Lock "service_instance_get_or_create_security_group" acquired by "manila.share.drivers.service_instance._get_or_create_security_group" :: waited 0.000s inner /usr
/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:253
2015-12-14 20:29:15.975 13695 DEBUG oslo_concurrency.lockutils [req-d02105f7-a583-4580-8520-7242b3161264 ] Lock "service_instance_get_or_create_security_group" released by "manila.share.drivers.service_instance._get_or_create_security_group" :: held 0.420s inner /usr/l
ib/python2.7/dist-packages/oslo_concurrency/lockutils.py:265
2015-12-14 20:29:15.976 13695 DEBUG oslo_concurrency.lockutils [req-d02105f7-a583-4580-8520-7242b3161264 ] Lock "instantiate_network_helper" acquired by "manila.share.drivers.service_instance.network_helper" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_
concurrency/lockutils.py:253
2015-12-14 20:29:15.977 13695 DEBUG oslo_concurrency.lockutils [req-d02105f7-a583-4580-8520-7242b3161264 ] Lock "instantiate_network_helper" released by "manila.share.drivers.service_instance.network_helper" :: held 0.001s inner /usr/lib/python2.7/dist-packages/oslo_co
ncurrency/lockutils.py:265
2015-12-14 20:29:15.979 13695 DEBUG manila.share.drivers.service_instance [req-d02105f7-a583-4580-8520-7242b3161264 ] Adding security group 'd35daa8d-c1a4-4ea7-96c1-3195558015c0' to server '0cdaefa9-5d6d-45c1-bd31-f7fb87fed1a6'. _create_service_instance /usr/lib/python
2.7/dist-packages/manila/share/drivers/service_instance.py:539
2015-12-14 20:29:16.615 13695 DEBUG oslo_concurrency.lockutils [req-d02105f7-a583-4580-8520-7242b3161264 ] Lock "instantiate_network_helper" acquired by "manila.share.drivers.service_instance.network_helper" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_
concurrency/lockutils.py:253
2015-12-14 20:29:16.616 13695 DEBUG oslo_concurrency.lockutils [req-d02105f7-a583-4580-8520-7242b3161264 ] Lock "instantiate_network_helper" released by "manila.share.drivers.service_instance.network_helper" :: held 0.001s inner /usr/lib/python2.7/dist-packages/oslo_co
ncurrency/lockutils.py:265
2015-12-14 20:29:16.617 13695 DEBUG manila.share.drivers.service_instance [req-d02105f7-a583-4580-8520-7242b3161264 ] Checking server availability. _check_server_availability /usr/lib/python2.7/dist-packages/manila/share/drivers/service_instance.py:581
2015-12-14 20:29:19.618 13695 DEBUG manila.share.drivers.service_instance [req-d02105f7-a583-4580-8520-7242b3161264 ] [Errno 113] EHOSTUNREACH _test_server_connection /usr/lib/python2.7/dist-packages/manila/share/drivers/service_instance.py:595
2015-12-14 20:29:19.618 13695 DEBUG manila.share.drivers.service_instance [req-d02105f7-a583-4580-8520-7242b3161264 ] Server 10.254.0.10 is not available via SSH. Waiting... _test_server_connection /usr/lib/python2.7/dist-packages/manila/share/drivers/service_instance.
py:597
2015-12-14 20:29:24.619 13695 DEBUG manila.share.drivers.service_instance [req-d02105f7-a583-4580-8520-7242b3161264 ] Checking server availability. _check_server_availability /usr/lib/python2.7/dist-packages/manila/share/drivers/service_instance.py:581
2015-12-14 20:29:24.623 13695 DEBUG manila.share.drivers.service_instance [req-d02105f7-a583-4580-8520-7242b3161264 ] [Errno 111] ECONNREFUSED _test_server_connection /usr/lib/python2.7/dist-packages/manila/share/drivers/service_instance.py:595
2015-12-14 20:29:24.624 13695 DEBUG manila.share.drivers.service_instance [req-d02105f7-a583-4580-8520-7242b3161264 ] Server 10.254.0.10 is not available via SSH. Waiting... _test_server_connection /usr/lib/python2.7/dist-packages/manila/share/drivers/service_instance.
py:597
2015-12-14 20:29:29.624 13695 DEBUG manila.share.drivers.service_instance [req-d02105f7-a583-4580-8520-7242b3161264 ] Checking server availability. _check_server_availability /usr/lib/python2.7/dist-packages/manila/share/drivers/service_instance.py:581
2015-12-14 20:29:29.626 13695 DEBUG manila.share.drivers.service_instance [req-d02105f7-a583-4580-8520-7242b3161264 ] Server 10.254.0.10 is available via SSH. _test_server_connection /usr/lib/python2.7/dist-packages/manila/share/drivers/service_instance.py:592
2015-12-14 20:29:29.627 13695 DEBUG manila.share.drivers.generic [req-d02105f7-a583-4580-8520-7242b3161264 ] Creating share server 'a7f80545-2809-4bef-9ca2-799f4230642c' complete. _setup_server /usr/lib/python2.7/dist-packages/manila/share/drivers/generic.py:810
2015-12-14 20:29:29.629 13695 DEBUG paramiko.transport [-] starting thread (client mode): 0xbe6f9590L _log /usr/lib/python2.7/dist-packages/paramiko/transport.py:1428
2015-12-14 20:29:29.653 13695 INFO paramiko.transport [-] Connected (version 2.0, client OpenSSH_6.6.1p1)
2015-12-14 20:29:43.705 13696 DEBUG oslo_service.periodic_task [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Running periodic task ShareManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-12-14 20:29:43.706 13696 DEBUG manila.manager [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/dist-packages/manila/manager.py:142
2015-12-14 20:29:43.711 13696 DEBUG oslo_service.periodic_task [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Running periodic task ShareManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-12-14 20:29:43.711 13696 INFO manila.share.manager [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Updating share status
2015-12-14 20:29:43.712 13696 DEBUG manila.share.driver [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Updating share stats. _update_share_stats /usr/lib/python2.7/dist-packages/manila/share/driver.py:795
2015-12-14 20:30:08.828 13695 DEBUG oslo_service.periodic_task [req-c6400ba1-e313-4ea1-b678-407ace7e3954 ] Running periodic task ShareManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-12-14 20:30:08.829 13695 DEBUG manila.manager [req-c6400ba1-e313-4ea1-b678-407ace7e3954 ] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/dist-packages/manila/manager.py:142
2015-12-14 20:30:08.834 13695 DEBUG oslo_service.periodic_task [req-c6400ba1-e313-4ea1-b678-407ace7e3954 ] Running periodic task ShareManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-12-14 20:30:08.835 13695 INFO manila.share.manager [req-c6400ba1-e313-4ea1-b678-407ace7e3954 ] Updating share status
2015-12-14 20:30:08.836 13695 DEBUG manila.share.driver [req-c6400ba1-e313-4ea1-b678-407ace7e3954 ] Updating share stats. _update_share_stats /usr/lib/python2.7/dist-packages/manila/share/driver.py:795
2015-12-14 20:30:43.705 13696 DEBUG oslo_service.periodic_task [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Running periodic task ShareManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-12-14 20:30:43.706 13696 DEBUG manila.manager [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/dist-packages/manila/manager.py:142
2015-12-14 20:30:43.710 13696 DEBUG oslo_service.periodic_task [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Running periodic task ShareManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-12-14 20:30:43.711 13696 INFO manila.share.manager [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Updating share status
2015-12-14 20:30:43.712 13696 DEBUG manila.share.driver [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Updating share stats. _update_share_stats /usr/lib/python2.7/dist-packages/manila/share/driver.py:795
2015-12-14 20:31:08.828 13695 DEBUG oslo_service.periodic_task [req-c6400ba1-e313-4ea1-b678-407ace7e3954 ] Running periodic task ShareManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-12-14 20:31:08.829 13695 DEBUG manila.manager [req-c6400ba1-e313-4ea1-b678-407ace7e3954 ] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/dist-packages/manila/manager.py:142
2015-12-14 20:31:08.835 13695 DEBUG oslo_service.periodic_task [req-c6400ba1-e313-4ea1-b678-407ace7e3954 ] Running periodic task ShareManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-12-14 20:31:08.836 13695 INFO manila.share.manager [req-c6400ba1-e313-4ea1-b678-407ace7e3954 ] Updating share status
2015-12-14 20:31:08.836 13695 DEBUG manila.share.driver [req-c6400ba1-e313-4ea1-b678-407ace7e3954 ] Updating share stats. _update_share_stats /usr/lib/python2.7/dist-packages/manila/share/driver.py:795
2015-12-14 20:31:43.705 13696 DEBUG oslo_service.periodic_task [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Running periodic task ShareManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-12-14 20:31:43.706 13696 DEBUG manila.manager [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/dist-packages/manila/manager.py:142
2015-12-14 20:31:43.710 13696 DEBUG oslo_service.periodic_task [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Running periodic task ShareManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-12-14 20:31:43.711 13696 INFO manila.share.manager [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Updating share status
2015-12-14 20:31:43.712 13696 DEBUG manila.share.driver [req-25a24faa-860f-4b7c-aef0-aaaa1f356fd9 ] Updating share stats. _update_share_stats /usr/lib/python2.7/dist-packages/manila/share/driver.py:795
2015-12-14 20:32:08.828 13695 DEBUG oslo_service.periodic_task [req-c6400ba1-e313-4ea1-b678-407ace7e3954 ] Running periodic task ShareManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213

This thread will stuck forever, and this share-instance will stuck at 'Creating'.
I think we can use timeout param on creating SSHPool to avoid a loooong time create.

Shaohui Wang (darkwsh-g)
Changed in manila:
assignee: nobody → Shaohui Wang (darkwsh-g)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.openstack.org/261641

Changed in manila:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.openstack.org/261641
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=f6d54dde0c6746e6a756141cfd46928d445fc86e
Submitter: Jenkins
Branch: master

commit f6d54dde0c6746e6a756141cfd46928d445fc86e
Author: darkwsh <email address hidden>
Date: Fri Dec 25 23:05:03 2015 -0800

    Set timeout for parmiko ssh connection

    Set manila-server connection timeout enabled via configuration
    file on generic driver to prevent ssh connections getting
    stuck if manila-server is slow in responding or some network
    problems occur.

    Closes-Bug: #1528203
    Change-Id: Icd72c561aa37fad447e3f6e1eade3b2f57d74521

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/manila 2.0.0.0b2

This issue was fixed in the openstack/manila 2.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.