EqualLogic driver makes cinder-volume service unavailable if eqlx server closes the ssh connection

Bug #1410627 reported by Dunrong Huang
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Undecided
Dunrong Huang
Juno
Fix Released
Undecided
Eric Harney

Bug Description

The eqlx driver would identify one CLI command completion by looking for the system name prompt in the output ("ARRAY_NAME>"). But some cases where there is no "ARRAY_NAME>" in the output. The eqlx server closes the ssh connection immediately after sends such message, which lead to a infinite loop in _get_output() and 100% CPU utilization.

For example, by default, the maximum ssh connections cinder-volume will use is 5, if we have one more cinder-volume service, the server will send eqlx driver below message(no "ARRAY_NAME>") and close the connection.
       "Error: Number of open cli sessions reached maximum allowed value of 7"

Another example is that sometimes eqlx server just sends a message like this:
             Welcome to Group Manager
        Copyright 2001-2014 Dell Inc.
then close the connection.

Tags: drivers eqlx
Dunrong Huang (dunrong)
Changed in cinder:
assignee: nobody → Dunrong Huang (dunrong)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/147070

Mike Perez (thingee)
Changed in cinder:
status: New → In Progress
tags: added: juno-backport
tags: added: backport for proposed
removed: juno-backport
tags: added: juno
Eric Harney (eharney)
tags: removed: backport for juno proposed
Eric Harney (eharney)
summary: - EqualLogic driver makes cidner-volume service unavailable if eqlx server
+ EqualLogic driver makes cinder-volume service unavailable if eqlx server
closes the ssh connection
Changed in cinder:
assignee: Dunrong Huang (dunrong) → Eric Harney (eharney)
Eric Harney (eharney)
Changed in cinder:
assignee: Eric Harney (eharney) → Dunrong Huang (dunrong)
Mike Perez (thingee)
tags: added: drivers eqlx
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/147070
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=a762665595f0f38bacb50d1a032df1c82b3a4c60
Submitter: Jenkins
Branch: master

commit a762665595f0f38bacb50d1a032df1c82b3a4c60
Author: Dunrong Huang <email address hidden>
Date: Wed Jan 14 13:56:43 2015 +0800

    Fix eqlx endless loop when server closes the connection

    The eqlx driver would identify one CLI command completion by looking
    for the system name prompt in the output ("ARRAY_NAME>"). But some
    cases where there is no "ARRAY_NAME>" in the output. The eqlx server
    closes the ssh connection immediately after sends such message, which
    lead to a infinite loop in _get_output() and 100% CPU utilization.

    What the patch does is to check the return of chan.recv(), if the
    length of returned string is zero, which means server has closed the
    connection, the loop will be exited.

    Closes-Bug: #1410627

    Signed-off-by: Dunrong Huang <email address hidden>
    Change-Id: I600c38d98f0f808c98df010a89ac58ca8e43f1a3

Changed in cinder:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/147958

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/juno)

Reviewed: https://review.openstack.org/147958
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=7ebaf68a900ddf33dab086d331da78a9c8bd5603
Submitter: Jenkins
Branch: stable/juno

commit 7ebaf68a900ddf33dab086d331da78a9c8bd5603
Author: Dunrong Huang <email address hidden>
Date: Wed Jan 14 13:56:43 2015 +0800

    Fix eqlx endless loop when server closes the connection

    The eqlx driver would identify one CLI command completion by looking
    for the system name prompt in the output ("ARRAY_NAME>"). But some
    cases where there is no "ARRAY_NAME>" in the output. The eqlx server
    closes the ssh connection immediately after sends such message, which
    lead to a infinite loop in _get_output() and 100% CPU utilization.

    What the patch does is to check the return of chan.recv(), if the
    length of returned string is zero, which means server has closed the
    connection, the loop will be exited.

    Closes-Bug: #1410627

    Signed-off-by: Dunrong Huang <email address hidden>
    Change-Id: I600c38d98f0f808c98df010a89ac58ca8e43f1a3
    (cherry picked from commit a762665595f0f38bacb50d1a032df1c82b3a4c60)

Thierry Carrez (ttx)
Changed in cinder:
milestone: none → kilo-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: kilo-2 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.