HP LeftHand driver fails with Paramiko 1.13.0

[Jim Branen]

When the HP LeftHand driver is configured in legacy mode it will fail with the following exception, if paramiko 1.13.0 is installed:

2014-03-27 13:33:22.189 DEBUG cinder.openstack.common.lockutils [req-c2080e55-ec3e-40e3-a7a6-329e48d22295 None None] Released file lock "lefthand" at /opt/stack
/data/cinder/cinder-lefthand for method "get_volume_stats"... from (pid=35801) inner /opt/stack/cinder/cinder/openstack/common/lockutils.py:239
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 346, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/semaphore.py", line 121, in _do_acquire
    waiter.switch()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/cinder/cinder/openstack/common/service.py", line 483, in run_service
    service.start()
  File "/opt/stack/cinder/cinder/service.py", line 103, in start
    self.manager.init_host()
  File "/opt/stack/cinder/cinder/volume/manager.py", line 308, in init_host
    self.publish_service_capabilities(ctxt)
  File "/opt/stack/cinder/cinder/volume/manager.py", line 1105, in publish_service_capabilities
    self._report_driver_status(context)
  File "/opt/stack/cinder/cinder/volume/manager.py", line 1094, in _report_driver_status
    volume_stats = self.driver.get_volume_stats(refresh=True)
  File "/opt/stack/cinder/cinder/openstack/common/lockutils.py", line 233, in inner
    retval = f(*args, **kwargs)
  File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_lefthand_iscsi.py", line 121, in get_volume_stats
    data = self.proxy.get_volume_stats(refresh)
  File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_lefthand_cliq_proxy.py", line 421, in get_volume_stats
    self._update_backend_status()
  File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_lefthand_cliq_proxy.py", line 436, in _update_backend_status
    'clusterName': self.configuration.san_clustername})
  File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_lefthand_cliq_proxy.py", line 109, in _cliq_run_xml
    result_xml = etree.fromstring(out)
  File "lxml.etree.pyx", line 3003, in lxml.etree.fromstring (src/lxml/lxml.etree.c:67314)
  File "parser.pxi", line 1724, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:101147)
ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

This bug also exists with the Havana version of the driver. cinder.volume.drivers.san.hp_lefthand.HpSanISCSIDriver

Workaround: install paramiko 1.10.0

[John Griffith (john-griffith)]

seems like we need to either get rid of "legacy mode" (I'm uncertain if/why we need the multiple modes at this point) or fix the issues that you're hitting here.

Pinning paramiko is not something I want to do because of other drivers and projects that are using paramiko and the global req's file.

Also, it's certainly too late to do anything here for Icehouse other than "fixing" the lefthand driver. Same holds true for Havana, modifying the requirement is not really an option.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/85483

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/85489

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/85483
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=57e8cdb9dd8e24919bc2243438a6bded53491b3e
Submitter: Jenkins
Branch: master

commit 57e8cdb9dd8e24919bc2243438a6bded53491b3e
Author: Jim Branen <email address hidden>
Date: Fri Apr 4 13:36:42 2014 -0700

    Fixes HP LeftHand driver with Paramiko 1.13.0

    With Paramiko 1.13.0, the method exec_command now returns Unicode.
    This causes a problem when the driver tries to build the XML returned
    from the LeftHand array. The XML header returned from the array defines
    the encoding as encoding=UTF-8. Therefore, we must now ensure the
    encoding passed to the parser is utf-8.

    Change-Id: I7b504626e2d9a0ee2b62820b11f56eb136e31987
    closes-bug: #1298608

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/88091

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/110800

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/110800
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=00b7574d9a6e3818b3d032a8de05a03ddd94c2d6
Submitter: Jenkins
Branch: stable/icehouse

commit 00b7574d9a6e3818b3d032a8de05a03ddd94c2d6
Author: Jim Branen <email address hidden>
Date: Fri Apr 4 13:36:42 2014 -0700

    Fixes HP LeftHand driver with Paramiko 1.13.0

    With Paramiko 1.13.0, the method exec_command now returns Unicode.
    This causes a problem when the driver tries to build the XML returned
    from the LeftHand array. The XML header returned from the array defines
    the encoding as encoding=UTF-8. Therefore, we must now ensure the
    encoding passed to the parser is utf-8.

    Change-Id: I7b504626e2d9a0ee2b62820b11f56eb136e31987
    closes-bug: #1298608
    (cherry picked from commit 57e8cdb9dd8e24919bc2243438a6bded53491b3e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/88091
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=6b05dbf43467b1b39fee865a94623cdcac15f09d
Submitter: Jenkins
Branch: stable/havana

commit 6b05dbf43467b1b39fee865a94623cdcac15f09d
Author: Jim Branen <email address hidden>
Date: Fri Apr 4 13:36:42 2014 -0700

    Fixes HP LeftHand driver with Paramiko 1.13.0

    With Paramiko 1.13.0, the method exec_command now returns Unicode.
    This causes a problem when the driver tries to build the XML returned
    from the LeftHand array. The XML header returned from the array defines
    the encoding as encoding=UTF-8. Therefore, we must now ensure the
    encoding passed to the parser is utf-8.

    closes-bug: #1298608
    (cherry picked from commit 57e8cdb9dd8e24919bc2243438a6bded53491b3e)

    Conflicts:

     cinder/tests/test_hplefthand.py
     cinder/volume/drivers/san/hp_lefthand.py

    Change-Id: I7b504626e2d9a0ee2b62820b11f56eb136e31987