[SRU] There is now a dependency on paramiko v1.8.0

[Geraint North]

When bringing up Cinder with the storwize_svc driver, the default paramiko install on Power RHEL 6.3 (1.7.6) produced the following error when cinder-volume started:

SSHException: Error connecting via ssh: PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()

Moving up to paramiko 1.9.0 fixed the issue. We haven't diagnosed the root cause of the failure using 1.7.6.

[Avishay Traeger (avishay-il)]

I have seen this from another user as well.

[Feilong Wang (flwang)]

I also ran into the same issue on RHEL 6.3 with paramiko 1.7.6.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/24143

[Mark McLoughlin (markmc)]

It looks like paramiko 1.8.0 released in Oct 2012 fixed this

This is the fix AFAICT: https://github.com/paramiko/paramiko/commits/f8239015

[James Page (james-page)]

Raising a distro task for Ubuntu; its to late to upgrade paramiko for 13.04 so we will have to cherry pick a fix to support this (and patch out the 1.9.0 requirement in cinder packaging).

[Avishay Traeger (avishay-il)]

I have confirmation that Cinder works with 1.8.0, and have changed the requirement to 1.8.0. Thank you.

[Chuck Short (zulcss)]

[Impact] When starting cinder-volume version 2012.1 on 12.04, cinder-volume will give an error and exit:

SSHException: Error connecting via ssh: PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()

[Test Case]

1. Install 2012.1 on 12.04 from the Ubuntu Cloud Archive
2. Activate the storwize_svc driver
3. Start cinder-volume
4. Observe the exception when starting cinder-volume

[Regression Potential]

None

[Launchpad Janitor (janitor)]

This bug was fixed in the package paramiko - 1.7.7.1-3.1ubuntu1

---------------
paramiko (1.7.7.1-3.1ubuntu1) raring; urgency=low

  * debian/patches/fix-paramiko-rng-errors.patch: Add
    additional Random.atfork() to prevent RNG errors in large runs,
    backported from upstream git repo. (LP: #1150720)
 -- Chuck Short <email address hidden> Mon, 18 Mar 2013 12:20:43 -0500

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/24756

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/24756
Committed: http://github.com/openstack/cinder/commit/db0595a4c516738f110482396e0fe90ecaf71b38
Submitter: Jenkins
Branch: stable/folsom

commit db0595a4c516738f110482396e0fe90ecaf71b38
Author: Avishay Traeger <email address hidden>
Date: Tue Mar 19 08:24:50 2013 +0200

    Backport paramiko 1.8.0 requirement to Folsom.

    Require paramiko >= 1.8.0

    There is a well-known bug in the Paramiko SSH library that causes
    these Exceptions:
    SSHException: Error connecting via ssh: PID check failed. RNG must be
    re-initialized after fork(). Hint: Try Random.atfork()

    This bug has been fixed in Paramiko, and therefore we should require
    the newer version for Cinder drivers that use Paramiko to function
    properly.

    Change-Id: Ia14b7b72393e2a2482ebde418b5477934779bf13
    Fixes: bug 1150720

[Brian Murray (brian-murray)]

Hello Geraint, or anyone else affected,

Accepted paramiko into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/paramiko/1.7.7.1-2ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adam Gandelman (gandelman-a)]

@ Chuck

Can you please post a better test case? I'm trying to reproduce and verify the fix but I haven't been able to trigger the issue. I've followed your test case, and even gone as far as setting up remote access to another system for cinder to create an SSH connection, but haven't been able to trigger this on precise using cinder-volume 2012.2.1-0ubuntu1.1~cloud (UCA) and python-paramiko 1.7.7.1-2 (from 12.04 archive).

Also you mention this affects cinder 2012.1 (Essex) but cinder didn't exist until 2012.2 (Folsom).

Thanks

[Adam Gandelman (gandelman-a)]

Using a variation of a test case for the same bug found at http://redes-privadas-virtuales.blogspot.com/2013/02/paramiko-object-with-process-rng-must.html.

Looks like this is actually still in issue using the patched package in raring python-paramiko 1.7.7.1-3.1, and the similarly patched package in precise-proposed doesn't fix the issue either.

Test case:

import paramiko

from multiprocessing import Process

class B():
    def __init__(self):
        self.__process = None
        self.__ssh = paramiko.SSHClient()
        self.__ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        self.__ssh.connect('somelocalsystem', username='adam', password='xxxxxx')

    def start(self):
        self.__process = Process(target=self.__run_process)
        self.__process.start()

    def __run_process(self):
        _, stdout, _ = self.__ssh.exec_command("hostname")

b = B()
b.start()

Results in a exception on both /w both raring and precise-proposed packages:

  File "/usr/lib/python2.7/dist-packages/Crypto/Random/_UserFriendlyRNG.py", line 138, in _check_pid
    raise AssertionError("PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()")
AssertionError: PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
  File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 1578, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

[Dave Walker (davewalker)]

Hello Geraint, or anyone else affected,

Accepted cinder into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cinder/2012.2.4-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[John Griffith (john-griffith)]

Any updates from folks on this?

[Geraint North (geraint-north)]

I'm afraid that I'm not able to test on X86 Linux distributions, so someone else will need to verify.

[Adam Gandelman (gandelman-a)]

Is there any way to test this issue using cinder without access to a storwize device? I've gone as far as creating a mock device using an ssh server and pointing cinder there, but have not been able to trigger the issue with the patched and unpatched Ubuntu paramiko packages.

[Adam Gandelman (gandelman-a)]

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, the OpenStack components have been deployed and configured across multiple nodes using quantal-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

These proposed packages were deployed and tested in several different configurations. Attached are tarballs with various test logs from each configuration. In addition to the base components, variables in deployments include:

quantal_folsom.tar: nova-network (FlatDHCP), glance (Ceph backend), cinder (Ceph backend),
quantal_folsom_nova-volume.tar: nova-network (FlatDHCP), glance (local file), nova-volume (iSCSI backend)
quantal_folsom_quantum.tar: quantum (OVS plugin), glance (Ceph backend), nova-volume (Ceph backend)

Please note the versions_tested file in each tarball, which contains details about relevant package versions installed and tested.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Stable review: https://review.openstack.org/24756

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

[Adam Gandelman (gandelman-a)]

Test coverage log.

[Adam Gandelman (gandelman-a)]

Test coverage log.

[Adam Gandelman (gandelman-a)]

Test coverage log.

[Adam Conrad (adconrad)]

Hello Geraint, or anyone else affected,

Accepted paramiko into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/paramiko/1.7.7.1-3ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adam Gandelman (gandelman-a)]

I've run through our openstack tests against quantal-proposed without issue.

Also verified no issue with using cinder's SSH connection pool class and python-paramiko_1.7.7.1-3ubuntu0.1, using the following test with up to 100 connections without issue:

import cinder.utils as utils
CONNS = 15

sshpool = utils.SSHPool('192.168.xx.xx', port=22, conn_timeout=10,
                        login='adam', password='xxx', min_size=CONNS, max_size=CONNS)
sshpool.create()

for i in range(1, CONNS):
    with sshpool.item() as ssh:
        print utils.ssh_execute(ssh, 'hostname', check_exit_code=False)

Not convinced the original issue here was ever something that affected our Ubuntu packages to begin with. The only reports here are reported using RHEL 6.4 , paramiko 1.7.6 and who-knows what combination of versions of other things in the mix (eventlet, greenlet, etc). (Fairly certain I was chasing another issue in comment #13)

[Launchpad Janitor (janitor)]

This bug was fixed in the package paramiko - 1.7.7.1-2ubuntu1

---------------
paramiko (1.7.7.1-2ubuntu1) precise-proposed; urgency=low

  * debian/patches/fix-paramiko-rng-errors.patch: Backport upstream
    fix to prevent rng errors in large runs,
    f8239015ec427a2b5e62afa8370885894483a356. (LP: #1150720)
 -- Chuck Short <email address hidden> Mon, 18 Mar 2013 12:47:37 -0500

[Adam Conrad (adconrad)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package paramiko - 1.7.7.1-3ubuntu0.1

---------------
paramiko (1.7.7.1-3ubuntu0.1) quantal-proposed; urgency=low

  * debian/patches/fix-paramiko-rng-errors.patch: Add
    additional Random.atfork() to prevent RNG errors in large runs,
    backported from upstream git repo. (LP: #1150720)
 -- Adam Gandelman <email address hidden> Wed, 05 Jun 2013 10:55:10 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package cinder - 2012.2.4-0ubuntu1

---------------
cinder (2012.2.4-0ubuntu1) quantal-proposed; urgency=low

  * Dropped patches, applied upstream:
    - debian/patches/CVE-2013-1664.patch: [fcf249d]
  * Resynchronize with stable/folsom (7916a2f4) (LP: #1179707):
    - [ec46ee0] NetApp: Folsom driver wrongly allows creation of volumes of
      different size from snapshots LP: 1164473
    - [00e8049] Cannot delete snapshot in "error" state LP: 1143661
    - [ce516f6] fanout_cast_to_server in kombu calls wrong method LP: 1074113
    - [cdb72ee] Enable direct testing with nosetests
    - [e63fa95] Netapp: delete_volume leaves qtree behind LP: 1099414
    - [8e702a1] NetApp: Can't delete a volume in error state LP: 1090167
    - [2e7f717] Don't have permission to delete a volume in error state
      LP: 1084273
    - [ad2dddd] Volume can't be deleted if tgt has had a reconnect. LP: 1159948
    - [db0595a] [SRU] There is now a dependency on paramiko v1.8.0
      (LP: #1150720)
    - [a616001] Only use iscsi_helper config option if using ISCSIDriver
    - [cbad3e3] Can not create volume snapshot when using NfsDriver LP: 1097266
    - [95c9f6f] Volume type extra specs update with empty body returns HTTP
      status 422 LP: 1090320
    - [aeece14] ISCSITargetRemoveFailed: Failed to remove iscsi target
      LP: 1101071
    - [fcf249d] DoS through XML entity expansion (CVE-2013-1664) LP: 1100282
  * debian/patches/remove_paramiko_req_vers.patch: Remove version bump on
    paramiko requirement that was backported to stable/folsom.
 -- Adam Gandelman <email address hidden> Thu, 25 Apr 2013 17:29:58 -0400

[Rolf Leggewie (r0lf)]

raring has seen the end of its life and is no longer receiving any updates. Marking the raring task for this ticket as "Won't Fix".

[Eric Harney (eharney)]

Cinder now depends on paramiko >=1.13.0.