[SRU] There is now a dependency on paramiko v1.8.0

Bug #1150720 reported by Geraint North on 2013-03-06
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Cinder
High
Avishay Traeger
Folsom
High
Avishay Traeger
cinder (Ubuntu)
High
Unassigned
Raring
High
Unassigned
paramiko (Ubuntu)
High
Unassigned
Precise
High
Unassigned
Quantal
High
Unassigned
Raring
High
Unassigned

Bug Description

When bringing up Cinder with the storwize_svc driver, the default paramiko install on Power RHEL 6.3 (1.7.6) produced the following error when cinder-volume started:

SSHException: Error connecting via ssh: PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()

Moving up to paramiko 1.9.0 fixed the issue. We haven't diagnosed the root cause of the failure using 1.7.6.

Avishay Traeger (avishay-il) wrote :

I have seen this from another user as well.

Changed in cinder:
status: New → Confirmed
Changed in cinder:
assignee: nobody → Avishay Traeger (avishay-il)
Feilong Wang (flwang) wrote :

I also ran into the same issue on RHEL 6.3 with paramiko 1.7.6.

Fix proposed to branch: master
Review: https://review.openstack.org/24143

Changed in cinder:
status: Confirmed → In Progress

It looks like paramiko 1.8.0 released in Oct 2012 fixed this

This is the fix AFAICT: https://github.com/paramiko/paramiko/commits/f8239015

James Page (james-page) wrote :

Raising a distro task for Ubuntu; its to late to upgrade paramiko for 13.04 so we will have to cherry pick a fix to support this (and patch out the 1.9.0 requirement in cinder packaging).

Changed in paramiko (Ubuntu Raring):
importance: Undecided → High
status: New → Triaged
James Page (james-page) on 2013-03-13
Changed in paramiko (Ubuntu Quantal):
status: New → Triaged
Changed in paramiko (Ubuntu Precise):
status: New → Triaged
importance: Undecided → High
Changed in paramiko (Ubuntu Quantal):
importance: Undecided → High
no longer affects: cinder (Ubuntu Precise)
no longer affects: cinder (Ubuntu Quantal)
Changed in cinder (Ubuntu Raring):
status: New → Triaged
importance: Undecided → High
Avishay Traeger (avishay-il) wrote :

I have confirmation that Cinder works with 1.8.0, and have changed the requirement to 1.8.0. Thank you.

Changed in cinder:
importance: Undecided → High
summary: - There is now a dependency on paramiko v1.9.0
+ There is now a dependency on paramiko v1.8.0
tags: added: folsom-backport-potential
Chuck Short (zulcss) on 2013-03-18
summary: - There is now a dependency on paramiko v1.8.0
+ [SRU] There is now a dependency on paramiko v1.8.0
Chuck Short (zulcss) wrote :

[Impact] When starting cinder-volume version 2012.1 on 12.04, cinder-volume will give an error and exit:

SSHException: Error connecting via ssh: PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()

[Test Case]

1. Install 2012.1 on 12.04 from the Ubuntu Cloud Archive
2. Activate the storwize_svc driver
3. Start cinder-volume
4. Observe the exception when starting cinder-volume

[Regression Potential]

None

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package paramiko - 1.7.7.1-3.1ubuntu1

---------------
paramiko (1.7.7.1-3.1ubuntu1) raring; urgency=low

  * debian/patches/fix-paramiko-rng-errors.patch: Add
    additional Random.atfork() to prevent RNG errors in large runs,
    backported from upstream git repo. (LP: #1150720)
 -- Chuck Short <email address hidden> Mon, 18 Mar 2013 12:20:43 -0500

Changed in paramiko (Ubuntu Raring):
status: Triaged → Fix Released

Reviewed: https://review.openstack.org/24756
Committed: http://github.com/openstack/cinder/commit/db0595a4c516738f110482396e0fe90ecaf71b38
Submitter: Jenkins
Branch: stable/folsom

commit db0595a4c516738f110482396e0fe90ecaf71b38
Author: Avishay Traeger <email address hidden>
Date: Tue Mar 19 08:24:50 2013 +0200

    Backport paramiko 1.8.0 requirement to Folsom.

    Require paramiko >= 1.8.0

    There is a well-known bug in the Paramiko SSH library that causes
    these Exceptions:
    SSHException: Error connecting via ssh: PID check failed. RNG must be
    re-initialized after fork(). Hint: Try Random.atfork()

    This bug has been fixed in Paramiko, and therefore we should require
    the newer version for Cinder drivers that use Paramiko to function
    properly.

    Change-Id: Ia14b7b72393e2a2482ebde418b5477934779bf13
    Fixes: bug 1150720

tags: added: in-stable-folsom

Hello Geraint, or anyone else affected,

Accepted paramiko into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/paramiko/1.7.7.1-2ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in paramiko (Ubuntu Precise):
status: Triaged → Fix Committed
tags: added: verification-needed
Mark McLoughlin (markmc) on 2013-04-09
tags: removed: folsom-backport-potential in-stable-folsom
Adam Gandelman (gandelman-a) wrote :

@ Chuck

Can you please post a better test case? I'm trying to reproduce and verify the fix but I haven't been able to trigger the issue. I've followed your test case, and even gone as far as setting up remote access to another system for cinder to create an SSH connection, but haven't been able to trigger this on precise using cinder-volume 2012.2.1-0ubuntu1.1~cloud (UCA) and python-paramiko 1.7.7.1-2 (from 12.04 archive).

Also you mention this affects cinder 2012.1 (Essex) but cinder didn't exist until 2012.2 (Folsom).

Thanks

Adam Gandelman (gandelman-a) wrote :

Using a variation of a test case for the same bug found at http://redes-privadas-virtuales.blogspot.com/2013/02/paramiko-object-with-process-rng-must.html.

Looks like this is actually still in issue using the patched package in raring python-paramiko 1.7.7.1-3.1, and the similarly patched package in precise-proposed doesn't fix the issue either.

Test case:

import paramiko

from multiprocessing import Process

class B():
    def __init__(self):
        self.__process = None
        self.__ssh = paramiko.SSHClient()
        self.__ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        self.__ssh.connect('somelocalsystem', username='adam', password='xxxxxx')

    def start(self):
        self.__process = Process(target=self.__run_process)
        self.__process.start()

    def __run_process(self):
        _, stdout, _ = self.__ssh.exec_command("hostname")

b = B()
b.start()

Results in a exception on both /w both raring and precise-proposed packages:

  File "/usr/lib/python2.7/dist-packages/Crypto/Random/_UserFriendlyRNG.py", line 138, in _check_pid
    raise AssertionError("PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()")
AssertionError: PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
  File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 1578, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

Hello Geraint, or anyone else affected,

Accepted cinder into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cinder/2012.2.4-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

John Griffith (john-griffith) wrote :

Any updates from folks on this?

Geraint North (geraint-north) wrote :

I'm afraid that I'm not able to test on X86 Linux distributions, so someone else will need to verify.

Adam Gandelman (gandelman-a) wrote :

Is there any way to test this issue using cinder without access to a storwize device? I've gone as far as creating a mock device using an ssh server and pointing cinder there, but have not been able to trigger the issue with the patched and unpatched Ubuntu paramiko packages.

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, the OpenStack components have been deployed and configured across multiple nodes using quantal-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

These proposed packages were deployed and tested in several different configurations. Attached are tarballs with various test logs from each configuration. In addition to the base components, variables in deployments include:

quantal_folsom.tar: nova-network (FlatDHCP), glance (Ceph backend), cinder (Ceph backend),
quantal_folsom_nova-volume.tar: nova-network (FlatDHCP), glance (local file), nova-volume (iSCSI backend)
quantal_folsom_quantum.tar: quantum (OVS plugin), glance (Ceph backend), nova-volume (Ceph backend)

Please note the versions_tested file in each tarball, which contains details about relevant package versions installed and tested.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Stable review: https://review.openstack.org/24756

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Adam Gandelman (gandelman-a) wrote :

Test coverage log.

Adam Gandelman (gandelman-a) wrote :

Test coverage log.

Adam Gandelman (gandelman-a) wrote :

Test coverage log.

tags: added: verification-done
removed: verification-needed

Hello Geraint, or anyone else affected,

Accepted paramiko into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/paramiko/1.7.7.1-3ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in paramiko (Ubuntu Quantal):
status: Triaged → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Adam Gandelman (gandelman-a) wrote :

I've run through our openstack tests against quantal-proposed without issue.

Also verified no issue with using cinder's SSH connection pool class and python-paramiko_1.7.7.1-3ubuntu0.1, using the following test with up to 100 connections without issue:

import cinder.utils as utils
CONNS = 15

sshpool = utils.SSHPool('192.168.xx.xx', port=22, conn_timeout=10,
                        login='adam', password='xxx', min_size=CONNS, max_size=CONNS)
sshpool.create()

for i in range(1, CONNS):
    with sshpool.item() as ssh:
        print utils.ssh_execute(ssh, 'hostname', check_exit_code=False)

Not convinced the original issue here was ever something that affected our Ubuntu packages to begin with. The only reports here are reported using RHEL 6.4 , paramiko 1.7.6 and who-knows what combination of versions of other things in the mix (eventlet, greenlet, etc). (Fairly certain I was chasing another issue in comment #13)

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package paramiko - 1.7.7.1-2ubuntu1

---------------
paramiko (1.7.7.1-2ubuntu1) precise-proposed; urgency=low

  * debian/patches/fix-paramiko-rng-errors.patch: Backport upstream
    fix to prevent rng errors in large runs,
    f8239015ec427a2b5e62afa8370885894483a356. (LP: #1150720)
 -- Chuck Short <email address hidden> Mon, 18 Mar 2013 12:47:37 -0500

Changed in paramiko (Ubuntu Precise):
status: Fix Committed → Fix Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package paramiko - 1.7.7.1-3ubuntu0.1

---------------
paramiko (1.7.7.1-3ubuntu0.1) quantal-proposed; urgency=low

  * debian/patches/fix-paramiko-rng-errors.patch: Add
    additional Random.atfork() to prevent RNG errors in large runs,
    backported from upstream git repo. (LP: #1150720)
 -- Adam Gandelman <email address hidden> Wed, 05 Jun 2013 10:55:10 -0700

Changed in paramiko (Ubuntu Quantal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cinder - 2012.2.4-0ubuntu1

---------------
cinder (2012.2.4-0ubuntu1) quantal-proposed; urgency=low

  * Dropped patches, applied upstream:
    - debian/patches/CVE-2013-1664.patch: [fcf249d]
  * Resynchronize with stable/folsom (7916a2f4) (LP: #1179707):
    - [ec46ee0] NetApp: Folsom driver wrongly allows creation of volumes of
      different size from snapshots LP: 1164473
    - [00e8049] Cannot delete snapshot in "error" state LP: 1143661
    - [ce516f6] fanout_cast_to_server in kombu calls wrong method LP: 1074113
    - [cdb72ee] Enable direct testing with nosetests
    - [e63fa95] Netapp: delete_volume leaves qtree behind LP: 1099414
    - [8e702a1] NetApp: Can't delete a volume in error state LP: 1090167
    - [2e7f717] Don't have permission to delete a volume in error state
      LP: 1084273
    - [ad2dddd] Volume can't be deleted if tgt has had a reconnect. LP: 1159948
    - [db0595a] [SRU] There is now a dependency on paramiko v1.8.0
      (LP: #1150720)
    - [a616001] Only use iscsi_helper config option if using ISCSIDriver
    - [cbad3e3] Can not create volume snapshot when using NfsDriver LP: 1097266
    - [95c9f6f] Volume type extra specs update with empty body returns HTTP
      status 422 LP: 1090320
    - [aeece14] ISCSITargetRemoveFailed: Failed to remove iscsi target
      LP: 1101071
    - [fcf249d] DoS through XML entity expansion (CVE-2013-1664) LP: 1100282
  * debian/patches/remove_paramiko_req_vers.patch: Remove version bump on
    paramiko requirement that was backported to stable/folsom.
 -- Adam Gandelman <email address hidden> Thu, 25 Apr 2013 17:29:58 -0400

Changed in cinder (Ubuntu):
status: Triaged → Fix Released
Rolf Leggewie (r0lf) wrote :

raring has seen the end of its life and is no longer receiving any updates. Marking the raring task for this ticket as "Won't Fix".

Changed in cinder (Ubuntu Raring):
status: Triaged → Won't Fix
Eric Harney (eharney) wrote :

Cinder now depends on paramiko >=1.13.0.

Changed in cinder:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers