devstack gate tests fail intermittently with 401 error

Bug #1075838 reported by John Griffith
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Critical
Vish Ishaya
Folsom
Won't Fix
Undecided
Unassigned

Bug Description

devstack gate tests fail intermittently with 401 error on volume-create

Changed in cinder:
status: New → Confirmed
importance: Undecided → Critical
milestone: none → grizzly-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/15541

Changed in cinder:
assignee: nobody → Vish Ishaya (vishvananda)
status: Confirmed → In Progress
Revision history for this message
Rushi Agrawal (rushiagr) wrote :

Is this bug giving error as mentioned in
https://bugs.launchpad.net/devstack/+bug/1075630
? I submitted it in devstack yesterday.

Revision history for this message
John Griffith (john-griffith) wrote :

Rushi... yep! That was the one I was looking for, thanks for pointing it out. I've linked yours to this one since this has a patch proposed.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/15541
Committed: http://github.com/openstack/cinder/commit/4e619fbef8671697bfec8e752818944c9f96cd72
Submitter: Jenkins
Branch: master

commit 4e619fbef8671697bfec8e752818944c9f96cd72
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Nov 6 20:12:00 2012 -0800

    Fix 401 from auth_token middleware

    The openssl popen call to verify the token returns an error code
    of 2 when a file doesn't exist. Normally this is caught and the
    relevant file is downloaded, but monkeypatching os causes this
    error code to be eaten and cinder to return a 401. Note that this
    is likely a race condition that depends on exactly when greenthreads
    are switching that is causing the return code from Popen to get
    overwritten.

    It is possible that this is exposing an underlying bug in eventlet
    but the workaround is simply to make sure that we don't monkeypatch
    os.

    Fixes bug 1075838

    Change-Id: I69ba75136e32e95c6fdf108f0c4fe21a5e3bdbe6

Changed in cinder:
status: In Progress → Fix Committed
tags: added: backport-potential folsom
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/15594

Thierry Carrez (ttx)
Changed in cinder:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/17071

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/17071
Committed: http://github.com/openstack/cinder/commit/44a2b8949fba07d9a3427137d59fe81fafbc3458
Submitter: Jenkins
Branch: master

commit 44a2b8949fba07d9a3427137d59fe81fafbc3458
Author: John Griffith <email address hidden>
Date: Wed Nov 28 12:19:14 2012 -0700

    Revert changes to monkey_patch.

    As a work-around to deal with bug 1075838, we modified
    monkey_patch in the Cinder binaries. The root cause of the
    issue is/was actually a bug in eventlet but this was a quick
    work-around.

    Since the change Change-Id: I69ba75136e32e95c6fdf108f0c4fe21a5e3bdbe6
    to do this landed, the problem has also been addressed in Keystone.

    So, let's revert this patch to get back the greenthread performance
    that we may have lost with this change. For now, everything should
    be addressed as long as we don't implement multi api worker in Cinder,
    if we do and the evenlet fix hasn't been released we'll need to turn
    the modification back on for bin/cinder-api.

    Change-Id: Ie845bade85dbceacd91f15e082325602d9a79c74

Thierry Carrez (ttx)
Changed in cinder:
milestone: grizzly-1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.