keystone ipv6 tests fail

Bug #1176204 reported by Brant Knudson on 2013-05-03
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Medium
Brant Knudson
Folsom
Undecided
Unassigned
Grizzly
Undecided
Dolph Mathews

Bug Description

Keystone unit tests failed when run on oneiric3 with an error that it couldn't resolve IPv6 addresses (::1).

http://logs.openstack.org/27787/7/check/gate-keystone-python26/3785/console.html
http://logs.openstack.org/27787/7/check/gate-keystone-python27/3797/console.html

The change under test wasn't related to IP changes.

For the test that fails, it's trying to start up Keystone listening on ::1 and it fails to resolve ::1.

I couldn't recreate this on my local system: ./run_tests.sh test_ipv6

Clark Boylan (cboylan) wrote :

So I am not entirely convinced this is a problem with the Jenkins slaves. On oneiric3 which is where the python26 test failed `nc -l ::1 0` works fine. And looking in the eventlet source code I see that getaddrinfo (which is in the tracebacks provided by nose) does not support ipv6. https://bitbucket.org/eventlet/eventlet/src/e0f578180d7d82d2ed3d8a96d520103503c524ec/eventlet/support/greendns.py?at=0.12#cl-166

I am not sure yet why this just cropped up and why it does not fail on your local node. Maybe the test is optimistic? eg if ipv6 available then run the tests else skip. I haven't read the tests yet... I am going to add keystone to the bug for now while we sort out where the issue is.

Clark Boylan (cboylan) wrote :

The ipv6 tests are run optimistically based on the results of skip_if_no_ipv6() at https://github.com/openstack/keystone/blob/master/keystone/test.py#L352.

Brant Knudson (blk-u) wrote :

If the library Keystone is using doesn't support IPv6, then Keystone shouldn't be testing IPv6.

Brant Knudson (blk-u) wrote :

test_ipv6's test_ip_v6 is not skipped if there's no ipv6. It doesn't use the skip_if_no_ipv6 checker.

Brant Knudson (blk-u) wrote :

I tried setting a breakpoint in my .venv/lib/python2.7/site-packages/eventlet/support/greendns.py and it looks like its getaddrinfo isn't called when I run the test on my system.

Brant Knudson (blk-u) wrote :

Got some help from enikanorov-w on irc, and was able to recreate the problem, by doing ".venv/bin/pip install dnspython"

Then, when I run_tests.sh it fails just like Jenkins.

Brant Knudson (blk-u) wrote :

My plan is to change the code to skip these 3 tests, referencing this bug and add a TODO to check sometime if eventlet has been fixed. I'm not going to remove ipv6 support, assuming it works for someone, just not test it.

Changed in keystone:
assignee: nobody → Brant Knudson (blk-u)
no longer affects: openstack-ci

Fix proposed to branch: master
Review: https://review.openstack.org/28338

Changed in keystone:
status: New → In Progress
Brant Knudson (blk-u) on 2013-05-06
summary: - keystone ipv6 fails on oneiric3
+ keystone ipv6 tests fail
Brant Knudson (blk-u) wrote :
Changed in keystone:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/28338
Committed: http://github.com/openstack/keystone/commit/b55620d7ee0f87b0339285b16dd6fd68e5516b10
Submitter: Jenkins
Branch: master

commit b55620d7ee0f87b0339285b16dd6fd68e5516b10
Author: Brant Knudson <email address hidden>
Date: Mon May 6 13:42:26 2013 -0500

    Skip IPv6 tests for eventlet dns

    Eventlet address resolving doesn't support IPv6. This is causing the
    gate to fail.

    The failure happens when the dnspython package is installed in the
    environment. When dnspython is there then eventlet takes over
    getaddrinfo, and eventlet's getaddrinfo doesn't support IPv6. If
    dnspython isn't there, then the eventlet patching doesn't happen
    and IPv6 address resolution works.

    It appears that the pip dependencies have recently been updated
    so that dnspython gets installed now.

    The fix is to skip the 3 tests that force the server to listen
    on "::1".

    Fixes bug 1176204

    Once eventlet is updated, we can add the tests back in. Here's the
    eventlet code in question:

    https://bitbucket.org/eventlet/eventlet/src/e0f578180d7d82d2ed3d8a96d520103503c524ec/eventlet/support/greendns.py?at=0.12#cl-166

    Change-Id: Ib7a5a5f0dcac82229b7b7fb3df4b1ef141586c5f

Dolph Mathews (dolph) on 2013-05-07
tags: added: grizzly-backport-potential

Reviewed: https://review.openstack.org/28431
Committed: http://github.com/openstack/keystone/commit/b874c8f581fd5fb1e6657f9ec9348076e34a3849
Submitter: Jenkins
Branch: stable/grizzly

commit b874c8f581fd5fb1e6657f9ec9348076e34a3849
Author: Brant Knudson <email address hidden>
Date: Mon May 6 13:42:26 2013 -0500

    Skip IPv6 tests for eventlet dns

    Eventlet address resolving doesn't support IPv6. This is causing the
    gate to fail.

    The failure happens when the dnspython package is installed in the
    environment. When dnspython is there then eventlet takes over
    getaddrinfo, and eventlet's getaddrinfo doesn't support IPv6. If
    dnspython isn't there, then the eventlet patching doesn't happen
    and IPv6 address resolution works.

    It appears that the pip dependencies have recently been updated
    so that dnspython gets installed now.

    The fix is to skip the 3 tests that force the server to listen
    on "::1".

    Fixes bug 1176204

    Once eventlet is updated, we can add the tests back in. Here's the
    eventlet code in question:

    https://bitbucket.org/eventlet/eventlet/src/e0f578180d7d82d2ed3d8a96d520103503c524ec/eventlet/support/greendns.py?at=0.12#cl-166

    Change-Id: Ib7a5a5f0dcac82229b7b7fb3df4b1ef141586c5f

tags: removed: grizzly-backport-potential
Thierry Carrez (ttx) on 2013-05-29
Changed in keystone:
milestone: none → havana-1
status: Fix Committed → Fix Released
Dolph Mathews (dolph) on 2013-08-28
Changed in keystone:
importance: Undecided → Medium
Thierry Carrez (ttx) on 2013-10-17
Changed in keystone:
milestone: havana-1 → 2013.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers