neutron-2013.2 testsuite issues

[IAN DELANEY]

See https://bugs.launchpad.net/python-swiftclient/+bug/1251507

Now observe.

testuser@archtester /mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2 $ ./run_tests.shNo virtual environment found...create one? (Y/n) n
Running ` python setup.py testr --slowest --testr-args='--subunit '`
Searching for python-subunit>=0.0.10
Reading https://pypi.python.org/simple/python-subunit/
Reading http://launchpad.net/subunit
Best match: python-subunit 0.0.15
Downloading https://pypi.python.org/packages/source/p/python-subunit/python-subunit-0.0.15.tar.gz#md5=65e43979f1156911a468fa76cc192c0a
Traceback (most recent call last):
  File "setup.py", line 22, in <module>
    pbr=True)
  File "/usr/lib64/python2.7/distutils/core.py", line 138, in setup
    ok = dist.parse_command_line()
  File "/usr/lib64/python2.7/site-packages/setuptools/dist.py", line 276, in parse_command_line
    result = _Distribution.parse_command_line(self)
  File "/usr/lib64/python2.7/distutils/dist.py", line 467, in parse_command_line
    args = self._parse_command_opts(parser, args)
  File "/usr/lib64/python2.7/site-packages/setuptools/dist.py", line 602, in _parse_command_opts
    nargs = _Distribution._parse_command_opts(self, parser, args)
  File "/usr/lib64/python2.7/distutils/dist.py", line 523, in _parse_command_opts
    cmd_class = self.get_command_class(command)
  File "/usr/lib64/python2.7/site-packages/setuptools/dist.py", line 406, in get_command_class
    ep.require(installer=self.fetch_build_egg)
  File "/usr/lib64/python2.7/site-packages/pkg_resources.py", line 2272, in require
    working_set.resolve(self.dist.requires(self.extras),env,installer)))
  File "/usr/lib64/python2.7/site-packages/pkg_resources.py", line 618, in resolve
    dist = best[req.key] = env.best_match(req, self, installer)
  File "/usr/lib64/python2.7/site-packages/pkg_resources.py", line 862, in best_match
    return self.obtain(req, installer) # try and download/install
  File "/usr/lib64/python2.7/site-packages/pkg_resources.py", line 874, in obtain
    return installer(requirement)
  File "/usr/lib64/python2.7/site-packages/setuptools/dist.py", line 339, in fetch_build_egg
    return cmd.easy_install(req)
  File "/usr/lib64/python2.7/site-packages/setuptools/command/easy_install.py", line 611, in easy_install
    self.local_index
  File "/usr/lib64/python2.7/site-packages/setuptools/package_index.py", line 539, in fetch_distribution
    return dist.clone(location=self.download(dist.location, tmpdir))
  File "/usr/lib64/python2.7/site-packages/setuptools/package_index.py", line 453, in download
    found = self._download_url(scheme.group(1), spec, tmpdir)
  File "/usr/lib64/python2.7/site-packages/setuptools/package_index.py", line 705, in _download_url
    return self._attempt_download(url, filename)
  File "/usr/lib64/python2.7/site-packages/setuptools/package_index.py", line 712, in _attempt_download
    headers = self._download_to(url, filename)
  File "/usr/lib64/python2.7/site-packages/setuptools/package_index.py", line 608, in _download_to
    "Can't download %s: %s %s" % (url, fp.code,fp.msg)
distutils.errors.DistutilsError: Can't download https://pypi.python.org/packages/source/p/python-subunit/python-subunit-0.0.15.tar.gz: 502 Bad Gateway

----------------------------------------------------------------------
Ran 0 tests in 5.706s

OK
cat: .testrepository/next-stream: No such file or directory

So what was that about python-subunit vs. gentoo's naming of just subunit.
This is what happens when relying on creating a VIRTUAL env.
The system has all these installed already, so why not USE THEM??

cat: .testrepository/next-stream: No such file or directory

indeed.

neutron-2013.2 $ ls -ld .testrepository
ls: cannot access .testrepository: No such file or directory

So, running nosetests -I __init__.py -I test_wsgi.py -I test_ovs_tunnel.py \
yields
FAILED (SKIP=328, errors=14, failures=4)

there are 9 of

AttributeError: 'AgentMechanismFlatTestCase' object has no attribute 'driver'

ERROR: neutron.tests.unit.ml2._test_mech_agent.AgentMechanismFlatTestCase.test_type_flat_bad
ERROR: neutron.tests.unit.ml2._test_mech_agent.AgentMechanismFlatTestCase.test_type_flat
ERROR: neutron.tests.unit.ml2._test_mech_agent.AgentMechanismGenericTestCase.test_unknown_type
ERROR: neutron.tests.unit.ml2._test_mech_agent.AgentMechanismGreTestCase.test_type_gre
ERROR: neutron.tests.unit.ml2._test_mech_agent.AgentMechanismGreTestCase.test_type_gre_bad
ERROR: neutron.tests.unit.ml2._test_mech_agent.AgentMechanismLocalTestCase.test_type_local
ERROR: neutron.tests.unit.ml2._test_mech_agent.AgentMechanismLocalTestCase.test_type_local
ERROR: neutron.tests.unit.ml2._test_mech_agent.AgentMechanismLocalTestCase.test_type_local_dead
ERROR: neutron.tests.unit.ml2._test_mech_agent.AgentMechanismVlanTestCase.test_type_vlan

---------------------------------------------
a few of raise TypeError(repr(o) + " is not JSON serializable");

ERROR: neutron.tests.unit._test_extension_portbindings.PortBindingsTestCase.test_create_port_binding_profile_none
----------------------------------------------------------------------
_StringException: pythonlogging:'': {{{
2013-11-15 21:19:37,486 INFO [neutron.manager] Loading Plugin: neutron.db.db_base_plugin_v2.NeutronDbPluginV2
2013-11-15 21:19:37,491 ERROR [neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent] Failed reporting state!
Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/plugins/linuxbridge/agent/linuxbridge_neutron$
    self.agent_state)
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/agent/rpc.py", line 74, in report_state
    return self.cast(context, msg, topic=self.topic)
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/openstack/common/rpc/proxy.py", line 171, in $
    rpc.cast(context, self._get_topic(topic), msg)
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/openstack/common/rpc/__init__.py", line 158, $
    return _get_impl().cast(CONF, context, topic, msg)
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/openstack/common/rpc/impl_fake.py", line 166,$
    check_serialize(msg)
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/openstack/common/rpc/impl_fake.py", line 131,$
    json.dumps(msg)
  File "/usr/lib64/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib64/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")

------------------------------------------------
some of HTTPBadRequest: Unrecognized attribute(s) 'binding:profile'

2013-11-15 21:19:37,730 ERROR [neutron.api.v2.resource] create failed
Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/api/v2/resource.py", line 84, in resource
    result = method(request=request, **args)
 .........................................
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/api/v2/base.py", line 602, in _verify_attributes
    raise webob.exc.HTTPBadRequest(msg)
HTTPBadRequest: Unrecognized attribute(s) 'binding:profile'
}}}

Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/tests/unit/_test_extension_portbindings.py",
  line 100, in test_create_port_binding_profile_none
    self._test_create_port_binding_profile(None)
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/tests/unit/_test_extension_portbindings.py",
  line 96, in _test_create_port_binding_profile
    **profile_arg) as port:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/tests/unit/test_db_plugin.py", line 573, in port
    port = self._make_port(fmt or self.fmt, net_id, **kwargs)
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/tests/unit/test_db_plugin.py", line 436, in _make_port
    raise webob.exc.HTTPClientError(code=res.status_int)
HTTPClientError: The server could not comply with the request since it is either malformed or otherwise incorrect.

Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py",
  line 798, in _report_state
    self.agent_state)
.................
  File "/usr/lib64/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")

possibly a wrong version here

[system installed;
dev-python/simplejson-3.3.0
dev-python/jsonschema-0.8.0]

ditto

neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent: ERROR: Failed reporting state!
Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 798, in _report_state
    self.agent_state)

-----------------------------------------
a few of KeyError: 'binding:vif_type'

Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/tests/unit/_test_extension_portbindings.py", line 61, in test_port_vif_details
    self._check_response_portbindings(port['port'])
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/tests/unit/_test_extension_portbindings.py", line 41, in _check_response_portbindings
    self.assertEqual(port['binding:vif_type'], self.VIF_TYPE)
KeyError: 'binding:vif_type'

ERROR: Failure: TypeError (expected string or buffer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib64/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib64/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/quantum/api/__init__.py", line 29, in <module>
    _('You are using old configuration values for the api-paste config. '
TypeError: expected string or buffer

seems it picked on quantum/api/__init__.py despite attempting to Ignore it with -I. Likely got pointed to it independently. nosetests has a habit of trawling __init__.py files, just needs setting the right set of test_ path(s) to avoid it as far as I recall.

----------------------------------------------------------
a few of

Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/tests/unit/_test_extension_portbindings.py", line 136, in test_port_create_portinfo_no$
    **profile_arg):
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/tests/unit/test_db_plugin.py", line 573, in port
    port = self._make_port(fmt or self.fmt, net_id, **kwargs)
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/tests/unit/test_db_plugin.py", line 432, in _make_port
    res = self._create_port(fmt, net_id, expected_res_status, **kwargs)
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/neutron/tests/unit/test_db_plugin.py", line 373, in _create_port
    self.assertEqual(port_res.status_int, expected_res_status)
  File "/usr/lib64/python2.7/site-packages/testtools/testcase.py", line 322, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/usr/lib64/python2.7/site-packages/testtools/testcase.py", line 417, in assertThat
    raise MismatchError(matchee, matcher, mismatch, verbose)
MismatchError: 400 != 403

if you wish for the whole log it's 351.124 kbytes in size

Now your test uses nosetests anyways however it wants the subunit thingy which I don't know how to implement 'properly' or fully, so running nosetests is cutting pretty close to the bone.

I shall attempt to exclude these failing tests in the short term. 18 out of 9283 isn't too bad

[IAN DELANEY (johneed)]

ok I am up to

ERROR: Failure: TypeError (expected string or buffer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib64/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib64/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/quantum/api/__init__.py", line 29, in <module>
    _('You are using old configuration values for the api-paste config. '
TypeError: expected string or buffer

----------------------------------------------------------------------
Ran 9266 tests in 7273.004s

FAILED (SKIP=328, errors=1)

by use of

nosetests -I __init__.py -I test_wsgi.py -I test_ovs_tunnel.py -I _test_extension_portbindings.py -I _test_mech_agent.py \

on running a separate

nosetests -I __init__.py -I test_wsgi.py -I test_ovs_tunnel.py

I got

Ran 9283 tests in 6688.621s

FAILED (SKIP=328, errors=21, failures=4)

however seeing the tests by their nature use tcp and ports, them running concurrently likely saw the error rate raise to 21 by virtue of competing for port address or some network address of some sort, so not too concerned ocer that for now. However seeing the suite takes ~ 6688 secs to run. concurrent sessions are MOST tempting.

Here's a concern.

...........................Password:
Password:
Password:
................................................................................................................................

Part of that extended time to run is awaiting this test asking for a password to timeout.
again

awaiting this TEST ASKING FOR A PASSWORD.

A test in a testsuite asking for a password. Hmmm. Well are they not supposed to run UNATTENDED?
At minimum please tell me which test it is so I can nosetest --exclude it.

Re the trawling of the __init__.py file under the antiquated quantum folder, please also suggest how I can get nosetets to skip it.

sys-cluster/neutron $ ls neutron-2013.2/quantum/
api auth.py __init__.py

There are no test_ files under this folder so some test MUST be pointing to it. Normally stipulating /folder/test_.py suffices to skirt an __init__.py file but it appears to me that 1 of the over 9,000 test_x.py files is sending it to the quantum folder. Sheeesh. Alternately you can fix the test so it actually goes looking to use a string or buffer.

[Eugene Nikanorov (enikanorov)]

Apparently it's an issue with setting up venv not directly related to neutron project

[Matthew Thode (prometheanfire)]

Could you let us know what exactly we are missing? We have testing support in our package manager and see to leverage it.

[IAN DELANEY (johneed)]

well, past the above now.
On applying the patch from;

neutron/+bug/1234857

python_test() {
            testr init
            testr run || die "failed testsuite under python2.7"

now yields

Ran 12259 (+12259) tests in 2746.766s \ FAILED (id=5, failures=2 (+2), skips=171)

running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ neutron/tests/unit
======================================================================
FAIL: process-returncode
tags: worker-0
----------------------------------------------------------------------
Binary content:
  traceback (test/plain; charset="utf8")
Ran 12233 tests in 2123.316s
FAILED (id=0, failures=2, skips=171)

says no coverage output with a totally absent backtrace, meaning it may come from nicira plugin unit tests from one of the many runs that actually cited edge Edge edge-1 . Don't know what edge hardware is, the output hinted it's some virtual kind of hardware, so don't know if it's looking for it when it ought be there OR if it's looking for it when it erroneously, or ... Without a backtrace leaves you poking around in the dark.

The use of the --parallel option persistently invokes errors resulting fron the system mem. resources becoming over-taxed. Short of a running them on a fully blown server, don't know what it takes, but ignoring the --parallel option is merely a minor shame.

You MIGHT consider re-oening, but then again ..

[Matthew Thode (prometheanfire)]

On 11/18/2013 04:14 AM, IAN DELANEY wrote:
> well, past the above now.
> On applying the patch from;
>
> neutron/+bug/1234857
>
> python_test() {
> testr init
> testr run || die "failed testsuite under python2.7"
>
> now yields
>
> Ran 12259 (+12259) tests in 2746.766s \ FAILED (id=5, failures=2 (+2),
> skips=171)
>
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ neutron/tests/unit
> ======================================================================
> FAIL: process-returncode
> tags: worker-0
> ----------------------------------------------------------------------
> Binary content:
> traceback (test/plain; charset="utf8")
> Ran 12233 tests in 2123.316s
> FAILED (id=0, failures=2, skips=171)
>
> says no coverage output with a totally absent backtrace, meaning it may
> come from nicira plugin unit tests from one of the many runs that
> actually cited edge Edge edge-1 . Don't know what edge hardware is, the
> output hinted it's some virtual kind of hardware, so don't know if it's
> looking for it when it ought be there OR if it's looking for it when it
> erroneously, or ... Without a backtrace leaves you poking around in the
> dark.
>
> The use of the --parallel option persistently invokes errors resulting
> fron the system mem. resources becoming over-taxed. Short of a running
> them on a fully blown server, don't know what it takes, but ignoring the
> --parallel option is merely a minor shame.
>
> You MIGHT consider re-oening, but then again ..
>
We shouldn't need to concern ourselves with nicera atm. Disabling it
may help. Not having the backtrace sucks though.

--
-- Matthew Thode (prometheanfire)

[IAN DELANEY (johneed)]

without the backtrace I don't know where to disable it that's the WHOLE point

[IAN DELANEY (johneed)]

neutron/tests/unit/nicira/vshield tests appear to be run out of order, 'pinned' in some discussion by dev in irc channel

[IAN DELANEY (johneed)]

so from

python_test() {
        # https://bugs.launchpad.net/nova/+bug/1235955
        local DISTUTILS_NO_PARALLEL_BUILD=1
        mv $(find . -name test_ovs_tunnel.py) ./troubled || die
        sed -e 's:test_router_schedule_with_candidates:_&:' -i neutron/tests/unit/openvswitch/test_agent_scheduler.py
        testr init
        testr run --parallel || die "failed testsuite under python2.7"
}

we have

======================================================================
FAIL: process-returncode
tags: worker-0
----------------------------------------------------------------------
Binary content:
  traceback (test/plain; charset="utf8")
======================================================================
FAIL: process-returncode
tags: worker-1
----------------------------------------------------------------------
Binary content:
  traceback (test/plain; charset="utf8")
Ran 18665 tests in 1689.939s
FAILED (id=0, failures=2, skips=328)
 * ERROR: sys-cluster/neutron-2013.2-r1::gentoo failed (test phase):
 * failed testsuite under python2.7

as always with NO traceback, setting a rather nouvelle mystery.

from http://paste.openstack.org/show/53681/ from .testrepository/0
we have

pythonlogging:''

ok python logging gets turned on

2013-11-20 22:49:57,995 INFO [neutron.manager] Loading Plugin: neutron.plugins.bigswitch.plugin.NeutronRestProxyV2

the traceback begins at neutron.plugins.bigswitch.plugin,
who knows what the instigating test was. Here's a hint;

successful: neutron.tests.unit.bigswitch.test_restproxy_plugin.TestBigSwitchProxyNetworksV2.

just occurred before, so it appears it's likely from neutron.tests.unit.bigswitch.test_restproxy_plugin.py

2013-11-20 22:49:58,108 ERROR [neutron.api.extensions] Extension path 'unit/extensions' doesn't exist!

Oh dear, that's problematic.

$ find ...../neutron-2013.2/ -name extensions
  .....neutron-2013.2/neutron/tests/unit/extensions

perhaps?

should it not be seeking Extension path 'neutron/tests/unit/extensions' ??????????

testuser@archtester ~/cvsPortage/gentoo-x86/sys-cluster/neutron $ grep _emulate_bulk_create /mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/.testrepository/0
yields

  File "neutron/api/v2/base.py", line 329, in _emulate_bulk_create

no less than 36 times. That's problematic. Now note;

testuser@archtester ~/cvsPortage/gentoo-x86/sys-cluster/neutron $ ls -ld /path/to/neutron-2013.2/.testrepository/0
yields
-rw------- 1 testuser testuser 78318772 Nov 20 23:18 /mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2-r1/work/neutron-2013.2/.testrepository/0
Is this 78 mb of log that backtraces to neutron system packages, never reaches the actual instigating tests and lists 2 FAILS despite listing _emulate_bulk_create on 36 separate occasions. In the filed of analysing testsuites, this one really breaks new ground.
The good news is 2 out of 18665 tests in 1689.939s is not so bad
The bad news is for all the 78 mb of logging we have no idea WHAT THE TESTS ARE, in addition to this test still having an invalid invalid status

[IAN DELANEY (johneed)]

Oh my;

# https://bugs.launchpad.net/swift/+bug/1249727
was the point all the time it seems.

/path/to/neutron-2013.2.9999/lib/ nosetests neutron/tests/unit/test_wsgi.py $ nosetests PYTHONPATH=../neutron-2013.2.9999-python2_7/lib/ nosetests neutron/tests/unit/test_wsgi.py

yields

.............................................................E.E.................
======================================================================
ERROR: neutron.tests.unit.test_wsgi.TestWSGIServer.test_start_random_port_with_ipv6
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nose/case.py", line 133, in run
    self.runTest(result)
  File "/usr/lib64/python2.7/site-packages/nose/case.py", line 151, in runTest
    test(result)
  File "/usr/lib64/python2.7/unittest/case.py", line 395, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib64/python2.7/site-packages/testtools/testcase.py", line 526, in run
    return self.__RunTest(self, self.exception_handlers).run(result)
  File "/usr/lib64/python2.7/site-packages/testtools/runtest.py", line 74, in run
    return self._run_one(actual_result)
  File "/usr/lib64/python2.7/site-packages/testtools/runtest.py", line 88, in _run_one
    return self._run_prepared_result(ExtendedToOriginalDecorator(result))
  File "/usr/lib64/python2.7/site-packages/testtools/runtest.py", line 100, in _run_prepared_result
    self._run_core()
  File "/usr/lib64/python2.7/site-packages/testtools/runtest.py", line 125, in _run_core
    self.case._run_test_method, self.result):
  File "/usr/lib64/python2.7/site-packages/testtools/runtest.py", line 173, in _run_user
    return self._got_user_exception(sys.exc_info())
  File "/usr/lib64/python2.7/site-packages/testtools/runtest.py", line 200, in _got_user_exception
    raise e
SystemExit: 1
-------------------- >> begin captured logging << --------------------
neutron.wsgi: ERROR: Unable to listen on ::1:0
Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/sys-cluster/neutron-2013.2.9999/work/neutron-2013.2.9999/neutron/wsgi.py", line 103, in _get_socket
    socket.SOCK_STREAM)[0]
  File "/usr/lib64/python2.7/site-packages/eventlet/support/greendns.py", line 174, in getaddrinfo
    rrset = resolve(host)
  File "/usr/lib64/python2.7/site-packages/eventlet/support/greendns.py", line 133, in resolve
    raise socket.gaierror(error)
gaierror: (-5, 'No address associated with hostname')
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: neutron.tests.unit.test_wsgi.TestWSGIServerWithSSL.test_app_using_ipv6_and_ssl
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nose/case.py", line 133, in run
    self.runTest(result)
  File "/usr/lib64/python2.7/site-packages/nose/case.py", line 151, in runTest
    test(result)
  File "/usr/lib64/python2.7/unittest/case.py", line 395, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib64/python2.7/site-packages/testtools/testc...

[IAN DELANEY (johneed)]

and the _test once revereted to the proper test_ in neutron/tests/unit/test_wsgi.py

/path/to/neutron-2013.2 $ testr run neutron/tests/unit/test_wsgi.py

yields

neutron-2013.2 $ testr run neutron/tests/unit/test_wsgi.py
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ neutron/tests/unit/ --list
/usr/lib64/python2.7/site-packages/sqlalchemy/ext/declarative.py:1341: SAWarning: The classname 'NetworkBinding' is already in the registry of this declarative base, mapped to <class 'neutron.plugins.linuxbridge.db.l2network_models_v2.NetworkBinding'>
  _as_declarative(cls, classname, cls.__dict__)
/usr/lib64/python2.7/site-packages/sqlalchemy/ext/declarative.py:1341: SAWarning: The classname 'NetworkBinding' is already in the registry of this declarative base, mapped to <class 'neutron.plugins.openvswitch.ovs_models_v2.NetworkBinding'>
  _as_declarative(cls, classname, cls.__dict__)
/usr/lib64/python2.7/site-packages/sqlalchemy/ext/declarative.py:1341: SAWarning: The classname 'Credential' is already in the registry of this declarative base, mapped to <class 'neutron.plugins.cisco.db.network_models_v2.Credential'>
  _as_declarative(cls, classname, cls.__dict__)
/usr/lib64/python2.7/site-packages/sqlalchemy/ext/declarative.py:1341: SAWarning: The classname 'NexusPortBinding' is already in the registry of this declarative base, mapped to <class 'neutron.plugins.cisco.db.nexus_models_v2.NexusPortBinding'>
  _as_declarative(cls, classname, cls.__dict__)
/usr/lib64/python2.7/site-packages/sqlalchemy/ext/declarative.py:1341: SAWarning: The classname 'VlanAllocation' is already in the registry of this declarative base, mapped to <class 'neutron.plugins.openvswitch.ovs_models_v2.VlanAllocation'>
  _as_declarative(cls, classname, cls.__dict__)
PASSED (id=3)

I do hope $ testr run neutron/tests/unit/test_wsgi.py is correct as in a legitimate run. It seems to throw up PASSED (id=x) as long as it doesn't fail rather than as long as it actually ran, otherwise,

suggesting it always was a race condition

[IAN DELANEY (johneed)]

/path/to/neutron-2013.2.9999 $ ls -l .testrepository/

total 86232
-rw------- 1 testuser testuser 86859472 Nov 22 10:52 0
-rw------- 1 testuser testuser 0 Nov 22 11:04 1
-rw------- 1 testuser testuser 236 Nov 22 11:04 failing
-rw-r--r-- 1 testuser testuser 2 Nov 22 10:27 format
-rw-r--r-- 1 testuser testuser 2 Nov 22 11:04 next-stream
-rw-r--r-- 1 testuser testuser 1429504 Nov 22 11:04 times.dbm

so there is a failing with content. Can't decipher what any of it means, mind you, but I don't see any citing of the test_ s that failed, which would be, well, nice @ minimum.

/path/to/neutron-2013.2.9999 $ testr run --failing

yields

........................................................................
PASSED (id=2)

so this supports but falls short of proving the race condition. Go the winner (of the race). Everyone loves a winner.

[Matthew Thode (prometheanfire)]

Can we get this reopened in refrence to the specific failing test suites. I suspect they should not fail...