neutron ml2 plugin test failures

[James Page]

./run_tests.sh -N neutron.tests.unit.ml2.test_agent_scheduler
Running ` python setup.py testr --slowest --testr-args='--subunit neutron.tests.unit.ml2.test_agent_scheduler'`
running testr
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ neutron/tests/unit --list
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ neutron/tests/unit --load-list /tmp/tmpYUaXvW
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ neutron/tests/unit --load-list /tmp/tmp1h3tWw
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ neutron/tests/unit --load-list /tmp/tmpsYVCGc
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ neutron/tests/unit --load-list /tmp/tmpZS9P2j

process-returncode
process-returncode ... FAIL
process-returncode
process-returncode ... FAIL

process-returncode
process-returncode ... FAIL

process-returncode
process-returncode ... FAIL
math domain error
Slowest Tests
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_disabled
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_disabled ... FAIL
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_scheduler_with_hosted_network
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_scheduler_with_hosted_network ... FAIL
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_hosted
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_hosted ... FAIL
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_add_to_dhcp_agent
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_add_to_dhcp_agent ... FAIL

======================================================================
FAIL: process-returncode
process-returncode
----------------------------------------------------------------------
testtools.testresult.real._StringException: Binary content:
  traceback (test/plain; charset="utf8")

======================================================================
FAIL: process-returncode
process-returncode
----------------------------------------------------------------------
testtools.testresult.real._StringException: Binary content:
  traceback (test/plain; charset="utf8")

======================================================================
FAIL: process-returncode
process-returncode
----------------------------------------------------------------------
testtools.testresult.real._StringException: Binary content:
  traceback (test/plain; charset="utf8")

======================================================================
FAIL: process-returncode
process-returncode
----------------------------------------------------------------------
testtools.testresult.real._StringException: Binary content:
  traceback (test/plain; charset="utf8")

======================================================================
FAIL: neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_disabled
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_disabled
----------------------------------------------------------------------
testtools.testresult.real._StringException

======================================================================
FAIL: neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_scheduler_with_hosted_network
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_scheduler_with_hosted_network
----------------------------------------------------------------------
testtools.testresult.real._StringException

======================================================================
FAIL: neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_hosted
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_hosted
----------------------------------------------------------------------
testtools.testresult.real._StringException

======================================================================
FAIL: neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_add_to_dhcp_agent
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_add_to_dhcp_agent
----------------------------------------------------------------------
testtools.testresult.real._StringException

----------------------------------------------------------------------
Ran 8 tests in 3.552s

FAILED (failures=8)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sukhdev Kapur (sukhdev-8)]

Numerous ml2 tests are failing which seem to be related to this. Following is the list of test that are failing, along with the traceback which is for each of the failed tests:

neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase
neutron.tests.unit.ml2.test_agent_scheduler.Ml2DhcpAgentNotifierTestCase
neutron.tests.unit.ml2.test_agent_scheduler.Ml2L3AgentNotifierTestCase
neutron.tests.unit.ml2.test_ml2_plugin.TestMl2BasicGet
neutron.tests.unit.ml2.test_ml2_plugin.TestMl2NetworksV2
neutron.tests.unit.ml2.test_ml2_plugin.TestMl2PortBindingHost
neutron.tests.unit.ml2.test_ml2_plugin.TestMl2PortsV2
neutron.tests.unit.ml2.test_ml2_plugin.TestMl2V2HTTPResponse
neutron.tests.unit.ml2.test_security_group.TestMl2SecurityGroups
neutron.tests.unit.ml2.test_security_group.TestMl2SecurityGroupsXML

All of these tests fail with the same traceback log. It seems like this is related to table "ml2_vlan_allocations"

Here is the traceback log:

Traceback (most recent call last):
  File "/home/jenkins/workspace/gate-neutron-python26/neutron/tests/unit/ml2/test_ml2_plugin.py", line 36, in setUp
    super(Ml2PluginV2TestCase, self).setUp(PLUGIN_NAME)
  File "/home/jenkins/workspace/gate-neutron-python26/neutron/tests/unit/test_db_plugin.py", line 122, in setUp
    self.api = APIRouter()
  File "/home/jenkins/workspace/gate-neutron-python26/neutron/api/v2/router.py", line 76, in __init__
    plugin = manager.NeutronManager.get_plugin()
  File "/home/jenkins/workspace/gate-neutron-python26/neutron/manager.py", line 206, in get_plugin
    return cls.get_instance().plugin
  File "/home/jenkins/workspace/gate-neutron-python26/neutron/manager.py", line 201, in get_instance
    cls._create_instance()
  File "/home/jenkins/workspace/gate-neutron-python26/neutron/openstack/common/lockutils.py", line 247, in inner
    retval = f(*args, **kwargs)
  File "/home/jenkins/workspace/gate-neutron-python26/neutron/manager.py", line 195, in _create_instance
    cls._instance = cls()
  File "/home/jenkins/workspace/gate-neutron-python26/neutron/manager.py", line 120, in __init__
    self.plugin = plugin_klass()
  File "/home/jenkins/workspace/gate-neutron-python26/neutron/plugins/ml2/plugin.py", line 93, in __init__
    self.type_manager.initialize()
  File "/home/jenkins/workspace/gate-neutron-python26/neutron/plugins/ml2/managers.py", line 79, in initialize
    driver.obj.initialize()
  File "/home/jenkins/workspace/gate-neutron-python26/neutron/plugins/ml2/drivers/type_vlan.py", line 160, in initialize
    self._sync_vlan_allocations()
  File "/home/jenkins/workspace/gate-neutron-python26/neutron/plugins/ml2/drivers/type_vlan.py", line 103, in _sync_vlan_allocations
    for alloc in allocs:
  File "/home/jenkins/workspace/gate-neutron-python26/.tox/py26/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
    return self._execute_and_instances(context)
  File "/home/jenkins/workspace/gate-neutron-python26/.tox/py26/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/jenkins/workspace/gate-neutron-python26/.tox/py26...

[Kyle Mestery (mestery)]

I'll take a look into this today.

[Kyle Mestery (mestery)]

I just ran the exact command above on my test rig [1], and it ran fine with no issues. Was this done using Ubuntu 12.04? I'll try to give that a shot next.

[1] Fedora 19

[Sukhdev Kapur (sukhdev-8)]

Hey Kyal,

In my case, I did not run the test command myself. It was run by Jenkins -
I do not know which platform is used.
I have submitted the patch little while ago and I will wait and watch to
let you know if this time it works or not.

On Wed, Aug 21, 2013 at 9:44 AM, Kyle Mestery <email address hidden>wrote:

> I just ran the exact command above on my test rig [1], and it ran fine
> with no issues. Was this done using Ubuntu 12.04? I'll try to give that
> a shot next.
>
> [1] Fedora 19
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1204456
>
> Title:
> neutron ml2 plugin test failures
>
> Status in “neutron” package in Ubuntu:
> Confirmed
>
> Bug description:
> ./run_tests.sh -N neutron.tests.unit.ml2.test_agent_scheduler
> Running ` python setup.py testr --slowest --testr-args='--subunit
> neutron.tests.unit.ml2.test_agent_scheduler'`
> running testr
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 ${PYTHON:-python} -m
> subunit.run discover -t ./ neutron/tests/unit --list
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 ${PYTHON:-python} -m
> subunit.run discover -t ./ neutron/tests/unit --load-list /tmp/tmpYUaXvW
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 ${PYTHON:-python} -m
> subunit.run discover -t ./ neutron/tests/unit --load-list /tmp/tmp1h3tWw
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 ${PYTHON:-python} -m
> subunit.run discover -t ./ neutron/tests/unit --load-list /tmp/tmpsYVCGc
> running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 ${PYTHON:-python} -m
> subunit.run discover -t ./ neutron/tests/unit --load-list /tmp/tmpZS9P2j
>
> process-returncode
> process-returncode ... FAIL
> process-returncode
> process-returncode ... FAIL
>
> process-returncode
> process-returncode ... FAIL
>
> process-returncode
> process-returncode ... FAIL
> math domain error
> Slowest Tests
>
> neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_disabled
>
> neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_disabled
> ... FAIL
>
> neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_scheduler_with_hosted_network
>
> neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_scheduler_with_hosted_network
> ... FAIL
>
> neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_hosted
>
> neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_hosted
> ... FAIL
>
> neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_add_to_dhcp_agent
>
> neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_add_to_dhcp_agent
> ... FAIL
>
> ======================================================================
> FAIL: process-returncode
> process-returncode
> ----------------------------------------------------------------------
> testtools.testresult.real._StringException: Binary content:
> traceback (test/plain; ch...

[Xiang Hui (xianghui)]

Hi,

When I ran run_tests.sh on ubuntu, there are always fails as below:
Seems it looks like the failure above.
Is there any ways to fix this? because it has blocked my commit.

Thanks in advance.

root@ubuntu:/opt/stack/neutron# ./run_tests.sh

======================================================================
FAIL: process-returncode
process-returncode
----------------------------------------------------------------------
_StringException: Binary content:
  traceback (test/plain; charset="utf8")

======================================================================
FAIL: process-returncode
process-returncode
----------------------------------------------------------------------
_StringException: Binary content:
  traceback (test/plain; charset="utf8")

======================================================================
FAIL: neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_add_to_dhcp_agent
neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_add_to_dhcp_agent
----------------------------------------------------------------------
_StringException

======================================================================
FAIL: neutron.tests.unit.ml2.test_security_group.TestMl2SecurityGroups.test_create_security_group_rule_tcp_protocol_as_number
neutron.tests.unit.ml2.test_security_group.TestMl2SecurityGroups.test_create_security_group_rule_tcp_protocol_as_number
----------------------------------------------------------------------
_StringException

----------------------------------------------------------------------

[Bo Lin (linb)]

It is the same in my Ubuntu-12.04 Server

[Justin Hammond (justin-hammond)]

Has there been any progress or updates on this? I req

[Justin Hammond (justin-hammond)]

oops.

I require a work around if possible?

[Bo Lin (linb)]

It is tricky in my venv environment. take "neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_disabled" as an example, using $./run_tests.sh neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_disabled (testr) would run okay, while using $./run_tests.sh -d neutron.tests.unit.ml2.test_agent_scheduler.Ml2AgentSchedulerTestCase.test_network_auto_schedule_with_disabled (testtools.run) would report following error:
 File "/home/stack/neutron/.venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (OperationalError) no such table: ml2_vlan_allocations u'SELECT ml2_vlan_allocations.physical_network AS ml2_vlan_allocations_physical_network, ml2_vlan_allocations.vlan_id AS ml2_vlan_allocations_vlan_id, ml2_vlan_allocations.allocated AS ml2_vlan_allocations_allocated \nFROM ml2_vlan_allocations' ()

[stanzgy (stanzgy)]

I experienced this failure these days and my unittest always failed at case
"neutron.tests.unit.ml2.drivers.cisco.nexus.test_cisco_mech.TestCiscoBasicGet.test_single_get_admin".
After debugging, I found this bug is caused by ml2 type_drivers entry points not found.

way to reproduce:
1. cd neutron
2. rm -rf neutron.egg-info
3. ./run_tests.sh neutron.tests.unit.ml2.drivers.cisco.nexus.test_cisco_mech.TestCiscoBasicGet.test_single_get_admin

...
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit} --list
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit} --load-list /tmp/tmp2MIKLY

process-returncode
process-returncode ... FAIL
math domain error
Slowest Tests
neutron.tests.unit.ml2.drivers.cisco.nexus.test_cisco_mech.TestCiscoBasicGet.test_single_get_admin
neutron.tests.unit.ml2.drivers.cisco.nexus.test_cisco_mech.TestCiscoBasicGet.test_single_get_admin ... FAIL

======================================================================
FAIL: process-returncode
process-returncode
----------------------------------------------------------------------
_StringException: Binary content:
  traceback (test/plain; charset="utf8")

======================================================================
FAIL: neutron.tests.unit.ml2.drivers.cisco.nexus.test_cisco_mech.TestCiscoBasicGet.test_single_get_admin
neutron.tests.unit.ml2.drivers.cisco.nexus.test_cisco_mech.TestCiscoBasicGet.test_single_get_admin
----------------------------------------------------------------------
_StringException

----------------------------------------------------------------------
Ran 2 tests in 25.174s

FAILED (failures=2)

way to fix:
1. python setup.py egg_info
2. ./run_tests.sh neutron.tests.unit.ml2.drivers.cisco.nexus.test_cisco_mech.TestCiscoBasicGet.test_single_get_admin

...
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit} --list
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./neutron/tests/unit} --load-list /tmp/tmp1bq15x
neutron.tests.unit.ml2.drivers.cisco.nexus.test_cisco_mech.TestCiscoBasicGet.test_single_get_admin
neutron.tests.unit.ml2.drivers.cisco.nexus.test_cisco_mech.TestCiscoBasicGet.test_single_get_admin ... ok
Slowest Tests
Test id Runtime (s)
-------------------------------------------------------------------------------------------------- -----------
neutron.tests.unit.ml2.drivers.cisco.nexus.test_cisco_mech.TestCiscoBasicGet.test_single_get_admin 1.362

----------------------------------------------------------------------
Ran 1 test in 25.259s

OK

The reason is that when testcase setup Ml2Plugin, Ml2Plugin will initialize TypeManager()(https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/plugin.py#L105). Then TypeManager() will loaded type drivers using stevedor...

[Henry Gessau (gessau)]

Works without the -N option:
$ ./run_tests.sh neutron.tests.unit.ml2.test_agent_scheduler

Also works with tox (version 1.6.1):
$ tox -e py27 -- neutron.tests.unit.ml2.test_agent_scheduler

The above two methods use a virtualenv to run the tests in. The -N option to run_tests.sh tells it to not use a virtualenv and run the unit tests in the system environment. If the system environment does not satisfy the test-requirements correctly or has some settings that interfere, then the tests may not work.

The recommended way of running unit tests is in a virtualenv. If someone wants to run tests in their system environment it is up to them to adjust their environment appropriately.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Salvatore Orlando (<email address hidden>) on branch: master
Review: https://review.openstack.org/100186
Reason: This patch has been inactive long enough that I think it's safe to abandon.
The author can resurrect it if needed.