unit test neutron.tests.unit.test_dhcp_scheduler.TestNetworksFailover fails

[Miguel Angel Ajo]

Unit tests failing non deterministically:

neutron.tests.unit.test_dhcp_scheduler.TestNetworksFailover.test_reschedule_network_from_down_agent_StringException: Empty attachments:
  pythonlogging:'neutron.api.extensions'
  stderr
  stdout

pythonlogging:'': {{{
2015-03-17 04:56:58,726 WARNING [neutron.db.agentschedulers_db] Removing network foo_network_id from agent 699132a9-7e81-4dee-b1c3-d1056d7e1598 because the agent did not report to the server in the last 150 seconds.
2015-03-17 04:56:58,801 INFO [neutron.db.agentschedulers_db] Scheduling unhosted network foo_network_id
2015-03-17 04:56:58,825 ERROR [neutron.plugins.ml2.managers] Network foo_network_id has no segments
2015-03-17 04:56:58,896 WARNING [neutron.scheduler.dhcp_agent_scheduler] No more DHCP agents
}}}

Traceback (most recent call last):
  File "neutron/tests/unit/test_dhcp_scheduler.py", line 216, in test_reschedule_network_from_down_agent
    notify=False)
  File "/home/jenkins/workspace/gate-neutron-python27/.tox/py27/local/lib/python2.7/site-packages/mock.py", line 831, in assert_called_with
    raise AssertionError('Expected call: %s\nNot called' % (expected,))
AssertionError: Expected call: remove_network_from_dhcp_agent(<ANY>, '699132a9-7e81-4dee-b1c3-d1056d7e1598', 'foo_network_id', notify=False)
Not called

[Armando Migliaccio (armando-migliaccio)]

Is this happening in the gate? I couldn't find any entry.

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQXNzZXJ0aW9uRXJyb3I6IEV4cGVjdGVkIGNhbGw6IHJlbW92ZV9uZXR3b3JrX2Zyb21fZGhjcF9hZ2VudFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI2NTc2MjQxNjkyfQ==

[Armando Migliaccio (armando-migliaccio)]

Sorry, I spoke too soon :(

[Armando Migliaccio (armando-migliaccio)]

The only change I saw this happen was https://review.openstack.org/#/c/155436/, and it's not merged yet.

[Abhishek Raut (abhraut)]

got hit by this bug -> https://review.openstack.org/#/c/157243/

[Armando Migliaccio (armando-migliaccio)]

Yes it's getting triggered:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwidGVzdF9yZXNjaGVkdWxlX25ldHdvcmtfZnJvbV9kb3duX2FnZW50XCIgQU5EIGJ1aWxkX25hbWU6XCJnYXRlLW5ldXRyb24tcHl0aG9uMjdcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyNzIyODAxMTk4MiwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

[Armando Migliaccio (armando-migliaccio)]

It looks like from [1], that the periodic task that does the removal of the network from the agent gets in the way of the test. IMO the periodic task should be mocked so that it does not interfere with the test.

[1] http://logs.openstack.org/08/160608/3/check/gate-neutron-python27/61c8ee1/console.html#_2015-03-24_19_42_57_107

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/167414

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/167414
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c0289458fa9055ee488989828951d15fa2a06da3
Submitter: Jenkins
Branch: master

commit c0289458fa9055ee488989828951d15fa2a06da3
Author: armando-migliaccio <email address hidden>
Date: Tue Mar 24 15:09:35 2015 -0700

    Fix intermittent failure in TestNetworksFailover UT

    Ensure that the periodic check does not get in the way of method
    calls being tested, by stopping the periodic task from running.

    This patch moves the mock for the periodic check into the
    setup_coreplugin call so it gets called by unit tests that
    use the core plugin.

    The previous location after the construction of the API router was
    too late because the core plugin was already constructed by the
    neutron manager. This led to random failures because the periodic
    tasks leaked by all of the unit tests would occasionally preempt
    test_reschedule_network_from_down_agent in automatically removing
    a network from an agent.

    Co-Author: Jenkins <email address hidden>

    Change-Id: I60ad7fa8ca874f93b7f806a0e035be84180a5de9
    Closes-bug: #1432958