Unittest fails due to unexpected ovs-vsctl calling in ryu plugin test

[Akihiro Motoki]

In the unit test, ovs-vsctl command is unexpectedly called in ryu plugin test.

It occurs in the latest master branch (4b47717b132336396cdbea9d168acaaa30bd5a02).
In gating test, the followings hit this issue:
http://logs.openstack.org/70/58270/2/check/gate-neutron-python27/79ef6dd/console.html
http://logs.openstack.org/25/58125/4/check/gate-neutron-python27/2cc0dc5/console.html#_2013-11-27_01_46_09_003

According to the result of debugging by adding print_traceback in ovs_lib.OVSBridge.get_vif_port_by_id,
the following tests fails and the following stack trace is got:

neutron.tests.unit.ryu.test_ryu_plugin.TestRyuPortsV2.test_update_port_update_ip_address_only
neutron.tests.unit.ryu.test_ryu_plugin.TestRyuPortsV2.test_update_port
neutron.tests.unit.ryu.test_ryu_plugin.TestRyuPortsV2.test_update_port_update_ips
neutron.tests.unit.ryu.test_ryu_plugin.TestRyuPortsV2.test_update_port_delete_ip
neutron.tests.unit.ryu.test_ryu_plugin.TestRyuPortsV2.test_update_port_add_additional_ip
neutron.tests.unit.ryu.test_ryu_plugin.TestRyuPortsV2.test_update_port_not_admin
neutron.tests.unit.ryu.test_ryu_plugin.TestRyuPortsV2.test_update_port_update_ip

  File "/home/ubuntu/neutron/.venv/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "neutron/openstack/common/rpc/impl_fake.py", line 67, in _inner
    namespace, **args)
  File "neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
    result = getattr(proxyobj, method)(ctxt, **kwargs)
  File "neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 296, in port_update
    vif_port = self.int_br.get_vif_port_by_id(port['id'])
  File "neutron/agent/linux/ovs_lib.py", line 362, in get_vif_port_by_id
    print traceback.print_stack()

More intersting thing is that it occurs only when both OVS plugin test and Ryu plugin test are run.
More precisely, it happens when we run
- first neutron.tests.unit.openvswitch.test_ovs_neutron_agent.TestOvsNeutronAgent, and
- then neutron.tests.unit.ryu.test_ryu_plugin.TestRyuPortsV2.test_update_port

$ source .venv/bin/activate
$ OS_DEBUG=1 python setup.py testr --testr-args='--concurrency=4 neutron.tests.unit.openvswitch.test_ovs_neutron_agent.TestOvsNeutronAgent neutron.tests.unit.ryu.test_ryu_plugin.TestRyuPortsV2.test_update_port'

[Akihiro Motoki (amotoki)]

LOG output of neutron.tests.unit.ryu.test_ryu_plugin.TestRyuPortsV2.test_update_port_not_admin
It is one of the unit tests where run-vsctl is called.

http://paste.openstack.org/show/54037/

[Akihiro Motoki (amotoki)]

I haven't identified the cause. If anyone jumps in, it is really appreciated.

[Akihiro Motoki (amotoki)]

It seems AgentRPCNotifier is not mocked in test_ryu_plugin, so PRC notification methods are invoked and as a result impl_fake dispatcher calls callback methods in previous tests. I am not sure impl_fake RPC is cleared after each test or not. Anyway, mocking ryu_plugin.notifier solves this problem.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/58711

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/58711
Committed: http://github.com/openstack/neutron/commit/dc2f567df3972465797d4d5b492ac1c243737cfe
Submitter: Jenkins
Branch: master

commit dc2f567df3972465797d4d5b492ac1c243737cfe
Author: Akihiro Motoki <email address hidden>
Date: Wed Nov 27 19:09:01 2013 +0900

    Mocking ryu plugin notifier in ryu plugin test

    Previously RPC notifier is not mocked in ryu plugin test and the
    plugin tries to send RPC notifications. It may interact with
    impl_fake RPC set up in previous tests by other plugin
    and sometimes leads to unexpected ovs-vsctl invoking.

    Change-Id: I11714a160faa4d3ded7f312c39afba0b426ccd8a
    Closes-Bug: #1255421

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/60291

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60291
Committed: http://github.com/openstack/neutron/commit/b52de8a3a9e156b440d9ec7a533888ceee1753a5
Submitter: Jenkins
Branch: stable/havana

commit b52de8a3a9e156b440d9ec7a533888ceee1753a5
Author: Akihiro Motoki <email address hidden>
Date: Wed Nov 27 19:09:01 2013 +0900

    Mocking ryu plugin notifier in ryu plugin test

    Previously RPC notifier is not mocked in ryu plugin test and the
    plugin tries to send RPC notifications. It may interact with
    impl_fake RPC set up in previous tests by other plugin
    and sometimes leads to unexpected ovs-vsctl invoking.

    Change-Id: I11714a160faa4d3ded7f312c39afba0b426ccd8a
    Closes-Bug: #1255421
    (cherry picked from commit dc2f567df3972465797d4d5b492ac1c243737cfe)