Functional dhcp agent tests fails to spawn metadata proxy

[Slawek Kaplonski]

In case when during spawning of the setup of dhcp agent, there will be iptabls error "Another app is currently holding the xtables lock" tests like neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_enable_isolated_metadata_for_subnet_create_delete and neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_force_metadata_for_subnet_create_delete may fail with error like below:

2021-03-30 20:57:01.829 61168 DEBUG neutron.agent.linux.dhcp [req-1d14cf38-d8a8-4f3a-858d-4ab6e9b888da - - - - -] Previous DHCP port information: <MagicMock name='DhcpPluginApi().create_dhcp_port()' id='139833378414448'>. Updated DHCP port information: <Mock name='DhcpPluginApi().get_dhcp_port()' id='139833360938464'>. _check_dhcp_port_subnet /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py:1582
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent [req-1d14cf38-d8a8-4f3a-858d-4ab6e9b888da - - - - -] Unable to enable dhcp for 24e1cf2a-a60d-41a9-9666-a38a90117cf9.: TypeError: can not serialize 'MagicMock' object
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/dhcp/agent.py", line 227, in call_driver
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent rv = getattr(driver, action)(**action_kwargs)
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py", line 266, in enable
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300)
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 707, in wait_until_true
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent while not predicate():
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py", line 278, in _enable
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network)
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/dhcp.py", line 1692, in setup
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent if ip_lib.ensure_device_is_ready(interface_name,
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 963, in ensure_device_is_ready
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent if not dev.link.exists or not dev.link.address:
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 500, in exists
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent return privileged.interface_exists(self.name, self._parent.namespace)
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent return self.channel.remote_call(name, args, kwargs)
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 214, in remote_call
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent result = self.send_recv((Message.CALL.value, name, args, kwargs))
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/comm.py", line 170, in send_recv
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent self.writer.send((myid, msg))
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/comm.py", line 54, in send
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent buf = msgpack.packb(msg, use_bin_type=True,
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent return Packer(**kwargs).pack(o)
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 298, in msgpack._cmsgpack.Packer.pack
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 295, in msgpack._cmsgpack.Packer.pack
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer._pack
2021-03-30 20:57:01.852 61168 ERROR neutron.agent.dhcp.agent TypeError: can not serialize 'MagicMock' object

Example of such error https://e9a79c0822aeb557489f-25bbaed08360ac14bb61b985414d524a.ssl.cf2.rackcdn.com/783743/1/check/neutron-functional-with-uwsgi/6f0aa76/controller/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase.test_enable_isolated_metadata_for_subnet_create_delete.txt

It happens like that because in that test there is no mock of neutron.agent.linux.dhcp.DeviceManager._update_dhcp_port() method. Workflow of such test is like below:

1. Call DeviceManager.setup() method
2. This method as one of the first steps will call _update_dhcp_port which will replace dhcp port prepared for test with some other mock,
3. During first run "port" variable in DeviceManager.setup() method is correct so all will work fine but if we hit iptables xlock error, setup() method will be called again and
4. Now "port" local variable is already update by _update_dhcp_port method thus test setup() method will fail as interface_name is now wrong.

[Slawek Kaplonski (slaweq)]

Fix proposed https://bugs.launchpad.net/neutron/+bug/1922684

[Slawek Kaplonski (slaweq)]

Sorry, correct link https://review.opendev.org/c/openstack/neutron/+/784903

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/785925
Committed: https://opendev.org/openstack/neutron/commit/d028f5977250d8fcdd42b3f9eb2e0dd8b0befb9b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit d028f5977250d8fcdd42b3f9eb2e0dd8b0befb9b
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 6 11:18:03 2021 +0200

    [Functional] Fix mocks of the create_dhcp_port method

    In dhcp agent functional tests
    test_force_metadata_for_subnet_create_delete and
    test_enable_isolated_metadata_for_subnet_create_delete it may happen
    that it will hit xlock error from iptables-restore command.
    Normally it is handled properly and DeviceManager.setup() method is then
    called again.
    But in those functional tests there was no mock of the create_dhcp_port
    but there was mock for get_dhcp_port and update_dhcp_port instead.
    Because of that when during first call of setup() method iptables
    exception was raised, during the second call of the setup() method wrong
    object was put in the network.ports and test was failing due to that.

    Workflow of that test is like below:
    1. Call DeviceManager.setup() method
    2. This method as one of the first steps will call _update_dhcp_port
    which will replace dhcp port prepared for test with some other mock,
    3. During first run "port" variable in DeviceManager.setup() method is
    correct so all will work fine but if we hit iptables xlock error,
    setup() method will be called again and
    4. Now "port" local variable is already update by _update_dhcp_port
    method thus test setup() method will fail as interface_name is now
    wrong.

    To avoid such issue this patch changed dhcp_port_mock to be "proper"
    DictModel() object instead of mock.Mock() and ensures that this will be
    set in network.ports even after "create_dhcp_port" will be called.

    Closes-Bug: #1922684
    Change-Id: I3f7dfdcbb3a54252bb1b3d2fa50eebcac3d00cba
    (cherry picked from commit 6533337716e5776567e0c9e9a8a20e2ad704e258)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/785926
Committed: https://opendev.org/openstack/neutron/commit/4407f75c83f0be2ce52647aded0bdd35f62d1f57
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 4407f75c83f0be2ce52647aded0bdd35f62d1f57
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 6 11:18:03 2021 +0200

    [Functional] Fix mocks of the create_dhcp_port method

    In dhcp agent functional tests
    test_force_metadata_for_subnet_create_delete and
    test_enable_isolated_metadata_for_subnet_create_delete it may happen
    that it will hit xlock error from iptables-restore command.
    Normally it is handled properly and DeviceManager.setup() method is then
    called again.
    But in those functional tests there was no mock of the create_dhcp_port
    but there was mock for get_dhcp_port and update_dhcp_port instead.
    Because of that when during first call of setup() method iptables
    exception was raised, during the second call of the setup() method wrong
    object was put in the network.ports and test was failing due to that.

    Workflow of that test is like below:
    1. Call DeviceManager.setup() method
    2. This method as one of the first steps will call _update_dhcp_port
    which will replace dhcp port prepared for test with some other mock,
    3. During first run "port" variable in DeviceManager.setup() method is
    correct so all will work fine but if we hit iptables xlock error,
    setup() method will be called again and
    4. Now "port" local variable is already update by _update_dhcp_port
    method thus test setup() method will fail as interface_name is now
    wrong.

    To avoid such issue this patch changed dhcp_port_mock to be "proper"
    DictModel() object instead of mock.Mock() and ensures that this will be
    set in network.ports even after "create_dhcp_port" will be called.

    Closes-Bug: #1922684
    Change-Id: I3f7dfdcbb3a54252bb1b3d2fa50eebcac3d00cba
    (cherry picked from commit 6533337716e5776567e0c9e9a8a20e2ad704e258)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.1.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.