Functional test neutron.tests.functional.agent.linux.test_ip_lib.IpMonitorTestCase. test_add_remove_ip_address_and_interface is failing

[Slawek Kaplonski]

Some examples of failure:

http://logs.openstack.org/60/635060/7/check/neutron-functional-python27/be0cb14/testr_results.html.gz

http://logs.openstack.org/64/663464/2/check/neutron-functional-python27/43cf142/testr_results.html.gz

http://logs.openstack.org/14/639814/10/check/neutron-functional-python27/82513e0/testr_results.html.gz

[Brian Haley (brian-haley)]

So I looked at:

http://logs.openstack.org/14/639814/10/check/neutron-functional-python27/82513e0/testr_results.html.gz

And it looks like the last address getting added never shows up in the temp file:

Defined IP addresses:
[{'event': 'added', 'name': 'int_3375df43-ef', 'cidr': '192.168.250.1/24'},
{'event': 'added', 'name': 'int_35f82664-c5', 'cidr': '192.168.250.2/24'},
{'event': 'added', 'name': 'int_b45a7b12-d9', 'cidr': '192.168.250.3/24'},
{'event': 'added', 'name': 'int_fbd453a0-32', 'cidr': '192.168.250.10/24'},
{'event': 'removed', 'name': 'int_fbd453a0-32', 'cidr': '192.168.250.10/24'},
{'event': 'added', 'name': 'int_ac1ff1e9-ab', 'cidr': '2001:db8::1/64'},
{'event': 'added', 'name': 'int_ac1ff1e9-ab', 'cidr': '2001:db8::2/64'}]

IP addresses registered:
{"broadcast": "192.168.250.255", "dadfailed": false, "name": "int_3375df43-ef", "tentative": false, "scope": "global", "cidr": "192.168.250.1/24", "dynamic": false, "event": "added"}
{"broadcast": "192.168.250.255", "dadfailed": false, "name": "int_35f82664-c5", "tentative": false, "scope": "global", "cidr": "192.168.250.2/24", "dynamic": false, "event": "added"}
{"broadcast": "192.168.250.255", "dadfailed": false, "name": "int_b45a7b12-d9", "tentative": false, "scope": "global", "cidr": "192.168.250.3/24", "dynamic": false, "event": "added"}
{"broadcast": "192.168.250.255", "dadfailed": false, "name": "int_fbd453a0-32", "tentative": false, "scope": "global", "cidr": "192.168.250.10/24", "dynamic": false, "event": "added"}
{"broadcast": null, "dadfailed": false, "name": "int_ac1ff1e9-ab", "tentative": false, "scope": "global", "cidr": "2001:db8::1/64", "dynamic": false, "event": "added"}
...

There's other addresses on other interfaces in the logs after that, but not '2001:db8::2/64'

Running a test in a namespace, one window running 'ip -o monitor address' and another running /sbin/ip commands as above showed all the addresses being output.

One of the other traces showed the 250.10 IP never getting added, but getting removed, but the IPv6 ones were fine.

So we'll need to run the code in question to try and debug further, or see what kernel version this was running if there is a clue there.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hi Brian:

My first suspect (and that was my main problem when implementing this feature) is the time spent in the monitor loop to other activities (for example, sleep(0)). ip.get() is reading a bufferless socket; the time spent out of this function makes this implementation prone to miss some updates.

  while not event_stop.ready():
    eventlet.sleep(0)
    ip_address = []
    with common_utils.Timer(timeout=2, raise_exception=False):
      ip_address = ip.get()

I'll propose a new implementation and tests to probe the robustness.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

*prove

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/664889

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/664889
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a20f4c08c421da2e2d0341d03d71654a82d87a21
Submitter: Zuul
Branch: master

commit a20f4c08c421da2e2d0341d03d71654a82d87a21
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Jun 12 10:39:55 2019 +0000

    Read IP monitor changes in a parallel thread

    In order to capture all IP address changes, the method reading the
    netlink socket will be executed in a parallel thread. Once the
    "ip_monitor" method is stopped, this blocking thread will be killed.

    A new functional test, "test_add_multiple_ips", is added in order to
    stress test this method.

    Change-Id: I8f1de4a31f97bab734a33f94c3069444defd870f
    Closes-Bug: #1832307

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Not in Stein stable branch, "neutron-proactive-backport-potential" tag should be removed.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.