Functional test neutron.tests.functional.agent.linux.test_ip_lib.IpMonitorTestCase. test_add_remove_ip_address_and_interface is failing

Bug #1832307 reported by Slawek Kaplonski on 2019-06-11
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Critical
Rodolfo Alonso
Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
Brian Haley (brian-haley) wrote :

So I looked at:

http://logs.openstack.org/14/639814/10/check/neutron-functional-python27/82513e0/testr_results.html.gz

And it looks like the last address getting added never shows up in the temp file:

Defined IP addresses:
[{'event': 'added', 'name': 'int_3375df43-ef', 'cidr': '192.168.250.1/24'},
{'event': 'added', 'name': 'int_35f82664-c5', 'cidr': '192.168.250.2/24'},
{'event': 'added', 'name': 'int_b45a7b12-d9', 'cidr': '192.168.250.3/24'},
{'event': 'added', 'name': 'int_fbd453a0-32', 'cidr': '192.168.250.10/24'},
{'event': 'removed', 'name': 'int_fbd453a0-32', 'cidr': '192.168.250.10/24'},
{'event': 'added', 'name': 'int_ac1ff1e9-ab', 'cidr': '2001:db8::1/64'},
{'event': 'added', 'name': 'int_ac1ff1e9-ab', 'cidr': '2001:db8::2/64'}]

IP addresses registered:
{"broadcast": "192.168.250.255", "dadfailed": false, "name": "int_3375df43-ef", "tentative": false, "scope": "global", "cidr": "192.168.250.1/24", "dynamic": false, "event": "added"}
{"broadcast": "192.168.250.255", "dadfailed": false, "name": "int_35f82664-c5", "tentative": false, "scope": "global", "cidr": "192.168.250.2/24", "dynamic": false, "event": "added"}
{"broadcast": "192.168.250.255", "dadfailed": false, "name": "int_b45a7b12-d9", "tentative": false, "scope": "global", "cidr": "192.168.250.3/24", "dynamic": false, "event": "added"}
{"broadcast": "192.168.250.255", "dadfailed": false, "name": "int_fbd453a0-32", "tentative": false, "scope": "global", "cidr": "192.168.250.10/24", "dynamic": false, "event": "added"}
{"broadcast": null, "dadfailed": false, "name": "int_ac1ff1e9-ab", "tentative": false, "scope": "global", "cidr": "2001:db8::1/64", "dynamic": false, "event": "added"}
...

There's other addresses on other interfaces in the logs after that, but not '2001:db8::2/64'

Running a test in a namespace, one window running 'ip -o monitor address' and another running /sbin/ip commands as above showed all the addresses being output.

One of the other traces showed the 250.10 IP never getting added, but getting removed, but the IPv6 ones were fine.

So we'll need to run the code in question to try and debug further, or see what kernel version this was running if there is a clue there.

Hi Brian:

My first suspect (and that was my main problem when implementing this feature) is the time spent in the monitor loop to other activities (for example, sleep(0)). ip.get() is reading a bufferless socket; the time spent out of this function makes this implementation prone to miss some updates.

  while not event_stop.ready():
    eventlet.sleep(0)
    ip_address = []
    with common_utils.Timer(timeout=2, raise_exception=False):
      ip_address = ip.get()

I'll propose a new implementation and tests to probe the robustness.

*prove

Fix proposed to branch: master
Review: https://review.opendev.org/664889

Changed in neutron:
status: Confirmed → In Progress

Reviewed: https://review.opendev.org/664889
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a20f4c08c421da2e2d0341d03d71654a82d87a21
Submitter: Zuul
Branch: master

commit a20f4c08c421da2e2d0341d03d71654a82d87a21
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Jun 12 10:39:55 2019 +0000

    Read IP monitor changes in a parallel thread

    In order to capture all IP address changes, the method reading the
    netlink socket will be executed in a parallel thread. Once the
    "ip_monitor" method is stopped, this blocking thread will be killed.

    A new functional test, "test_add_multiple_ips", is added in order to
    stress test this method.

    Change-Id: I8f1de4a31f97bab734a33f94c3069444defd870f
    Closes-Bug: #1832307

Changed in neutron:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers