"test_add_remove_fixed_ip" faling in "grenade-dvr-multinode" CI job

[Rodolfo Alonso]

This test is failing consistently in all patches (in master branch).

Error (some of them):
* https://685035fbfed20ba5f0bc-eca1b8de8e7b31708d9cffbfd5030903.ssl.cf2.rackcdn.com/779310/6/check/neutron-grenade-dvr-multinode/2ca9e82/controller/logs/grenade.sh_log.txt
* https://zuul.opendev.org/t/openstack/build/f687ab52350b419a8b0e976bc1daa6d5/log/controller/logs/grenade.sh_log.txt

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hi:

Checking the job executions [1], not all of them are failing but this test should be marked as unstable. Many executions are failing just because if this test.

Regards.

[1]https://zuul.opendev.org/t/openstack/builds?job_name=neutron-grenade-dvr-multinode

[Lajos Katona (lajos-katona)]

It's not always test_add_remove_fixed_ip, here (https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ab8/779721/3/check/neutron-grenade-dvr-multinode/ab87fa7/testr_results.html) it's test_network_basic_ops

[Lajos Katona (lajos-katona)]

The patch to make the job non-voting: https://review.opendev.org/c/openstack/neutron/+/782289

[Lajos Katona (lajos-katona)]

An earlier one: https://review.opendev.org/c/openstack/neutron/+/782275

[Slawek Kaplonski (slaweq)]

I was checking provided logs a bit and so far I just found couple of things:
1. Test is failing on "old" release,
2. It seems for me that connectivity is fine - SSH connection is established always but problem is with authentication, that suggests me some potential problem with metadata,
3. In neutron-metadata-agent's logs there is no any request from the IP address which belongs to that failed VM - that can be some clue to check deeper.

So far I don't know anything more. I will continue this investigation in next days.

[Oleg Bondarev (obondarev)]

From console log:

 Starting network: udhcpc: started, v1.29.3
 udhcpc: sending discover
 udhcpc: sending select for 10.1.0.6
 udhcpc: lease of 10.1.0.6 obtained, lease time 86400
 route: SIOCADDRT: File exists
 WARN: failed: route add -net "0.0.0.0/0" gw "10.1.0.1"
 OK
 checking http://169.254.169.254/2009-04-04/instance-id
 successful after 1/20 tries: up 9.52. iid=i-01ce8ea1
 failed to get http://169.254.169.254/2009-04-04/user-data
 warning: no ec2 metadata for user-data

So DHCP and metadata seem to work fine from neutron side. Looks like an issue with metadata itself: warning: no ec2 metadata for user-data

[Slawek Kaplonski (slaweq)]

Together with Rodolfo we probably found the problem. It was missing iptables nat rule which redirects packets from port 80 to 9697.
Patches proposed https://review.opendev.org/c/openstack/neutron/+/782677 and https://review.opendev.org/c/openstack/neutron/+/782690

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/782679
Committed: https://opendev.org/openstack/neutron/commit/ae07a9d9f61f9ae5539bfb074dcb24f618c495bb
Submitter: "Zuul (22348)"
Branch: master

commit ae07a9d9f61f9ae5539bfb074dcb24f618c495bb
Author: Slawek Kaplonski <email address hidden>
Date: Wed Mar 24 10:53:22 2021 +0100

    [CI] Enable debugging iptables rule in the L3 agent

    It should helps us understand why in some (rare) cases rule to redirect
    packets send 169.254.169.254:80 to port 9697 isn't installed in the
    qrouter namespace.

    Change-Id: I644ea3d6767db36bfe7f4122ec2c2afe9888dd07
    Related-Bug: #1920778

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/785600
Committed: https://opendev.org/openstack/neutron/commit/c028839647d6900997cf38b5eec63b7698515dec
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit c028839647d6900997cf38b5eec63b7698515dec
Author: Slawek Kaplonski <email address hidden>
Date: Wed Mar 24 12:02:14 2021 +0100

    Add locks for setting iptables rules in l3 and metadata agents

    Router_info class and metadata agent's driver are using same
    instance of the iptables manager class and it could happend that
    sometimes e.g. nat rule which packets send to 169.254.169.254:80
    redirects to the port 9697 so haproxy can process them, can be missed as
    they will be overwritten by the Router_info class manipulating other
    rules in the same 'nat' rules list.

    This patch fixed that by adding lock for methods which are changing
    rules in iptables_manager's nat table in both router_info and
    the metadata agent's driver.

    Closes-Bug: #1920778
    Change-Id: Ic3a324c0e608c7afc4b15dbc8becd33b75ee78f6
    (cherry picked from commit af3c1b84427cbe4c9d3dce8fc901ad0b099c5917)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/785649
Committed: https://opendev.org/openstack/neutron/commit/7af0b713ff21e27889f7b322b736cab4b0dadaf9
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 7af0b713ff21e27889f7b322b736cab4b0dadaf9
Author: Slawek Kaplonski <email address hidden>
Date: Wed Mar 24 12:02:14 2021 +0100

    Add locks for setting iptables rules in l3 and metadata agents

    Router_info class and metadata agent's driver are using same
    instance of the iptables manager class and it could happend that
    sometimes e.g. nat rule which packets send to 169.254.169.254:80
    redirects to the port 9697 so haproxy can process them, can be missed as
    they will be overwritten by the Router_info class manipulating other
    rules in the same 'nat' rules list.

    This patch fixed that by adding lock for methods which are changing
    rules in iptables_manager's nat table in both router_info and
    the metadata agent's driver.

    Conflicts:
        neutron/agent/metadata/driver.py

    Closes-Bug: #1920778
    Change-Id: Ic3a324c0e608c7afc4b15dbc8becd33b75ee78f6
    (cherry picked from commit af3c1b84427cbe4c9d3dce8fc901ad0b099c5917)
    (cherry picked from commit c028839647d6900997cf38b5eec63b7698515dec)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/785599
Committed: https://opendev.org/openstack/neutron/commit/51163ffa46426c1b805aee9c191ac66a718abdd9
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 51163ffa46426c1b805aee9c191ac66a718abdd9
Author: Slawek Kaplonski <email address hidden>
Date: Wed Mar 24 12:02:14 2021 +0100

    Add locks for setting iptables rules in l3 and metadata agents

    Router_info class and metadata agent's driver are using same
    instance of the iptables manager class and it could happend that
    sometimes e.g. nat rule which packets send to 169.254.169.254:80
    redirects to the port 9697 so haproxy can process them, can be missed as
    they will be overwritten by the Router_info class manipulating other
    rules in the same 'nat' rules list.

    This patch fixed that by adding lock for methods which are changing
    rules in iptables_manager's nat table in both router_info and
    the metadata agent's driver.

    Closes-Bug: #1920778
    Change-Id: Ic3a324c0e608c7afc4b15dbc8becd33b75ee78f6
    (cherry picked from commit af3c1b84427cbe4c9d3dce8fc901ad0b099c5917)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/785652
Committed: https://opendev.org/openstack/neutron/commit/657dccc566bc5699c2eea5b7ae6c10c24329b8b2
Submitter: "Zuul (22348)"
Branch: stable/train

commit 657dccc566bc5699c2eea5b7ae6c10c24329b8b2
Author: Slawek Kaplonski <email address hidden>
Date: Wed Mar 24 12:02:14 2021 +0100

    Add locks for setting iptables rules in l3 and metadata agents

    Router_info class and metadata agent's driver are using same
    instance of the iptables manager class and it could happend that
    sometimes e.g. nat rule which packets send to 169.254.169.254:80
    redirects to the port 9697 so haproxy can process them, can be missed as
    they will be overwritten by the Router_info class manipulating other
    rules in the same 'nat' rules list.

    This patch fixed that by adding lock for methods which are changing
    rules in iptables_manager's nat table in both router_info and
    the metadata agent's driver.

    Conflicts:
        neutron/agent/metadata/driver.py

    Closes-Bug: #1920778
    Change-Id: Ic3a324c0e608c7afc4b15dbc8becd33b75ee78f6
    (cherry picked from commit af3c1b84427cbe4c9d3dce8fc901ad0b099c5917)
    (cherry picked from commit c028839647d6900997cf38b5eec63b7698515dec)
    (cherry picked from commit 7af0b713ff21e27889f7b322b736cab4b0dadaf9)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/785653

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/785654

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/785655

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.3.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.3.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.1.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/827419

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/827419
Committed: https://opendev.org/openstack/neutron/commit/2d6b33445c2124a04f84d4d4e8d27675ba9d0b57
Submitter: "Zuul (22348)"
Branch: master

commit 2d6b33445c2124a04f84d4d4e8d27675ba9d0b57
Author: Slawek Kaplonski <email address hidden>
Date: Wed Feb 2 08:11:04 2022 +0100

    Revert "Switch neutron-grenade-dvr-multinode to be non voting temporary"

    This reverts commit 04e5a42f70605b0d1e5a3cc33bf4ee09a0da1052.

    Related bug is fixed, grenade dvr job is stable since long time so we
    can make it voting again.

    Related-Bug: #1920778
    Change-Id: Ica3bb4d44133e5d2d58f5bbb4a3747cf3d8811e2