Scenario test failed due to metadata service not available

[Slawek Kaplonski]

Failure example: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_141/818844/2/check/neutron-tempest-plugin-scenario-openvswitch/1415d28/testr_results.html

It seems that vm didn't get metadata properly:

Starting acpid: OK
Starting network: udhcpc: started, v1.29.3
udhcpc: sending discover
udhcpc: sending select for 10.10.210.29
udhcpc: lease of 10.10.210.29 obtained, lease time 86400
route: SIOCADDRT: File exists
WARN: failed: route add -net "0.0.0.0/0" gw "10.10.210.254"
OK
checking http://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 2.16. request failed
failed 2/20: up 51.23. request failed
failed 3/20: up 100.32. request failed
failed 4/20: up 149.39. request failed
failed 5/20: up 198.45. request failed
failed 6/20: up 247.51. request failed
failed 7/20: up 296.57. request failed
failed 8/20: up 345.63. request failed
failed 9/20: up 394.69. request failed
failed 10/20: up 443.75. request failed
failed 11/20: up 492.80. request failed
failed 12/20: up 541.86. request failed
failed 13/20: up 590.93. request failed
failed 14/20: up 640.00. request failed
failed 15/20: up 689.07. request failed
failed 16/20: up 738.13. request failed
failed 17/20: up 787.20. request failed
failed 18/20: up 836.24. request failed
failed 19/20: up 885.32. request failed

[Slawek Kaplonski (slaweq)]

As Yatin pointed during the CI meeting, it seems that it's more generic problem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22failed%2017%2F20%3A%20up%20%5C%22

[Slawek Kaplonski (slaweq)]

From the logstash query it seems that this issue isn't as common as it looked at first glance. Most of the failures there are in one patch and seems to be related to that change.
I investigated logs from the failed job https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_141/818844/2/check/neutron-tempest-plugin-scenario-openvswitch/1415d28/testr_results.html but I didn't found anything wrong there really.

In the qrouter namespace there were proper iptable rules configured:

2021-12-21 04:47:56,864 112502 DEBUG [neutron_tempest_plugin.common.shell] Command 'sudo ip netns exec qrouter-57c936f4-06e9-4e93-968f-dbef81872b9e iptables-save' succeeded:
stderr:

stdout:
# Generated by iptables-save v1.8.4 on Tue Dec 21 04:47:56 2021
*raw
:PREROUTING ACCEPT [1242:55018]
:OUTPUT ACCEPT [986:48740]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
COMMIT
# Completed on Tue Dec 21 04:47:56 2021
# Generated by iptables-save v1.8.4 on Tue Dec 21 04:47:56 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [67:4020]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [31:1860]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 172.24.5.182/32 -j DNAT --to-destination 10.10.210.29
-A neutron-l3-agent-POSTROUTING ! -o qg-cb0064f3-83 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 172.24.5.182/32 -j DNAT --to-destination 10.10.210.29
-A neutron-l3-agent-float-snat -s 10.10.210.29/32 -j SNAT --to-source 172.24.5.182 --random-fully
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-cb0064f3-83 -j SNAT --to-source 172.24.5.118 --random-fully
-A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 172.24.5.118 --random-fully
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
COMMIT
# Completed on Tue Dec 21 04:47:56 2021
# Generated by iptables-save v1.8.4 on Tue Dec 21 04:47:56 2021
*mangle
:PREROUTING ACCEPT [1217:54018]
:INPUT ACCEPT [1050:44138]
:FORWARD ACCEPT [160:9600]
:OUTPUT ACCEPT [967:47980]
:POSTROUTING ACCEPT [1127:57580]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-floatingip - [0:0]
:neutron-l3-agent-mark - [0:0]
:neutron-l3-agent-scope - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-l3-agent-OUTPUT
-...