Error: Request timed out. Details: Command: '... curl http://169.254.169.254/latest/meta-data/public-ipv4'

[Felipe Reyes]

Running the tempest smoke tests against focal-yoga-ovn and jammy-yoga-ovn fails when the test tries to reach the metadata service, curl fails with a timeout, although when manually spawning an instance, ssh into it and running curl succeeds, so this may be related to a race condition. In some tests runs the "public-ipv4" fails and in other the "password" endpoint fails, both endpoints are tested by the same test[0]

It's worth to mention that this testing was made on a s390x lpar.

A juju crashdump can be found at https://private-fileshare.canonical.com/~freyes/juju-crashdump-jammy-yoga-ovn-20220427.tar.xz

tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops[compute,id-7fff3fb3-91d8-4fd0-bd7d-0204f1f180ba,network,smoke]
----------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/ubuntu/git/zopenstack/tools/2-configure/tempest/tempest/common/utils/__init__.py", line 70, in wrapper
    return f(*func_args, **func_kwargs)

      File "/home/ubuntu/git/zopenstack/tools/2-configure/tempest/tempest/scenario/test_server_basic_ops.py", line 141, in test_server_basic_ops
    self.verify_metadata()

      File "/home/ubuntu/git/zopenstack/tools/2-configure/tempest/tempest/scenario/test_server_basic_ops.py", line 88, in verify_metadata
    CONF.compute.build_interval):

      File "/home/ubuntu/git/zopenstack/tools/2-configure/tempest/tempest/lib/common/utils/test_utils.py", line 116, in call_until_true
    if func(*args, **kwargs):

      File "/home/ubuntu/git/zopenstack/tools/2-configure/tempest/tempest/scenario/test_server_basic_ops.py", line 79, in exec_cmd_and_verify_output
    result = self.ssh_client.exec_command(cmd)

      File "/home/ubuntu/git/zopenstack/tools/2-configure/tempest/tempest/lib/common/utils/linux/remote_client.py", line 31, in wrapper
    return function(self, *args, **kwargs)

      File "/home/ubuntu/git/zopenstack/tools/2-configure/tempest/tempest/lib/common/utils/linux/remote_client.py", line 109, in exec_command
    return self.ssh_client.exec_command(cmd)

      File "/home/ubuntu/git/zopenstack/tools/2-configure/tempest/tempest/lib/common/ssh.py", line 204, in exec_command
    cmd, self.host))

    tempest.lib.exceptions.TimeoutException: Request timed out
Details: Command: 'set -eu -o pipefail; PATH=$PATH:/sbin:/usr/sbin; curl http://169.254.169.254/latest/meta-data/public-ipv4' executed on host '10.13.3.134'.

[0] https://opendev.org/openstack/tempest/src/branch/master/tempest/scenario/test_server_basic_ops.py#L69

[Felipe Reyes (freyes)]

This same test in a deployment running OVS + neutron-gateway shows no problems.