nova.ci: test_server_rescue: Instance <_id> could not be found.

[Amit Uniyal]

Description
===========
test failed in master as well (locally too).

test:
api.compute.servers.test_server_rescue.ServerStableDeviceRescueTest.test_stable_device_rescue_disk_virtio_with_volume_attached

Body: b'{"itemNotFound": {"code": 404, "message": "Instance b531df61-3356-41d5-94c1-a66e6e9384d2 could not be found."}}'

stack trace:
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 70, in wrapper
    return f(*func_args, **func_kwargs)

      File "/opt/stack/tempest/tempest/api/compute/servers/test_server_rescue.py", line 244, in test_stable_device_rescue_disk_virtio_with_volume_attached
    server, rescue_image_id = self._create_server_and_rescue_image(

      File "/opt/stack/tempest/tempest/api/compute/servers/test_server_rescue.py", line 139, in _create_server_and_rescue_image
    server = self.create_test_server(

      File "/opt/stack/tempest/tempest/api/compute/base.py", line 272, in create_test_server
    body, servers = compute.create_test_server(

      File "/opt/stack/tempest/tempest/common/compute.py", line 338, in create_test_server
    with excutils.save_and_reraise_exception():

      File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

      File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

      File "/opt/stack/tempest/tempest/common/compute.py", line 333, in create_test_server
    wait_for_ssh_or_ping(

      File "/opt/stack/tempest/tempest/common/compute.py", line 148, in wait_for_ssh_or_ping
    waiters.wait_for_ssh(

      File "/opt/stack/tempest/tempest/common/waiters.py", line 644, in wait_for_ssh
    raise lib_exc.TimeoutException()

    tempest.lib.exceptions.TimeoutException: Request timed out
Details: None

VM did get created as we can see console logs.

[Elod Illes (elod-illes)]

Hi Amit, as I see test_stable_device_rescue_disk_virtio_with_volume_attached is passing now. Was it fixed meanwhile?

[Amit Uniyal (auniyal)]

Hi Elod, yeah I checked last 2 days builds fails and this only failed once [1] only, but in that job many test were failing so may be setup issue. Regarding fix Dan, Sean and gmann (may be others too) were/are rigorously working on fixing gate they merged patches too, may be that fixed it.

So We can close it for now. thanks for looking into it.

[1] https://134dc932f805eb187ef3-b10bddb1d588c4ea6f074bb4d648cc95.ssl.cf1.rackcdn.com/889992/2/gate/tempest-integrated-compute-rbac-old-defaults/c190c25/testr_results.html

this test is failing lot though: test_server_actions.ServerActionsTestOtherA.test_rebuild_server_with_volume_attached

[Amit Uniyal (auniyal)]

I am not sure with which status we close this.

[Elod Illes (elod-illes)]

Good question, I don't know either. If we don't know which patch fixed it, then Invalid / Incomplete maybe?

(The other test failure you linked has an OOM kill line [1], so probably that test reaches the point where there is not enough free memory on the controller)

[1] https://zuul.opendev.org/t/openstack/build/c190c250606e4cf9b689707d2e08d4e1/log/controller/logs/syslog.txt#6189