I'm trying to track here a bug I have seen in nova gate appearing randomly through rechecks.
Typical stack traces:
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 90, in wrapper
return f(*func_args, **func_kwargs)
File "/opt/stack/tempest/tempest/api/compute/admin/test_volume_swap.py", line 110, in test_volume_swap
volume1['id'], 'available')
File "/opt/stack/tempest/tempest/common/waiters.py", line 288, in wait_for_volume_resource_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: volume a19743a3-4651-4c7f-a9a1-823735ea84a0 failed to reach available status (current in-use) within the required time (196 s).
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 90, in wrapper
return f(*func_args, **func_kwargs)
File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 190, in test_live_block_migration_with_attached_volume
self.attach_volume(server, volume, device='/dev/xvdb')
File "/opt/stack/tempest/tempest/api/compute/base.py", line 581, in attach_volume
volume['id'], 'in-use')
File "/opt/stack/tempest/tempest/common/waiters.py", line 288, in wait_for_volume_resource_status
raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: volume 92685b8f-4db0-4110-a1ac-016ea7c51d1f failed to reach in-use status (current available) within the required time (196 s).
Typical jobs and tests:
nova-multi-cell
test_volume_swap[id-1769f00d-a693-4d67-a631-6a3496773813]
nova-live-migration
test_live_block_migration_with_attached_volume[id-e19c0cc6-6720-4ed8-be83-b6603ed5c812]
Example hits with (affecting multiple branches):
$ logsearch log --project openstack/nova --job nova-live-migration --result FAILURE --limit 50 "test_live_block_migration_with_attached_volume .* ... FAILED"
...
Builds with matching logs 10/50:
+----------------------------------+---------------------+----------+-----------------------------------+-----------------+
| uuid | finished | pipeline | review | branch |
+----------------------------------+---------------------+----------+-----------------------------------+-----------------+
| 36b367b0d0bb46d2a7fc6af4eb7739ca | 2022-10-07T19:39:47 | check | https://review.opendev.org/860736 | stable/victoria |
| d02ed047fcfd4180902dc0bec0334c38 | 2022-10-03T10:37:00 | check | https://review.opendev.org/854980 | stable/victoria |
| 0df9b00df16c4bbc9e49baf853fe0cf5 | 2022-09-19T09:47:02 | check | https://review.opendev.org/854980 | stable/victoria |
| 0db0e8d510d04443a172cc43e537f973 | 2022-09-16T14:14:31 | check | https://review.opendev.org/857877 | stable/train |
| 6ca30836a1b34be58728dc5d69c44c21 | 2022-09-16T10:33:55 | check | https://review.opendev.org/858051 | stable/victoria |
| 684e7c37c61745829908495ba249afb7 | 2022-09-16T10:14:07 | check | https://review.opendev.org/854980 | stable/victoria |
| 6bcf4105d0fc476faf9ee56e7f0ed41f | 2022-09-15T14:22:01 | check | https://review.opendev.org/857877 | stable/train |
| 0ea47624757c48a8bcfa9fd5c35b6465 | 2022-09-13T10:33:52 | check | https://review.opendev.org/854980 | stable/victoria |
| ca0d5f750b3040ed99c1e6ec3414d154 | 2022-09-06T17:28:41 | check | https://review.opendev.org/836830 | master |
| 2ce6d7aa67404587b050a6b56f4d15e6 | 2022-08-29T11:58:59 | check | https://review.opendev.org/833090 | master |
+----------------------------------+---------------------+----------+-----------------------------------+-----------------+
Yes, the examples show this issue. Actually we see this error time to time, though this mostly caught in train branch and rarely on other branches. Besides, the error has multiple forms as sometimes the volume is in-use, some times in detached status, etc. Sometimes we can see Kernel Panics and segment faults (in newer branches the console printouts are also reported in the logs). So this is not a new bug, but weirdly i haven't found the originally reported bugs (as I'm quite sure we have this somewhere already reported)