Revert resize problem in neutron-tempest-dvr-ha-multinode-full

[Slawek Kaplonski]

I saw few times in last 2 weeks that neutron-tempest-dvr-ha-multinode-full job failed in Neutron CI due to failed test tempest.api.compute.admin.test_migrations.MigrationsAdminTest.test_revert_cold_migration.

Failure error:

Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/api/compute/admin/test_migrations.py", line 146, in test_revert_cold_migration
    self._test_cold_migrate_server(revert=True)
  File "/opt/stack/tempest/tempest/api/compute/admin/test_migrations.py", line 132, in _test_cold_migrate_server
    server['id'], 'ACTIVE')
  File "/opt/stack/tempest/tempest/common/waiters.py", line 96, in wait_for_server_status
    raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (MigrationsAdminTest:test_revert_cold_migration) Server a3d545d4-8441-4b9d-b805-1461b7dc55d8 failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: REVERT_RESIZE. Current task state: resize_reverting.

Examples of failed jobs:

https://3e447a3e4caf9c486a4d-b84d447537aa444ce20bcf5414a5ef0e.ssl.cf5.rackcdn.com/707248/1/check/neutron-tempest-dvr-ha-multinode-full/057233d/testr_results.html

https://3ceba9124358a5c9851b-33ba471340a760763569a038d91ca6b5.ssl.cf2.rackcdn.com/706875/2/check/neutron-tempest-dvr-ha-multinode-full/6d77e6d/testr_results.html

Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22Current%20status%3A%20REVERT_RESIZE.%20Current%20task%20state%3A%20resize_reverting.%5C%22

[Balazs Gibizer (balazs-gibizer)]

A more recent occurrence as the original logs are gone now
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dbb/718462/3/check/neutron-tempest-dvr-ha-multinode-full/dbb69d0/job-output.txt

[Balazs Gibizer (balazs-gibizer)]

And there was 3 occurrence in the last 7 days so this is still actively happening.

[Lee Yarwood (lyarwood)]

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cd4/778705/1/check/neutron-tempest-dvr-ha-multinode-full/cd4513d/compute1/logs/screen-n-cpu.txt

Jun 04 09:48:22.912172 ubuntu-focal-inap-mtl01-0024968758 nova-compute[45518]: INFO os_vif [None req-d4eb23db-f44a-4665-a78b-7b34009fc626 tempest-ServerActionsTestJSON-1704629622 tempest-ServerActionsTestJSON-1704629622-project] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:7c:7f:e5,bridge_name='br-int',has_traffic_filtering=True,id=b1603fa9-d3b2-494b-a918-14d45fcabf35,network=Network(68eac88c-e506-4ecc-96f9-36d90dc9a2c7),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapb1603fa9-d3')
[..]
Jun 04 09:53:23.803974 ubuntu-focal-inap-mtl01-0024968758 nova-compute[45518]: WARNING nova.virt.libvirt.driver [None req-d4eb23db-f44a-4665-a78b-7b34009fc626 tempest-ServerActionsTestJSON-1704629622 tempest-ServerActionsTestJSON-1704629622-project] [instance: 4672cad8-86ed-4b5d-b9b1-d3b518f87333] Timeout waiting for [('network-vif-plugged', 'b1603fa9-d3b2-494b-a918-14d45fcabf35')] for instance with vm_state resized and task_state resize_reverting.: eventlet.timeout.Timeout: 300 seconds

With lots of ovsdbapp.backend.ovs_idl.vlog spam inbetween. I wonder if this is a duplicate of https://bugs.launchpad.net/nova/+bug/1929446 that sean-k-mooney is working on.