Neturon vm in error status after resize revert due to "Virtual Interface creation failed"

Bug #1888178 reported by Yvonne Ding
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Invalid
Medium
Austin Sun

Bug Description

Brief Description
-----------------
VM is in ERROR status after resize revert nova action. "Virtual Interface creation failed" is thrown by nova.exception.VirtualInterfaceCreationException.

Severity
--------
Major

Steps to Reproduce
------------------
1. Get a glance image with mgmt net and tenant port id
2. Boot a vm and flavor with 1 mgmt and 15 avp/virtio Interfaces
3. Perform nova actions (live live-migrate, hard reboot, soft rebuild, resize revert)
4. Ping between base_vm and vm_under_test over mgmt & tenant network

TC-name:
test_vm_with_max_vnics_attached_during_boot

Expected Behavior
-----------------
VM status is ACTIVE

Actual Behavior
----------------
VM status is ERROR

Reproducibility
---------------
reproducible

System Configuration
--------------------
Regular standard 2+2

Lab-name:
wcp_7_10

Branch/Pull Time/Commit
-----------------------
BUILD_ID="r/stx.4.0"

Timestamp/Logs
--------------
[2020-07-17 04:18:10,301] 61 DEBUG MainThread conftest.update_results:: ***Failure at test call: /home/svc-cgcsauto/wassp-repos.new/testcases/cgcs/CGCSAuto/keywords/vm_helper.py:1816: utils.exceptions.VMTimeout: VM operation timed out.
***Details: base_vm = ('9fe23839-f230-4277-a26c-5fef46147413', {'net-id': '1cf1603d-66b6-42e6-9178-2ad5ab78301c'}, {'net-id': 'caaf0b27-1c8a...'040440e3-5587-45dd-b7e4-4b946e2575ee', 'caaf0b27-1c8a-4a17-83ff-ddcc34b430dd', '1cf1603d-66b6-42e6-9178-2ad5ab78301c')
guest_os = 'tis-centos-guest', nic_arg = 'port_id', boot_source = 'image'

......

> vm_helper.resize_vm(vm_under_test, dest_flavor_id, revert=True, fail_ok=False)

testcases/functional/neutron/test_interface_attach_detach.py:347:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
keywords/vm_helper.py:1674: in resize_vm
    fail_ok=fail_ok, con_ssh=con_ssh)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

......

> raise exceptions.VMTimeout(err_msg)
E utils.exceptions.VMTimeout: VM operation timed out.
E Details: Timed out waiting for vm status: ['ACTIVE']. Actual vm status: ERROR

Signature
==========
[2020-07-17 04:18:07,084] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show 11bb85d1-4b13-46d8-afa9-1a81e521c28a'
[2020-07-17 04:18:09,877] 436 DEBUG MainThread ssh.expect :: Output:
+-------------------------------------+----------------------------------------------------------
| Field | Value |
+-------------------------------------+----------------------------------------------------------
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute-1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000000a |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | 2020-07-17T04:07:03.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | tenant1-mgmt-net=192.168.121.90; tenant1-net4=172.16.4.129, 172.16.4.141, 172.16.4.136, 172.16.4.227, 172.16.4.138, 172.16.4.193, 172.16.4.250, 172.16.4.148, 172.16.4.234, 172.16.4.146, 172.16.4.240, 172.16.4.158, 172.16.4.164, 172.16.4.167, 172.16.4.202 |
| config_drive | |
| created | 2020-07-17T03:51:29Z |
| fault | {u'message': u'Virtual Interface creation failed', u'code': 500, u'details': u'Traceback (most recent call last):\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6501, in _create_domain_and_network\n post_xml_callback=post_xml_callback)\n File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__\n next(self.gen)\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 513, in wait_for_instance_event\n actual_event = event.wait()\n File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/event.py", line 125, in wait\n result = hub.switch()\n File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch\n return self.greenlet.switch()\neventlet.timeout.Timeout: 300 seconds\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 205, in decorated_function\n return function(self, context, *args, **kwargs)\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 5028, in finish_revert_resize\n power_on)\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 10375, in finish_revert_migration\n external_events=events)\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6524, in _create_domain_and_network\n raise exception.VirtualInterfaceCreateException()\nnova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed\n', u'created': u'2020-07-17T04:13:30Z'} |

logs of .tar and automation log as below,
https://files.starlingx.kube.cengn.ca/launchpad/1888171

Test Activity
-------------

Yvonne Ding (yding)
description: updated
description: updated
Revision history for this message
Frank Miller (sensfan22) wrote :

TC fails and may be related to recent upversion to Ussuri. Marking as stx.5.0 gating instead of stx.4.0 as a revert of a resize is a less common scenario so doesn't need to gate the stx.4.0 release. Assigning to distro.openstack PL to determine how to proceed.

tags: added: stx.5.0 stx.distro.openstack
Changed in starlingx:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → yong hu (yhu6)
Austin Sun (sunausti)
Changed in starlingx:
assignee: yong hu (yhu6) → Austin Sun (sunausti)
Revision history for this message
Austin Sun (sunausti) wrote :

[2020-07-17 03:53:35,532] 1047 INFO MainThread vm_helper.boot_vm :: VM 11bb85d1-4b13-46d8-afa9-1a81e521c28a is booted successfully. tenant1-max_vifs-tis-centos-guest-image-5 (compute-0)
flavor-default-size2 (5e55f349-d17b-40ab-8a33-3fd435e699e3)
+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 2 |
| id | 5e55f349-d17b-40ab-8a33-3fd435e699e3 |
| name | flavor-default-size2 |
| os-flavor-access:is_public | True |
| properties | |
| ram | 1024 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+--------------------------------------+
2020-07-17 03:57:51,033] Live migration (compute-1)
[2020-07-17 04:02:01,045] reboot --hard
[2020-07-17 04:04:17,890] reboot soft

[2020-07-17 04:06:24,950] Create destination flavor ({'hw:mem_page_size': 'large'})
+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 2 |
| id | 4394e5ba-6647-4e8e-bf59-fd9c05d8df27 |
| name | dest_flavor |
| os-flavor-access:is_public | True |
| properties | |
| ram | 1024 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 2 |
+----------------------------+--------------------------------------+

[2020-07-17 04:06:37,573] Resizing VM 11bb85d1-4b13-46d8-afa9-1a81e521c28a to flavor 4394e5ba-6647-4e8e-bf59-fd9c05d8df27
                          compute-1---->compute-0
[2020-07-17 04:08:04,783] resized
[2020-07-17 04:08:04,783] start revert
[2020-07-17 04:08:21,933] switch compute-0---->compute-1
[2020-07-17 04:14:26,463] vm_state:error , Virtual Interface creation failed

nova-compute log
2020-07-17T04:13:29.992148735Z stdout nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed

Revision history for this message
Austin Sun (sunausti) wrote :

Hi Yvonne:
   is this issue 100% reproducible ? or occasionally?

Thanks.
BR
Austin Sun

Revision history for this message
Yvonne Ding (yding) wrote : Re: [Bug 1888178] Re: Neturon vm in error status after resize revert due to "Virtual Interface creation failed"

Hi Austin,

It's 100% reproducible.

Yvonne

On 2020-08-03 2:44 a.m., Austin Sun wrote:
> Hi Yvonne:
> is this issue 100% reproducible ? or occasionally?
>
> Thanks.
> BR
> Austin Sun
>

Revision history for this message
Austin Sun (sunausti) wrote :

Hi Yvonne:
   do you meet this issue recently ?

Thanks.
BR
Austin Sun.

Revision history for this message
Austin Sun (sunausti) wrote :

Move this to Stx.6.0 release

tags: added: stx.6.0
removed: stx.5.0
Revision history for this message
Austin Sun (sunausti) wrote :

as no update for long time, close this issue.

Changed in starlingx:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers