Brief Description ----------------- VM is ERROR status after resize revert nova action. "Virtual Interface creation failed" is thrown by nova.exception.VirtualInterfaceCreationException. Severity -------- Major Steps to Reproduce ------------------ 1. Get a glance image with mgmt net and tenant port id 2. Boot a vm and flavor with 1 mgmt and 15 avp/virtio Interfaces 3. Perform nova actions (live live-migrate, hard reboot hard, soft rebuild, resize revert) 4. Ping between base_vm and vm_under_test over mgmt & tenant network TC-name: test_vm_with_max_vnics_attached_during_boot Expected Behavior ----------------- VM status is ACTIVE Actual Behavior ---------------- VM status is ERROR Reproducibility --------------- reproducible System Configuration -------------------- Regular standard 2+2 Lab-name: wcp_7_10 Branch/Pull Time/Commit ----------------------- BUILD_ID="r/stx.4.0" Timestamp/Logs -------------- [2020-07-17 04:18:10,301] 61 DEBUG MainThread conftest.update_results:: ***Failure at test call: /home/svc-cgcsauto/wassp-repos.new/testcases/cgcs/CGCSAuto/keywords/vm_helper.py:1816: utils.exceptions.VMTimeout: VM operation timed out. ***Details: base_vm = ('9fe23839-f230-4277-a26c-5fef46147413', {'net-id': '1cf1603d-66b6-42e6-9178-2ad5ab78301c'}, {'net-id': 'caaf0b27-1c8a...'040440e3-5587-45dd-b7e4-4b946e2575ee', 'caaf0b27-1c8a-4a17-83ff-ddcc34b430dd', '1cf1603d-66b6-42e6-9178-2ad5ab78301c') guest_os = 'tis-centos-guest', nic_arg = 'port_id', boot_source = 'image' ...... > vm_helper.resize_vm(vm_under_test, dest_flavor_id, revert=True, fail_ok=False) testcases/functional/neutron/test_interface_attach_detach.py:347: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ keywords/vm_helper.py:1674: in resize_vm fail_ok=fail_ok, con_ssh=con_ssh) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...... > raise exceptions.VMTimeout(err_msg) E utils.exceptions.VMTimeout: VM operation timed out. E Details: Timed out waiting for vm status: ['ACTIVE']. Actual vm status: ERROR Signature ========== [2020-07-17 04:18:07,084] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show 11bb85d1-4b13-46d8-afa9-1a81e521c28a' [2020-07-17 04:18:09,877] 436 DEBUG MainThread ssh.expect :: Output: +-------------------------------------+---------------------------------------------------------- | Field | Value | +-------------------------------------+---------------------------------------------------------- | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | compute-1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1 | | OS-EXT-SRV-ATTR:instance_name | instance-0000000a | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | error | | OS-SRV-USG:launched_at | 2020-07-17T04:07:03.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | tenant1-mgmt-net=192.168.121.90; tenant1-net4=172.16.4.129, 172.16.4.141, 172.16.4.136, 172.16.4.227, 172.16.4.138, 172.16.4.193, 172.16.4.250, 172.16.4.148, 172.16.4.234, 172.16.4.146, 172.16.4.240, 172.16.4.158, 172.16.4.164, 172.16.4.167, 172.16.4.202 | | config_drive | | | created | 2020-07-17T03:51:29Z | | fault | {u'message': u'Virtual Interface creation failed', u'code': 500, u'details': u'Traceback (most recent call last):\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6501, in _create_domain_and_network\n post_xml_callback=post_xml_callback)\n File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__\n next(self.gen)\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 513, in wait_for_instance_event\n actual_event = event.wait()\n File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/event.py", line 125, in wait\n result = hub.switch()\n File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch\n return self.greenlet.switch()\neventlet.timeout.Timeout: 300 seconds\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 205, in decorated_function\n return function(self, context, *args, **kwargs)\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 5028, in finish_revert_resize\n power_on)\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 10375, in finish_revert_migration\n external_events=events)\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6524, in _create_domain_and_network\n raise exception.VirtualInterfaceCreateException()\nnova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed\n', u'created': u'2020-07-17T04:13:30Z'} | logs of .tar and automation log as below, Test Activity -------------