Fuel for OpenStack

resume instance with SR-IOV failed

Bug #1550937 reported by Noam Angel
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
MOS Linux
Fuel for OpenStack 9.0
8.0.x
Won't Fix
High
MOS Linux
Fuel for OpenStack 8.0-updates
Mitaka
Fix Released
High
MOS Linux
Fuel for OpenStack 9.0

Bug Description

From libvirt debug log:
2016-02-28 09:58:10.482+0000: 32424: debug : virJSONValueToString:1303 : result={"id":"libvirt-20","error":{"class":"GenericError","desc":"Device initialization failed."}}
2016-02-28 09:58:10.482+0000: 32419: debug : virEventPollCalculateTimeout:348 : Got a timeout scheduled for 1456653495480
2016-02-28 09:58:10.482+0000: 32424: debug : qemuMonitorJSONCheckError:366 : unable to execute QEMU command {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"02:02.0","id":"hostdev0","bus":"pci.0","addr":"0x4"},"id":"libvirt-20"}: {"id":"libvirt-20","error":{"class":"GenericError","desc":"Device initialization failed."}}
2016-02-28 09:58:10.482+0000: 32419: debug : virEventPollCalculateTimeout:361 : Schedule timeout then=1456653495480 now=1456653490482
2016-02-28 09:58:10.482+0000: 32419: debug : virEventPollCalculateTimeout:370 : Timeout at 1456653495480 due in 4998 ms
2016-02-28 09:58:10.482+0000: 32424: error : qemuMonitorJSONCheckError:377 : internal error: unable to execute QEMU command 'device_add': Device initialization failed.

Scenario:

nova suspend 080e547e-a8ca-45db-999d-a90e58c2e707
nova show 080e547e-a8ca-45db-999d-a90e58c2e707
+--------------------------------------+-------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------------------+
| AUTO_PRIVATE_NETWORK_0 network | 192.168.112.4, 10.209.86.32 |
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | node-43.domain.tld |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-43.domain.tld |
| OS-EXT-SRV-ATTR:instance_name | instance-0000005f |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | suspending |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-02-26T16:47:15.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-02-26T16:47:08Z |
| flavor | m1.small (2) |
| hostId | 7ffd82381e36b1274a115a313ce8cc120043260fb86fbfb90cde43bb |
| id | 080e547e-a8ca-45db-999d-a90e58c2e707 |
| image | AUTO_CentOS7.2_inbox (a640cb67-1122-48ce-ba1c-c888e8fb0c77) |
| key_name | - |
| metadata | {} |
| name | AUTO_VM0 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | 828400f227844e0c93947f83987affcb |
| updated | 2016-02-26T16:48:33Z |
| user_id | 8ace843b4fe04aeaa9a0f6248d96787e |
+--------------------------------------+-------------------------------------------------------------+

nova show 080e547e-a8ca-45db-999d-a90e58c2e707
+--------------------------------------+-------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------------------+
| AUTO_PRIVATE_NETWORK_0 network | 192.168.112.4, 10.209.86.32 |
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | node-43.domain.tld |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-43.domain.tld |
| OS-EXT-SRV-ATTR:instance_name | instance-0000005f |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | suspended |
| OS-SRV-USG:launched_at | 2016-02-26T16:47:15.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-02-26T16:47:08Z |
| flavor | m1.small (2) |
| hostId | 7ffd82381e36b1274a115a313ce8cc120043260fb86fbfb90cde43bb |
| id | 080e547e-a8ca-45db-999d-a90e58c2e707 |
| image | AUTO_CentOS7.2_inbox (a640cb67-1122-48ce-ba1c-c888e8fb0c77) |
| key_name | - |
| metadata | {} |
| name | AUTO_VM0 |
| os-extended-volumes:volumes_attached | [] |
| security_groups | default |
| status | SUSPENDED |
| tenant_id | 828400f227844e0c93947f83987affcb |
| updated | 2016-02-26T16:48:42Z |
| user_id | 8ace843b4fe04aeaa9a0f6248d96787e |
+--------------------------------------+-------------------------------------------------------------+

nova show 080e547e-a8ca-45db-999d-a90e58c2e707
+--------------------------------------+-------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------------------+
| AUTO_PRIVATE_NETWORK_0 network | 192.168.112.4, 10.209.86.32 |
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | node-43.domain.tld |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-43.domain.tld |
| OS-EXT-SRV-ATTR:instance_name | instance-0000005f |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | suspended |
| OS-SRV-USG:launched_at | 2016-02-26T16:47:15.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-02-26T16:47:08Z |
| flavor | m1.small (2) |
| hostId | 7ffd82381e36b1274a115a313ce8cc120043260fb86fbfb90cde43bb |
| id | 080e547e-a8ca-45db-999d-a90e58c2e707 |
| image | AUTO_CentOS7.2_inbox (a640cb67-1122-48ce-ba1c-c888e8fb0c77) |
| key_name | - |
| metadata | {} |
| name | AUTO_VM0 |
| os-extended-volumes:volumes_attached | [] |
| security_groups | default |
| status | SUSPENDED |
| tenant_id | 828400f227844e0c93947f83987affcb |
| updated | 2016-02-26T16:48:42Z |
| user_id | 8ace843b4fe04aeaa9a0f6248d96787e |
+--------------------------------------+-------------------------------------------------------------+

nova resume 080e547e-a8ca-45db-999d-a90e58c2e707

nova show 080e547e-a8ca-45db-999d-a90e58c2e707
+--------------------------------------+-------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------------------+
| AUTO_PRIVATE_NETWORK_0 network | 192.168.112.4, 10.209.86.32 |
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | node-43.domain.tld |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-43.domain.tld |
| OS-EXT-SRV-ATTR:instance_name | instance-0000005f |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | resuming |
| OS-EXT-STS:vm_state | suspended |
| OS-SRV-USG:launched_at | 2016-02-26T16:47:15.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-02-26T16:47:08Z |
| flavor | m1.small (2) |
| hostId | 7ffd82381e36b1274a115a313ce8cc120043260fb86fbfb90cde43bb |
| id | 080e547e-a8ca-45db-999d-a90e58c2e707 |
| image | AUTO_CentOS7.2_inbox (a640cb67-1122-48ce-ba1c-c888e8fb0c77) |
| key_name | - |
| metadata | {} |
| name | AUTO_VM0 |
| os-extended-volumes:volumes_attached | [] |
| security_groups | default |
| status | SUSPENDED |
| tenant_id | 828400f227844e0c93947f83987affcb |
| updated | 2016-02-26T16:49:07Z |
| user_id | 8ace843b4fe04aeaa9a0f6248d96787e |
+--------------------------------------+-------------------------------------------------------------+

nova show 080e547e-a8ca-45db-999d-a90e58c2e707
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| AUTO_PRIVATE_NETWORK_0 network | 192.168.112.4, 10.209.86.32 |
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | node-43.domain.tld |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-43.domain.tld |
| OS-EXT-SRV-ATTR:instance_name | instance-0000005f |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | 2016-02-26T16:47:15.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-02-26T16:47:08Z |
| fault | {"message": "internal error: unable to execute QEMU command 'device_add': Device initialization failed.", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 366, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 4138, in resume_instance |
| | block_device_info) |
| | File \"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py\", line 2421, in resume |
| | self._attach_sriov_ports(context, instance, guest, network_info) |
| | File \"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py\", line 3173, in _attach_sriov_ports |
| | guest.attach_device(cfg) |
| | File \"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py\", line 234, in attach_device |
| | self._domain.attachDeviceFlags(conf.to_xml(), flags=flags) |
| | File \"/usr/lib/python2.7/dist-packages/eventlet/tpool.py\", line 183, in doit |
| | result = proxy_call(self._autowrap, f, *args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/eventlet/tpool.py\", line 141, in proxy_call |
| | rv = execute(f, *args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/eventlet/tpool.py\", line 122, in execute |
| | six.reraise(c, e, tb) |
| | File \"/usr/lib/python2.7/dist-packages/eventlet/tpool.py\", line 80, in tworker |
| | rv = meth(*args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/libvirt.py\", line 528, in attachDeviceFlags |
| | if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self) |
| | ", "created": "2016-02-26T16:49:14Z"} |
| flavor | m1.small (2) |
| hostId | 7ffd82381e36b1274a115a313ce8cc120043260fb86fbfb90cde43bb |
| id | 080e547e-a8ca-45db-999d-a90e58c2e707 |
| image | AUTO_CentOS7.2_inbox (a640cb67-1122-48ce-ba1c-c888e8fb0c77) |
| key_name | - |
| metadata | {} |
| name | AUTO_VM0 |
| os-extended-volumes:volumes_attached | [] |
| security_groups | default |
| status | ERROR |
| tenant_id | 828400f227844e0c93947f83987affcb |
| updated | 2016-02-26T16:49:14Z |
| user_id | 8ace843b4fe04aeaa9a0f6248d96787e |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

Snapshot: https://drive.google.com/file/d/0B26TG8UdpL-ENXhVZkFHaW1ZdlU/view?usp=sharing

See original description
Tags: area-linux
Noam Angel (noama)
description: updated
description: updated
Roman Podoliaka (rpodolyaka)
tags: added: area-nova
Revision history for this message
Aviram Bar-Haim (aviramb) wrote :

when not using vfio driver (trying to use the kvm driver name) we see in dmesg:
[95784.780339] type=1400 audit(1456837752.529:380): apparmor="DENIED" operation="open" profile="libvirt-f39306bb-e483-4bd7-9c7f-f5b5b80341b1" name="/sys/devices/pci0000:80/0000:80:01.0/0000:81:02.0/vendor" pid=19407 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=109 ouid=0

Revision history for this message
Aviram Bar-Haim (aviramb) wrote :

After disabling apparmor and shutting off vfio kernel modules VM started with driver kvm as before and snapshot/suspend are working. So it seems that the main problem is with apparmor blocking libvirt drivers functionality.

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

MOS Linux, please take a look at this. We had a couple of similar issues with libvirt / apparmor before due to the fact we are using a custom package (from Debian?).

tags: added: area-linux
removed: area-nova
Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

Yeah, we saw that behavior as well. It was fixed in the following commit:
https://review.fuel-infra.org/gitweb?p=packages/trusty/libvirt.git;a=commit;h=4008d3238e2c62ae572efa17490b43a5232696fd

And it will be included in 9.0 release.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to packages/trusty/libvirt (8.0)

Fix proposed to branch: 8.0
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/30687

Revision history for this message
Ivan Suzdal (isuzdal) wrote :

8.0 uses upstream qemu which doesn't support sr-iov.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.