Boot a vm with pci-sriov and pci-passthrough failed by No valid host was found

Bug #1854516 reported by Peng Peng on 2019-11-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Medium
zhipeng liu

Bug Description

Brief Description
-----------------
Boot a vm with following vifs on same internal net: ('virtio', 'pci-sriov', 'pci-passthrough'), it is failed by 'No valid host was found. There are not enough hosts available.

Severity
--------
Major

Steps to Reproduce
------------------
see log for detail

TC-name: networking/test_multiple_ports.py::TestMutiPortsPCI::test_multiports_on_same_network_pci_vm_actions[virtio_pci-sriov_pci-passthrough]

Expected Behavior
------------------

Actual Behavior
----------------

Reproducibility
---------------
Seen once

System Configuration
--------------------
Multi-node system

Lab-name:

Branch/Pull Time/Commit
-----------------------
2019-11-21_20-00-00

Last Pass
---------
unknow

Timestamp/Logs
--------------
2019-11-25 02:06:20,260] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne port create --network=33db651b-a402-48c5-a432-e35560ecc079 --vnic-type=normal port_virtio-7'

[2019-11-25 02:06:22,850] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne port create --network=33db651b-a402-48c5-a432-e35560ecc079 --vnic-type=direct port_pci-sriov-8'

[2019-11-25 02:06:25,556] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne port create --network=33db651b-a402-48c5-a432-e35560ecc079 --vnic-type=direct-physical port_pci-passthrough-9'

[2019-11-25 02:06:27,976] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne port create --network=ce00e391-1eab-4bbe-9d4f-13e22be1a7c8 --vnic-type=direct-physical port_pci-passthrough-10'

[2019-11-25 02:06:37,788] 311 DEBUG MainThread ssh.send :: Send 'nova --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne boot --flavor=8bf0b131-4f9a-4e23-98e7-9ba54cd7a5aa --boot-volume=752ab134-6140-431b-8cf9-f366f0daee78 --key-name=keypair-tenant1 --poll --nic net-id=a35d11a3-7adc-4a73-a1c3-98ad4b547f23 --nic net-id=36498e81-3f8e-4b0a-815b-1ccc308c4bf4 --nic port-id=579913c4-a131-42fd-b211-bcd7322f000b --nic port-id=1bea07de-be5f-4142-9d82-17b69db26497 --nic port-id=f94315a7-36a5-4c85-b839-671f18c93842 --nic port-id=3cd6a181-1913-4f1a-b451-31e4fcebd194 tenant1-multiports_pci-21'
[2019-11-25 02:06:47,119] 433 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+---------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+---------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | FvFD5wFCBk8V |
| config_drive | |
| created | 2019-11-25T02:27:08Z |
| description | - |
| flavor:disk | 2 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {"hw:cpu_policy": "dedicated", "hw:mem_page_size": "large", "hw:pci_numa_affinity_policy": "preferred"} |
| flavor:original_name | dedicated-3 |
| flavor:ram | 2048 |
| flavor:swap | 0 |
| flavor:vcpus | 2 |
| hostId | |
| id | 4a791eab-d1eb-41a9-a1bb-92be03b001b9 |
| image | Attempt to boot from volume - no image supplied |
| key_name | keypair-tenant1 |
| locked | False |
| metadata | {} |
| name | tenant1-multiports_pci-21 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| server_groups | [] |
| status | BUILD |
| tags | [] |
| tenant_id | e6eda99366be44938afcdbbb941447f0 |
| trusted_image_certificates | - |
| updated | 2019-11-25T02:27:08Z |
| user_id | 6ba2a9882ccb476cb1b74f99a92daef7 |
+--------------------------------------+---------------------------------------------------------------------------------------------------------+

Server building... 0% complete
Error building server
ERROR (ResourceInErrorState):

[2019-11-25 02:06:47,180] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show 4a791eab-d1eb-41a9-a1bb-92be03b001b9'
                                                                                         |
| created | 2019-11-25T02:27:08Z |
| fault | {u'message': u'No valid host was found. There are not enough hosts available.', u'code': 500, u'details': u' File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 1346, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 800, in _schedule_instances\n return_alternates=return_alternates)\n File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call\n retry=self.retry)\n File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send\n retry=retry)\n File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send\n call_monitor_timeout, retry=retry)\n File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send\n raise result\n', u'created': u'2019-11-25T02:27:08Z'} |
| flavor | dedicated-3 (8bf0b131-4f9a-4e23-98e7-9ba54cd7a5aa) |
| hostId

Test Activity
-------------
Sanity
Feature Testing
Regression Testing
Developer Testing
Evaluation
Other - Please specify

Peng Peng (ppeng) wrote :
description: updated
Peng Peng (ppeng) wrote :
Ghada Khalil (gkhalil) wrote :

@Peng, Is this issue reproducible? It says "Seen once". Does that mean it passed in some cases? Or was the test run only once? Please run the test-case multiple times and update with the frequency of failure.

Please also indicate if the test-case was executed on stein previously (on the same setup) and whether it passed or not.

tags: added: stx.distro.openstack
Changed in starlingx:
status: New → Incomplete
assignee: nobody → Peng Peng (ppeng)
Peng Peng (ppeng) wrote :

Issue was 3/3 reproduced on train
2019-11-21_20-00-00
wcp_3-6

yong hu (yhu6) wrote :

@zhipeng, please support debugging this LP.

tags: added: stx.3.0
Changed in starlingx:
importance: Undecided → Medium
yong hu (yhu6) on 2019-12-03
Changed in starlingx:
assignee: Peng Peng (ppeng) → zhipeng liu (zhipengs)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers