System unlock rejected due to sriov vf assignment to host inventory

[Senthil Mukundakumar]

Brief Description
-----------------
In AIO-SX the controller failed to unlock due to sriov num vf not updated in host inventory.

Severity
--------
Critical: Controller failed to unlock

Steps to Reproduce:
------------------
Install sm-3 with following configuration:

[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0
+--------------------------------------+----------+-----------+----------+---------+-----------+-------------+---------------+---------------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |
+--------------------------------------+----------+-----------+----------+---------+-----------+-------------+---------------+---------------------------+
| 3ac0561f-278f-4cce-9d14-00ed7af2e4ac | data1 | data | ethernet | None | [u'eno4'] | [] | [] | MTU=1500,accelerated=True |
| 4859fb04-e660-4598-898d-398bfd576daa | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 |
| 610c39a2-df9b-4fe7-98b7-5bf67b293381 | sriov0 | pci-sriov | ethernet | None | [u'eno3'] | [] | [u'sriovfio'] | MTU=1500 |
| be2996f2-3f3a-476a-931a-6e03026000cc | sriovfio | pci-sriov | vf | None | [] | [u'sriov0'] | [] | MTU=1500 |
| ed8b54f8-4317-4eed-b7e5-2c95e5070f76 | lo | platform | virtual | None | [] | [] | [] | MTU=1500 |
+--------------------------------------+----------+-----------+----------+---------+-----------+-------------+---------------+---------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-port-list controller-0
+--------------------------------------+------+----------+--------------+--------+-----------+-------------+---------------------------------------------------+
| uuid | name | type | pci address | device | processor | accelerated | device type |
+--------------------------------------+------+----------+--------------+--------+-----------+-------------+---------------------------------------------------+
| 05c02a25-c7b4-4091-819f-83de2717d7ad | eno1 | ethernet | 0000:05:00.0 | 0 | 0 | True | I350 Gigabit Network Connection [1521] |
| 8224878a-0cb3-407c-a722-7fdc57dcbd94 | eno2 | ethernet | 0000:05:00.1 | 0 | 0 | True | I350 Gigabit Network Connection [1521] |
| d37ac1b8-4689-4362-8672-2f9af0265a5f | eno3 | ethernet | 0000:03:00.0 | 0 | 0 | True | Ethernet Connection X552/X557-AT 10GBASE-T [15ad] |
| 0bb24a6a-78e3-4e7f-824f-b8bbf009a0bb | eno4 | ethernet | 0000:03:00.1 | 0 | 0 | True | Ethernet Connection X552/X557-AT 10GBASE-T [15ad] |
+--------------------------------------+------+----------+--------------+--------+-----------+-------------+---------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-list controller-0
+--------------+--------------------------------------+--------+------------------+
| hostname | uuid | ifname | datanetwork_name |
+--------------+--------------------------------------+--------+------------------+
| controller-0 | 288957a1-0d67-400b-85da-f0128a22ed76 | sriov0 | group0-data0 |
| controller-0 | ac680132-9976-473d-970f-eab3671ad2b8 | data1 | group0-data1 |
+--------------+--------------------------------------+--------+------------------+

[sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0
Expecting number of interface sriov_numvfs=16. Please wait a few minutes for inventory update and retry host-unlock.

Expected Behavior
------------------
system should be installed and unlocked

Actual Behavior
----------------
Active controller failed to unlock

Reproducibility
---------------
Reproducible

System Configuration
--------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
master as of 20200116T212120Z

Timestamp
---------

[Yosief Gebremariam (ygebrema)]

E0117 20:33:03.987149 1 common.go:138] controller/host "msg"="user error" "error"="failed to unlock host: 3b9397dc-f7a6-40f1-864f-dd1485ae18f6, {\"action\":\"unlock\"}: Bad request with: [PATCH http://[abcd:204::1]:6385/v1/ihosts/3b9397dc-f7a6-40f1-864f-dd1485ae18f6], error message: {\"error_message\": \"{\\\"debuginfo\\\": null, \\\"faultcode\\\": \\\"Client\\\", \\\"faultstring\\\": \\\"Expecting number of interface sriov_numvfs=16. Please wait a few minutes for inventory update and retry host-unlock.\\\"}\"}" "request"={"Namespace":"deployment","Name":"controller-0"}

[Ghada Khalil (gkhalil)]

When was the last time this test-case passed? Which load was used?

[Ghada Khalil (gkhalil)]

Is this a reproducible issue on multiple systems or just one system?

[Steven Webster (swebster-wr)]

Is Intel virtualization technology and Intel VT-d enabled in the BIOS?

I see this in the kern.log:

2020-01-17T18:48:48.720 localhost kernel: err [ 4919.202247] ixgbe 0000:03:00.0: not enough MMIO resources for SR-IOV
2020-01-17T18:48:48.721 localhost kernel: warning [ 4919.209172] ixgbe 0000:03:00.0: Failed to enable PCI sriov: -12

Has SR-IOV ever worked on this system in the past?

[Ghada Khalil (gkhalil)]

@Senthil, any update on this?

[Yang Liu (yliu12)]

This is a configuration issue. system is unlocked successfully with SRIOV VFs after correcting the config. Please close.

[Ghada Khalil (gkhalil)]

Closing based on comment above