After successful backup/restore, AIO-SX with N3000 requires multiple unlock attempts

[Caio Bruchert]

Description of failure:

The backup restore was successful and waited for around 25 minutes after the playbook finished. Yet, the unlock attempt was failed with error message:
"Expecting number of interface sriov_numvfs=8. Please wait a few minutes for inventory update and retry host-unlock.". The subcloud got unlocked on third attempt. The same issue happened when they tried restoring backup using dcmanager too.

Is the issue intermittent or is it 100% Reproducible? Frequency of Occurrence?

So far it is tested on two subclouds and the reproducibility is 100% (2/2)

Impact of Failure
Major

Did the system automatically recover?
The system was unlocked on third attempt

Steps to Reproduce
Perform a system backup and then restore.

Expected Behavior

The subcloud should not take any longer once the Ansible playbook is run successfully to restore backup data. Also, it should get unlocked on the first unlock attempt.

Actual Behavior

The subcloud was unable to process unlock request after Ansible playbook is run successfully to restore backup data. The customer waited for around 25 minutes for 1st subcloud unlock attempt after the ansible playbook was completed. It kept on failing with error message:

"Expecting number of interface sriov_numvfs=8. Please wait a few minutes for inventory update and retry host-unlock"

Until third unlock attempt.

Timestamp/Logs
NA

[Caio Bruchert (cbrucher)]

https://review.opendev.org/c/starlingx/stx-puppet/+/888323