Debian: using VMs, bootstrap fails sysinv-api reload
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
In Progress
|
Undecided
|
Unassigned |
Bug Description
Brief Description
-----------------
We observe during a nightly sanity that during bootstrap sysinv-api may be slow to respond. Sanity is deploying AIO-SX.
Same behavior was observed during integration of AIO-DX, but work there was halted for a while.
Investigation:
---------------
This is the check that fails: https:/
Logging in to the affected systems and doing the curl will pass: curl http://
This proves it is just a slow system/optimization issue, and increasing the timeout will make it work.
Severity
--------
Critical: System/Feature is not usable due to the defect
Steps to Reproduce
------------------
run bootstrap
Expected Behavior
------------------
bootstrap passes
Actual Behavior
----------------
bootstrap fails
Reproducibility
---------------
Intermittent 50%.
System Configuration
-------
AIO-SX IPv4 on sanity
AIO-DX IPv4 on developer testing
maybe more, but tests not run.
Branch/Pull Time/Commit
-------
22, 23, 24 June
Last Pass
---------
?
Timestamp/Logs
--------------
2022-06-23 04:11:54,968 p=2357 u=sysadmin n=ansible | TASK [bootstrap/
2022-06-23 04:11:54,969 p=2357 u=sysadmin n=ansible | Thursday 23 June 2022 04:11:54 +0000 (0:00:00.049) 0:28:07.672 *********
2022-06-23 04:11:56,299 p=2357 u=sysadmin n=ansible | changed: [localhost] => (item=/
2022-06-23 04:12:17,023 p=2357 u=sysadmin n=ansible | failed: [localhost] (item=/
ansible_loop_var: item
cmd:
- /usr/lib/
- reload
delta: '0:00:19.959658'
end: '2022-06-23 04:12:16.883278'
item: /usr/lib/
msg: non-zero return code
rc: 1
start: '2022-06-23 04:11:56.923620'
stderr: |-
INFO: sysinv-api:status Sysinv API (sysinv-api) still hasn't stopped yet. Waiting ...
WARNING: /usr/lib/
INFO: sysinv-api:Old PID file found, but Sysinv API (sysinv-api) is not running
INFO: sysinv-api:Sysinv API (sysinv-api) is not running
INFO: sysinv-api:status Sysinv API (sysinv-api) stopped.
INFO: sysinv-api:start running with pid 73018
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #1
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #2
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #3
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #4
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #5
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #6
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #7
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #8
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #9
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #10
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #11
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #12
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #13
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #14
ERROR: Unable to run system show, trying direct request on sysinv-api URL (sysinv-api)
ERROR: Unable to communicate with the System Inventory Service (sysinv-api)
INFO: Retrying to connect to the System Inventory Service (sysinv-api), attempt #15
ERROR: Inventory Service (sysinv-api) failed to start (rc=1)
ERROR: System Inventory (sysinv-api) process failed to restart (rc=1)
stderr_lines: <omitted>
stdout: ''
stdout_lines: <omitted>
2022-06-23 04:12:17,034 p=2357 u=sysadmin n=ansible | PLAY RECAP *******
Test Activity
-------------
Sanity and Developer Testing
Workaround
----------
N/A
Changed in starlingx: | |
assignee: | nobody → Dan Voiculeasa (dvoicule) |
Changed in starlingx: | |
assignee: | Dan Voiculeasa (dvoicule) → nobody |
Fix proposed to branch: master /review. opendev. org/c/starlingx /config/ +/847526
Review: https:/