Grenade job fails due to systemd stopping n-cpu
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ironic |
Fix Released
|
Critical
|
Julia Kreger |
Bug Description
Observed as part of a grenade failure, we discovered that n-cpu was being stopped prior to the upgrade completing.
Jan 18 05:48:47.109960 ubuntu-
Jan 18 05:48:49.185150 ubuntu-
Jan 18 05:48:49.185907 ubuntu-
Jan 18 05:48:49.185967 ubuntu-
The errors in the log are due to the ironic-api service being stopped during the upgrade and the redirect to the other ironic API server having not been put into place yet. In this specific case, the redirect was loaded at 05:50, which means n-cpu was already stopped.
In reality, what appears to be occurring is that the nova conductor process is spontaneously failing and entering into a loop at some point between the glance and swift upgrades.
Jan 18 05:49:48.642840 ubuntu-
Jan 18 05:49:48.649246 ubuntu-
Jan 18 05:49:48.657002 ubuntu-
Jan 18 05:49:48.681224 ubuntu-
Jan 18 05:49:48.686965 ubuntu-
Jan 18 05:49:48.695042 ubuntu-
What this appears to be is that that the service is failing as underlying libraries are being upgraded, however failing with a segmentation fault.
If this is truly a python library compatibility issue, then the only way to proceed forth is to upgrade nova as part of the upgrade. The downside is the "upgraded" scenario that we execute is where ironic-api is not upgraded, and the nova service is not upgraded at all.
Without some sort of virtualenv or container level isolation, we are unable to really solve this issue short of identifying the exact library compatibility issue or upgrading. Just restarting the n-cpu service is not enough.
Changed in ironic: | |
status: | Triaged → In Progress |
Changed in ironic: | |
status: | In Progress → Fix Released |
Seems to be viable and seems to work: Upgrading nova - We then hit microversion issues.
Restaring n-cpu. We should also try to restart n-cond and see if that impacts it.
Python3 instead of python2 - Swift does not start