During load-import fail scenario, load is kept and can not be removed

Bug #1937168 reported by Adriano Oliveira
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Undecided
Unassigned

Bug Description

Brief Description
-----------------
system load-import failed during upgrade tests. There is no way to abort or delete the failed import.

Severity
--------
Minor: System/Feature is usable with minor issue

Steps to Reproduce
------------------
Attempt image load:

system load-import bootimage.iso bootimage.sig

Load failed with error:

 Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "start_import_load" info: "<unknown>"".

System fails to report aborted image import and is stuck in this state for ever. Reboot did not help. Delete or re-import will not work:

[sysadmin@controller-0 backups(keystone_admin)]$ system load-import bootimage.iso bootimage.sig
Max number of loads (2) reached. Please remove the old or unused load before importing a new one.
[sysadmin@controller-0 backups(keystone_admin)]$

Expected Behavior
-----------------
1. "system load-list" should reported correct state (i.e. not importing)

2. there should be a method for aborting or deleting the failed task

Should this happen in production environment it would halt the upgrade until a solution is provided.

Actual Behavior
---------------
system load-list claims an active import when there is none running.

Reproducibility
---------------
Seen once

System Configuration
--------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
n/a

Last Pass
---------
n/a.

Timestamp/Logs
--------------
...

sysinv.log:sysinv 2021-06-30 08:18:57.333 129725 INFO sysinv.api.controllers.v1.load [-] Load import request received.

sysinv.log:sysinv 2021-06-30 08:34:44.496 129725 ERROR wsme.api [-] Server-side error: "Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "start_import_load" info: "<unknown>"". Detail:

sysinv.log: File "/usr/lib64/python2.7/site-packages/sysinv/api/controllers/v1/load.py", line 292, in import_loadsysinv.log: return self._import_load()sysinv.log: File "/usr/lib64/python2.7/site-packages/sysinv/api/controllers/v1/load.py", line 346, in _import_load

sysinv.log: system_controller_import_active)

sysinv.log: File "/usr/lib64/python2.7/site-packages/sysinv/conductor/rpcapi.py", line 1226, in start_import_load

sysinv.log: import_active=import_active))sysinv.log:Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "start_import_load" info: "<unknown>"

sysinv.log:: Timeout: Timeout while waiting on RPC response - topic: "sysinv.conductor_manager", RPC method: "start_import_load" info: "<unknown>"

sysinv.log:sysinv 2021-06-30 10:48:08.612 129725 ERROR wsme.api [-] Server-side error: "Only a load in an imported or error state can be deleted". Detail:

sysinv.log: _("Only a load in an imported or error state can be deleted"))

sysinv.log:SysinvException: Only a load in an imported or error state can be deleted

...

[sysadmin@controller-0 backups(keystone_admin)]$ date
Thu Jul 1 11:24:29 UTC 2021
[sysadmin@controller-0 backups(keystone_admin)]$ system load-delete 2
Only a load in an imported or error state can be deleted
[sysadmin@controller-0 backups(keystone_admin)]$ uptime
{{ 11:24:43 up 1 day, 34 min, 1 user, load average: 2.54, 2.90, 3.45}}
[sysadmin@controller-0 backups(keystone_admin)]$

Test Activity
-------------
Feature testing

Workaround
----------
n/a

summary: - During load-import fail scenario load is kept and can not be removed
+ During load-import fail scenario, load is kept and can not be removed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/801706

Changed in starlingx:
status: New → In Progress
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers