Upgrade failed due to manual route config during upgrade
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
John Kung |
Bug Description
Description:
Upgrade Upgrade to stx6.0 failed when route config allowing during upgrade.
Severity
Major, blocking the progress of a number of DC related upgrades JIRA. Workaround is to lock/unlock. However, issue is due to manual config steps during upgrade.
Steps to Reproduce
Start the system controller upgrade job
Expected Behavior
System controller upgrade succeeds
Actual Behavior
Controller-0 upgrade fails
If you search "boot menu" in following web page:
http://
[2021-12-13 04:19:06,165] 95 INFO controller-0 menu.select :: Attempt to select boot device option [10;22HIBA XE Slot 8301 v2140 index 2
[2021-12-13 04:19:06,166] 99 INFO controller-0 menu.select :: Current index = 0
[2021-12-13 04:19:06,166] 125 INFO controller-0 menu.move_down:: Press: Down
[2021-12-13 04:19:07,168] 99 INFO controller-0 menu.select :: Current index = 1
[2021-12-13 04:19:07,168] 125 INFO controller-0 menu.move_down:: Press: Down
[2021-12-13 04:19:08,170] 686 INFO controller-0 menu.enter :: Press Enter (
)to select [10;22HIBA XE Slot 8301 v2140 option
In the job I initiated
http://
there is no such attempt.
Additional info:
It failed to swact to controller-1 due to the config-out-of-date alarm.
sysinv 2022-04-06 18:56:54.764 105950 INFO sysinv.
{u'classes': u'platform:
controller
sysinv 2022-04-06 18:56:54.765 105950 INFO sysinv.
sysinv 2022-04-06 18:56:54.765 105950 INFO sysinv.
sysinv 2022-04-06 18:56:55.652 114512 INFO sysinv.
host_uuids=
sysinv 2022-04-06 18:56:55.660 114512 INFO sysinv.
sysinv 2022-04-06 18:56:55.669 114512 WARNING sysinv.
sysinv 2022-04-06 18:56:55.670 114512 WARNING sysinv.
sysinv 2022-04-06 18:56:55.712 114512 INFO sysinv.
sysinv 2022-04-06 18:56:55.723 114512 INFO sysinv.
sysinv 2022-04-06 18:56:55.723 114512 INFO sysinv.
sysinv 2022-04-06 18:56:55.723 114512 INFO sysinv.
sysinv 2022-04-06 18:56:55.735 114512 INFO sysinv.
sysinv 2022-04-06 18:56:55.735 114512 WARNING sysinv.
The config target requires a lock/unlock because baf91f4b-
Reproducibility
Seen once.
System Configuration
Distributed Cloud
Load info (eg: 2022-03-
stx6.0
Branch and the time when code was pulled or git commit or cengn load info
Last Pass
Upgrade worked many times before in this lab
Timestamp/Logs
The issue is triggered due to subcloud add/delete operations during upgrade.
After a 'system upgrade-start' operation, the database is already snapshot between the 2 controllers.
The upgrade-start operation at 18:32:08
The route configuration changes are due to manual dcmanager cli commands to add/delete subcloud during the upgrade:
2022-04-
This results in a 250.001 Config out of date alarm which will prevent the host-swact of the upgrade step from completing (until host-lock/unlock workaround)
Alarms
250.001 Config out of date Alarm
Test Activity
Developer Testing
Workaround
host-lock/unlock controller to clear the (reboot-required) config out of date alarm
summary: |
- Upgrade to stx6.0 failed due to manual route config during upgrade + Upgrade failed due to manual route config during upgrade |
Changed in starlingx: | |
assignee: | nobody → John Kung (john-kung) |
tags: | added: stx.7.0 stx.config |
Changed in starlingx: | |
importance: | Undecided → Medium |
Fix proposed to branch: master /review. opendev. org/c/starlingx /config/ +/839202
Review: https:/