Failure in 95-watch-apps-upgrade.sh script.

[Joshua Reed]

Brief Description
-----------------
SX upgrade from stx8 to stx9 failed by activation-failed

Severity
--------
Major

Steps to Reproduce
------------------
SX upgrade from stx8 to stx9

TC-name:

Expected Behavior
------------------
SX upgrade from stx8 to stx9 success

Actual Behavior
----------------
SX upgrade from stx8 to stx9 failed

Reproducibility
---------------
Intermittent

System Configuration
--------------------
One node system - SX

Last Pass
---------
2023-11-07_14-56-04

Timestamp/Logs
--------------
[2023-11-10 10:11:39,058] 349 DEBUG MainThread ssh.send :: Send 'system --os-endpoint-type internalURL --os-region-name RegionOne upgrade-activate'

[2023-11-10 10:53:30,178] 349 DEBUG MainThread ssh.send :: Send 'system --os-endpoint-type internalURL --os-region-name RegionOne upgrade-show'
[2023-11-10 10:53:30,229] 551 DEBUG MainThread ssh.exec_cmd:: Expecting [.@controller-[01] .(keystone_admin)]\$ in prompt
[2023-11-10 10:53:33,552] 471 DEBUG MainThread ssh.expect :: Output:
--------------------------------------------------+

Property Value

--------------------------------------------------+

uuid e426a8da-de67-4e14-ad81-f5052b74ebe0
state activation-failed
from_release stx8
to_release stx9

--------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$

sysinv.log shows
sysinv 2023-11-10 10:48:14.146 16526 ERROR sysinv.conductor.manager [-] Upgrade activation failed: Exception: Migration script /etc/upgrade.d/95-watch-apps-upgrade.sh failed with returncode 1Script output:
2023-11-10 10:48:14.146 16526 ERROR sysinv.conductor.manager Traceback (most recent call last):
2023-11-10 10:48:14.146 16526 ERROR sysinv.conductor.manager File "/usr/lib/python3/dist-packages/sysinv/conductor/manager.py", line 12920, in activate_upgrade
2023-11-10 10:48:14.146 16526 ERROR sysinv.conductor.manager upgrades_management.activate_upgrade(from_version,
2023-11-10 10:48:14.146 16526 ERROR sysinv.conductor.manager File "/usr/lib/python3/dist-packages/controllerconfig/upgrades/management.py", line 360, in activate_upgrade
2023-11-10 10:48:14.146 16526 ERROR sysinv.conductor.manager utils.execute_migration_scripts(from_load, to_load, utils.ACTION_ACTIVATE)
2023-11-10 10:48:14.146 16526 ERROR sysinv.conductor.manager File "/usr/lib/python3/dist-packages/controllerconfig/upgrades/utils.py", line 126, in execute_migration_scripts
2023-11-10 10:48:14.146 16526 ERROR sysinv.conductor.manager raise Exception(msg)
2023-11-10 10:48:14.146 16526 ERROR sysinv.conductor.manager Exception: Migration script /etc/upgrade.d/95-watch-apps-upgrade.sh failed with returncode 1Script output:
2023-11-10 10:48:14.146 16526 ERROR sysinv.conductor.manager

Test Activity
-------------
upgrade

[Joshua Reed (jreed7)]

Please note - This is meant as a temporary fix.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/917773
Committed: https://opendev.org/starlingx/config/commit/43ebe161d0e3f533d62cf2783115b22d8be6f143
Submitter: "Zuul (22348)"
Branch: master

commit 43ebe161d0e3f533d62cf2783115b22d8be6f143
Author: Reed, Joshua <email address hidden>
Date: Tue Apr 30 11:15:24 2024 -0600

    Increase timeout of 95-watch-apps-upgrade.sh by 50%

    A long term solution to avoid apps not updating in time
    would require significant changes to the upgrade
    system, so a simple change is being made in the short
    term to see if some additional time will help. This
    script only fails intermittently, so testing is simple
    in nature.

    Test Plan:
    PASS: Upgrade from stx8 to stx9 without errors.

    Closes-Bug: 2064315

    Change-Id: If2571837b005e604a1001412401e35b8ce711867
    Signed-off-by: Reed, Joshua <email address hidden>