NFV-VIM: sw-deploy-strategy upgrade host step not setting timeouts

Bug #2093793 reported by Joshua Kraitberg
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Joshua Kraitberg

Bug Description

Brief Description
-----------------
Software deploy host step is not setting timeouts. This defaults it to 25 seconds instead of 180.

Severity
--------
Critical

Steps to Reproduce
------------------
Fresh install 24.09.1
sw-manager sw-deploy-strategy create WRCP-24.09.2 --alarm-restrictions relaxed --worker-apply-type parallel

Expected Behavior
------------------
Works

Actual Behavior
----------------
Timeout during deploy host step

Reproducibility
---------------
Intermittent. 100% for large patches.

System Configuration
--------------------
Any

Branch/Pull Time/Commit
-----------------------
Jan 10, 2025

Last Pass
---------
Never

Timestamp/Logs
--------------
2025-01-10T18:48:32.624 controller-0 VIM_Thread[1803563] ERROR _task_future.py.53 Set <function get_host_labels at 0x7f968fa9df70> timeout using module specific timeout: sysinv
2025-01-10T18:48:32.625 controller-0 VIM_Thread[1803563] DEBUG _task_work.py.36 TaskWork created, id=741, name=get_host_labels, timeout_in_secs=65.
2025-01-10T18:48:32.625 controller-0 VIM_Infrastructure-Worker-0_Thread[1803728] DEBUG _task_work.py.101 TaskWork run, id=741, name=get_host_labels.
2025-01-10T18:48:32.662 controller-0 VIM_Thread[1803563] INFO _strategy_steps.py.1149 Step (upgrade-hosts) apply for hosts ['controller-1'].
2025-01-10T18:48:32.662 controller-0 VIM_Thread[1803563] INFO _host_director.py.687 Upgrade hosts: ['controller-1']
2025-01-10T18:48:32.663 controller-0 VIM_Thread[1803563] DEBUG _host_director.py.692 Canceling previous host operation lock-hosts, before continuing with host operation upgrade-hosts.
2025-01-10T18:48:32.672 controller-0 VIM_Thread[1803563] DEBUG _host_tasks.py.607 Task (audit-disabled-host_controller-1) complete.
2025-01-10T18:48:32.673 controller-0 VIM_Thread[1803563] DEBUG _task_work.py.36 TaskWork created, id=742, name=sw_deploy_execute, timeout_in_secs=25.
2025-01-10T18:48:32.673 controller-0 VIM_Thread[1803563] DEBUG _task_work.py.36 TaskWork created, id=743, name=sw_deploy_get_upgrade_obj, timeout_in_secs=25.

Test Activity
-------------
Feature Testing

Workaround
----------
Reapply strategy
OR
Add timeouts in configs

Changed in starlingx:
assignee: nobody → Joshua Kraitberg (jkraitbe-wr)
Changed in starlingx:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-puppet (master)

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/938961
Committed: https://opendev.org/starlingx/stx-puppet/commit/2a97501d08b0d52d31c21dc70350b9dba7499e7e
Submitter: "Zuul (22348)"
Branch: master

commit 2a97501d08b0d52d31c21dc70350b9dba7499e7e
Author: Joshua Kraitberg <email address hidden>
Date: Fri Jan 10 15:46:45 2025 -0500

    Added NFV-VIM timeouts to puppet configuration

    These timeouts were missing on systems upgraded from previous releases.

    The absence of these timeouts can cause issues because default timeouts
    do not always work out.

    TEST PLAN
    PASS: Run platform::nfv::runtime manifest on already affected system
    * configs are updated
    PASS: AIO-SX patch upgrade
    * Post patch audit triggers platform::nfv::runtime
    PASS: AIO-SX major upgrade
    * Unlock triggers config updates

    Closes-Bug: 2093793
    Change-Id: Ie13534f548987a119499203574cbd403551c92a6
    Signed-off-by: Joshua Kraitberg <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Medium
tags: added: stx.11.0 stx.config stx.nfv
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.