StarlingX

Kube Orchestration patch apply timeout too short

Bug #1927532 reported by Al Bailey on 2021-05-06

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Fix Released	Low	Al Bailey

Bug Description

Brief Description
-----------------
It was observed in a lab environment that applying the second patch during the VIM kubernetes upgrade orchestration, that the patch apply timed out after 20 seconds, while the apply succeeded after 23 seconds.

Distributed Cloud orchestration uses a much longer timeout value, but the VIM defaults to 20 seconds.

Severity
--------
Minor

Steps to Reproduce
------------------
Attempt to do a kubernetes upgrade orchestration in the vim.
This was only seen once in one lab.

Expected Behavior
------------------
Applying the second kubernetes patch (called KUBE.2 in the StarlingX kubernetes upgrade steps) should pass, and the rest of orchestration should succeed.

Actual Behavior
----------------
The VIM orchestration failed due to the timeout.
It needed to be re-initiated to run to completion.

Reproducibility
---------------
Seen Once

System Configuration
--------------------
Duplex+1

Branch/Pull Time/Commit
-----------------------
March 4 20201

Last Pass
---------
This is the only time this failure has been seen.
Its a timing issue.

Timestamp/Logs
--------------

VIM log
2021-05-06T14:45:36.312 controller-0 VIM_Thread[496797] ERROR Caught exception while trying to apply software patches [[u'KUBE.2']], error=[OpenStack Exception: method=POST, url=https://[fd01:1::2]:5492/v1/apply/KUBE.2, headers=None, body=None, reason=('The read operation timed out',)].
Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/nfv_plugins/nfvi_plugins/nfvi_sw_mgmt_api.py", line 242, in apply_patches
future.result = (yield)
OpenStackException: [OpenStack Exception: method=POST, url=https://[fd01:1::2]:5492/v1/apply/KUBE.2, headers=None, body=None, reason=('The read operation timed out',)]

patching log shows the apply sync completed 4 seconds later
021-05-06T14:45:40: sw-patch-controller-daemon[102525]: patch_controller.py(313): INFO: Handling sync complete

Test Activity
-------------
Feature testing

Workaround
----------
Delete the failed strategy, create a new one, and apply it. The kubernetes upgrade orchestration will
resume properly.

Tags:

Al Bailey (albailey1974) on 2021-05-06

Changed in starlingx:
assignee:	nobody → Al Bailey (albailey1974)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-06: Fix proposed to nfv (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/790135

Changed in starlingx:
status:	New → In Progress

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2021-05-07:

screening: low / intermittent issue and there is a workaround

tags:	added: stx.nfv
Changed in starlingx:
importance:	Undecided → Low

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-10: Fix merged to nfv (master)

Reviewed: https://review.opendev.org/c/starlingx/nfv/+/790135
Committed: https://opendev.org/starlingx/nfv/commit/e5856549e51f10ae6818ec1d0ec43568225e9bd9
Submitter: "Zuul (22348)"
Branch: master

commit e5856549e51f10ae6818ec1d0ec43568225e9bd9
Author: albailey <email address hidden>
Date: Thu May 6 12:46:29 2021 -0500

Increase the patching apply_patch REST API timeout

    During a kubernetes upgrade orchestration, the kubernetes
    patch needs to be applied. The default timeout was 20 seconds
    but a lab took 24 seconds.

Thi update increases the timeout for that API call.

    Closes-Bug: 1927532
    Signed-off-by: albailey <email address hidden>
    Change-Id: I63a6c5616f6abf7a5b6879e5ebd458a8ecc52ba7

Changed in starlingx:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-19: Fix proposed to nfv (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/nfv/+/792239

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-27: Change abandoned on nfv (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/nfv/+/792239

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-14: Fix proposed to nfv (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/nfv/+/796295

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-15:

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/nfv/+/796327

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-15: Change abandoned on nfv (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/nfv/+/796295

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-15: Fix merged to nfv (f/centos8)

Download full text (14.5 KiB)

Reviewed: https://review.opendev.org/c/starlingx/nfv/+/796327
Committed: https://opendev.org/starlingx/nfv/commit/96fa4281d73e701e58388228c8e8e85491785c38
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 73c683d5337beff6062b40f011f3b775f3c70107
Author: Eric MacDonald <email address hidden>
Date: Fri May 21 17:25:38 2021 -0400

Update fw-update-strategy steps to load wait_time from_dict

The sw-manager fw-update-strategy feature is seen
to fail in a traceback.

    The __wait_time member of the FwUpdateHostsStep and
    FwUpdateAbortHostsStep objects are not de-serialized
    from the DB using the ‘from_dict’ methods. This means
    it does not run the ‘init’ method for those classes,
    but instead attempts to re-constitute the object
    directly which can lead to an exception\traceback.

This update adds the _wait_time member to each of these
fw-update-strategy class objects' 'from_dict' function.

    This update also removes another object member, this one
    currently unused, that would also not be de-serialized
    if it were to be put to use as is in the future.

Test Plan:

PASS: Verify end-to-end orchestrated fw update (x2)

    Closes-Bug: 1929251
    Change-Id: I4540d1712f4dfee74e592c4f3ebce9c7cc913ab2
    Signed-off-by: Eric MacDonald <email address hidden>

commit 5ff24cf13f9d8cacab9ec15ff193fc8c819d31f4
Author: albailey <email address hidden>
Date: Fri May 21 17:51:38 2021 -0500

Specify the nodeset for zuul jobs

The py2.7 jobs need to specify xenial
Changed py37 to py36 and specify bionic.

The un-specified python3 jobs work fine on either
focal or bionic.

    zuul is not setup to trigger off code changes in this repo
    so no source code changes are required to trigger the zuul
    jobs

    Partial-Bug: 1928978
    Signed-off-by: albailey <email address hidden>
    Change-Id: Iab9c8727a0f16fa7ff02c20ca3bec5622abe7bd7

commit 98d66c7f3bc46e1a990907db1c8f498f9841c885
Author: albailey <email address hidden>
Date: Thu May 6 12:03:15 2021 -0500

Fix swact issue when deserializing an old patch strategy

    If a patch strategy in a previous release is de-serialized
    in the vim running a load that contains this commit
    https://review.opendev.org/c/starlingx/nfv/+/780310

the vim would fail to startup due to key errors as it
expected fields that did not exist in the previous release.

    Closes-Bug: 1927526
    Signed-off-by: albailey <email address hidden>
    Change-Id: Ia72463feb50f7d6a2491242ec865f7c854c75419

commit e5856549e51f10ae6818ec1d0ec43568225e9bd9
Author: albailey <email address hidden>
Date: Thu May 6 12:46:29 2021 -0500

Increase the patching apply_patch REST API timeout

    During a kubernetes upgrade orchestration, the kubernetes
    patch needs to be applied. The default timeout was 20 seconds
    but a lab took 24 seconds.

Thi update increases the timeout for that API call.

    Closes-Bug: 1927532
    Signed-off-by: albailey <email address hidden>
    Change-Id: I63a6c5616f6abf7a5b6879e5ebd458a8ecc52ba7

commit 4ffec1...

Reviewed:  https://review.opendev.org/c/starlingx/nfv/+/796327
Committed: https://opendev.org/starlingx/nfv/commit/96fa4281d73e701e58388228c8e8e85491785c38
Submitter: "Zuul (22348)"
Branch:    f/centos8

commit 73c683d5337beff6062b40f011f3b775f3c70107
Author: Eric MacDonald <eric.macdonald@windriver.com>
Date:   Fri May 21 17:25:38 2021 -0400

Update fw-update-strategy steps to load wait_time from_dict
    
    The sw-manager fw-update-strategy feature is seen
    to fail in a traceback.
    
    The __wait_time member of the FwUpdateHostsStep and
    FwUpdateAbortHostsStep objects are not de-serialized
    from the DB using the ‘from_dict’ methods. This means
    it does not run the ‘init’ method for those classes,
    but instead attempts to re-constitute the object
    directly which can lead to an exception\traceback.
    
    This update adds the _wait_time member to each of these
    fw-update-strategy class objects' 'from_dict' function.
    
    This update also removes another object member, this one
    currently unused, that would also not be de-serialized
    if it were to be put to use as is in the future.
    
    Test Plan:
    
    PASS: Verify end-to-end orchestrated fw update (x2)
    
    Closes-Bug: 1929251
    Change-Id: I4540d1712f4dfee74e592c4f3ebce9c7cc913ab2
    Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>

commit 5ff24cf13f9d8cacab9ec15ff193fc8c819d31f4
Author: albailey <Al.Bailey@windriver.com>
Date:   Fri May 21 17:51:38 2021 -0500

Specify the nodeset for zuul jobs
    
    The py2.7 jobs need to specify xenial
    Changed py37 to py36 and specify bionic.
    
    The un-specified python3 jobs work fine on either
    focal or bionic.
    
    zuul is not setup to trigger off code changes in this repo
    so no source code changes are required to trigger the zuul
    jobs
    
    Partial-Bug: 1928978
    Signed-off-by: albailey <Al.Bailey@windriver.com>
    Change-Id: Iab9c8727a0f16fa7ff02c20ca3bec5622abe7bd7

commit 98d66c7f3bc46e1a990907db1c8f498f9841c885
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu May 6 12:03:15 2021 -0500

Fix swact issue when deserializing an old patch strategy
    
    If a patch strategy in a previous release is de-serialized
    in the vim running a load that contains this commit
    https://review.opendev.org/c/starlingx/nfv/+/780310
    
    the vim would fail to startup due to key errors as it
    expected fields that did not exist in the previous release.
    
    Closes-Bug: 1927526
    Signed-off-by: albailey <Al.Bailey@windriver.com>
    Change-Id: Ia72463feb50f7d6a2491242ec865f7c854c75419

commit e5856549e51f10ae6818ec1d0ec43568225e9bd9
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu May 6 12:46:29 2021 -0500

Increase the patching apply_patch REST API timeout
    
    During a kubernetes upgrade orchestration, the kubernetes
    patch needs to be applied. The default timeout was 20 seconds
    but a lab took 24 seconds.
    
    Thi update increases the timeout for that API call.
    
    Closes-Bug: 1927532
    Signed-off-by: albailey <Al.Bailey@windriver.com>
    Change-Id: I63a6c5616f6abf7a5b6879e5ebd458a8ecc52ba7

commit 4ffec12dc88cf43dbf86620193020be30bde33d6
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Apr 22 18:15:01 2021 -0500

Add missing debug logger entries for VIM
    
    The debug loggers are pre-created during process startup,
    and any that are missing can be created using lazy init.
    
    However the lazy init is not multi process safe and can
    lead to a subprocess deadlock during the import phase, which
    produces no logs or evidence that the thread is stalled.
    
    The workaround at the moment is to ensure that all the debug
    loggers defined at import time are configured to be setup when
    the process is created.
    
    There is still another cause of a stall. A separate commit
    will address it.
    
    Partial-Bug: 1925697
    Signed-off-by: albailey <Al.Bailey@windriver.com>
    Change-Id: I90436c1bd063a634ab8a1496236a78b01a04d51a

commit c2f818c959c5709691f0d5c832e8a1777d287684
Author: albailey <Al.Bailey@windriver.com>
Date:   Mon Apr 19 10:25:03 2021 -0500

Handle patch install failure scenario in patch orchestration
    
    When a patch installs an RPM, but there is a post-installation failure
    caused by a restart script error, the host is considered both patch current
    and also patch failed.
    
    However VIM patch orchestration was considering it as successful.
    The vim logic to check the patch-failed flag is now being done before
    the patch current check, so properly characterize this type
    of patch orchestration problem.
    
    Closes-Bug: 1842952
    Signed-off-by: albailey <Al.Bailey@windriver.com>
    Change-Id: I2f8a784be4702537abff7996156344a3b558aefe

commit 483dc779b5c56f49f73ff8fb0dcd6b8afa6ad2cf
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Mar 18 16:58:19 2021 -0500

Add sw-manager --active show command
    
    - Add a new --active option for 'show' commands to only show
    stages, phases, and steps that are in progress.
    
    - Refactor initialization and execution of the CLI commands
    to reuse common code and simplify adding new update orchestrations.
    
    - Add unit tests for all strategies and action types for the CLI
     which increases test coverage for shell.py from 69 to 95 percent
    
    Story: 2008137
    Task: 42199
    Signed-off-by: albailey <Al.Bailey@windriver.com>
    Change-Id: I4280ef41ba7586c822d4d39629e51b4592fd996a

commit e1a3b7c7614b9c8d85860c55f6732ffb2100aa9d
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Mar 31 19:20:38 2021 -0500

Adding parallel apply capabilities to kube orch
    
    When a kubernetes orch defines a parallel apply type for workers
    or storage, the code used by patch orchestration to group the
    hosts, and add appropriate steps is now being re-purposed during
    this phase of kube orch.
    
    This change moves three of the patch host stage methods into mixins
    so that the code can be reused among different strategy types.
    
    The kubelet stages are now similar to patch in how they handle
    parallel apply types and VM hosts.
    
    The action type can now be passed in, and defaults to stop-start
    instead of migrate.
    
    Story: 2008137
    Task: 42198
    Signed-off-by: albailey <Al.Bailey@windriver.com>
    Change-Id: I284c84dea962c1d8f5548472767f5f9b3c6325d7

commit 454c71ddc0ac32bf2ab11e2ec54b884249ea63bf
Author: Eric MacDonald <eric.macdonald@windriver.com>
Date:   Tue Apr 6 09:11:49 2021 -0400

Modify nfv and mtce-guest log rotation config files
    
    This updates make the following setting changes
    to the nfv log rotation configuration files
    
     - add 'create' with permissions to guest conf files
     - add 'delaycompress' option
     - remove all global settings
     - remove the nodateext setting
    
    Test Plan:
    
    PASS: Verify log rotation file naming convention
    PASS: Verify delaycompress option
    PASS: Verify log permissions after rotate are 0640
    
    Change-Id: I033ee0585aedd7c70ff55f2ce3eb70867b78097c
    Partial-Bug: 1918979
    Depends-On: https://review.opendev.org/c/starlingx/config-files/+/784943
    Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>

commit eafee58e62b273aa3b65e497a4e4efc89f9e0715
Author: albailey <Al.Bailey@windriver.com>
Date:   Tue Feb 16 12:15:30 2021 -0600

Not require recreate of tox env when running tox
    
    The upgrade test was written to copy a file required for
    the test, but did not handle the file already being processed.
    This fix adjusts the setup for the unit test to handle this
    and allows the recreate to be removed, thus reducing designer
    tox testing time (40 seconds down to 4 when re-running a single test)
    
    This also aligns the version of hacking and pylint as the same as
    other starlingx repos. A new pylint was released which was
    breaking tox.
    
    The code changes are to fix whitespace (rather than suppressing):
     E305 expected 2 blank lines after class or function definition
     E117 over-indented
    
    Three new hacking warnings are being suppressed:
      W503 line break before binary operator
      W504  line break after binary operator
      W605 invalid escape sequence
    
    Un-suppress these codes which are not broken in the code:
     E123 closing bracket does not match indentation of opening bracket
     E501 line too long (tox sets max length instead to a large val)
     F823 local variable referenced before assignment
    
    yamllint 1.26.1 does not work with py2 or py3 in our env, so clamp it.
    
    Story: 2008137
    Task: 42200
    Signed-off-by: albailey <Al.Bailey@windriver.com>
    Change-Id: Ia623315c5f1a690d20f597242b7722601776f26d

commit 19b3a3f385d2449595dcf1546dd948aa093ed89b
Author: albailey <Al.Bailey@windriver.com>
Date:   Wed Mar 10 13:16:40 2021 -0600

Add retry capability to host unlock during upgrade
    
    During the unlock of a host as part of an upgrade, the
    unlock can be rejected.
    
    This change introduces a retry mechanism for the unlock.
    
    Allow up to 5 retries with 2 minutes between attempts.
    
    Partial-Bug: 1914836
    Signed-off-by: albailey <Al.Bailey@windriver.com>
    Change-Id: Ic121e1a993c80e2fae32806181342c1e5ea8e688

commit 2e8dbb5a7b8e12af5371a5ad8a587ac29a42f5a8
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Tue Mar 2 17:46:32 2021 -0500

Properly serialize all values for upgrade strategy
    
    This commit adds _single_controller attribute to to the "from_dict"
    and "as_dict" methods of upgrade strategy to maintain consistency
    with other types of strategy.
    
    Story: 2008055
    Task: 41974
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Change-Id: I04e34c214e39a7fe34e8ea38ee213db670c4837b

commit 90b480c3450823b8f2980904db6a30462377f636
Author: albailey <Al.Bailey@windriver.com>
Date:   Thu Nov 19 16:17:36 2020 -0600

Kubernetes Upgrade Orchestration
    
    Provides the new CLI command:
      sw-manager kube-upgrade-strategy
    
    VIM build stages are:
     - query-alarms
     - query-kube-upgrade
     - query-kube-versions
     - query-patches
     - query-patch-hosts
    
    VIM apply stages are:
     - kube-upgrade-start
     - download images
     - first control plane
     - networking
     - second control plane
     - apply second kubernetes patch
      -- applies the patch
      -- host-install on each controller
      -- host-install on each storage
      -- host-install on each worker
     - kubelets (controllers)
     - kubelets (workers)
     - complete
     - cleanup
    
    Functionality includes:
     - kube-upgrade API endpoint for orchestration.
     - new rpc messages for create kube strategy and intermediate actions.
     - kube-upgrade event handling, as well as alarm and event logs.
     - 'upgrade start' uses the latest sysinv health api to include
     the vim auto apply alarm in the ignore list for the health check.
    
    New unit tests:
     - build strategy phase
     - simplex controller
     - duplex controller (no existing kube upgrade)
    
    Story: 2008137
    Task: 41436
    Depends-On: https://review.opendev.org/c/starlingx/fault/+/767374
    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/775824
    Signed-off-by: albailey <Al.Bailey@windriver.com>
    Change-Id: I36e1b3ff3550a9d656ba40754b47570acc82a525

commit af3bddc28de745cd81731a921f3bcaaffb7b65cb
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Mon Feb 8 17:07:37 2021 -0500

Upgrade Orchestration for AIO-DX
    
    This commit provides support for orchestration of platform
    upgrades across all teh hosts in a single AIO-DX deployment
    (with or without worker nodes).
    
    The changes include:
    1) Allow an upgrade strategy to be created for an AIO-DX
       configuration.
    2) When dealing with controller hosts, treat it like a worker
       rather than a controller
    3) Modify worker code to handle controllers i.e. check which
       step should be added to the stage (WaitDataSyncStep,
       WaitAlarmsClearStep or SystemStabilizeStep) depending on
       the host personality and openstack installation
    
    Change-Id: I42daaa357d04eacd02be08ceab628882cb8987da
    Story: 2008055
    Task: 41829
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 9b79211a3c14a52ba9d2728bace2c0788360c99c
Author: Andrei Grosu <andrei.grosu@windriver.com>
Date:   Thu Jan 21 13:26:50 2021 -0500

Increase timeout to wait for alarms to clear.
    
    This handles the case when OSDs are deployed on controller nodes and
    patching fails after a controller is unlocked while there is an ongoing
    HEALTH_WARN from ceph.
    
    Closes-Bug: 1907259
    Signed-off-by: Andrei Grosu <andrei.grosu@windriver.com>
    Change-Id: Ibc71987049bc1040ca2c3c8db72bbac74cb35457

commit 281178e6b3829a1f8debbd44e47e2148a8379e71
Author: Thiago Brito <tbrito@daitan.com>
Date:   Fri Dec 11 15:36:32 2020 -0300

Fix patch strategy alarms checking
    
    This patch includes the WaitAlarmsClearStep also for workers that act
    as Openstack compute nodes.
    
    Closes-Bug: 1907851
    Change-Id: I85f1ef85b7f21b23110d4fcd388b010b56f478a5
    Signed-off-by: Thiago Brito <thiago.brito@windriver.com>

commit 36dd77b7a4e20433b4cf7ce2650f1b8660e5d964
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Wed Jan 13 05:06:29 2021 -0500

Ignore alarms for standard subcloud upgrade
    
    During a standard subcloud upgrade, an alarm is raised
    "A configuration change requires a re-apply of cert-manager"
    soon after controller-1 is upgraded. This alarm blocks VIM
    upgrade strategy creation which handles upgrade of
    controller-0. Thus, we ignore this alarm to proceed with the
    upgrade.
    
    Additionally, an info log is updated to print ID's of active
    alarms which cause VIM strategy creation to fail.
    
    Story: 2008055
    Task: 41578
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
    Change-Id: I960aba568e5c0d8aaf37c4e05587092e9b4d069a

commit d62cffb7a87c638b79e597c0593b1722b3ab65a8
Author: Don Penney <don.penney@windriver.com>
Date:   Thu Dec 17 13:24:47 2020 -0500

Add auto-version for remaining stx/nfv packages
    
    Update remaining StarlingX packages with hardcoded TIS_PATCH_VER to
    use PKG_GITREVCOUNT where possible, with offsets as needed to ensure
    the version is incremented above the hardcoded version.
    
    Change-Id: I9adc5f2648fda75b14215a27075c93a851bc6faa
    Story: 2008455
    Task: 41453
    Signed-off-by: Don Penney <don.penney@windriver.com>

tags:

added: in-f-centos8

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.