sw-manager kube-upgrade-strategy reports failure on debian

Bug #1996057 reported by Al Bailey
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Al Bailey

Bug Description

Brief Description
-----------------
On Debian, a kube-upgrade strategy will report a failure during its final cleanup stage.
The upgrade works as expected, but the vim strategy will claim that the cleanup failed.

Severity
--------
Minor

Steps to Reproduce
------------------
sw-manager kube-upgrade-strategy create --to-version v1.24.4
sw-manager kube-upgrade-strategy apply
kube-upgrade-strategy show --details

Expected Behavior
------------------
After about 30 minutes on an AIO-SX, the kube upgrade should complete and the strategy should report as successful

sw-manager kube-upgrade-strategy show
Strategy Kubernetes Upgrade Strategy:
  strategy-uuid: bbb02d0a-b399-4b64-a5cb-fae02f0cfd7a
  controller-apply-type: serial
  storage-apply-type: serial
  worker-apply-type: serial
  default-instance-action: stop-start
  alarm-restrictions: strict
  current-phase: apply
  current-phase-completion: 100%
  state: applied
  apply-result: success
  apply-reason:

Actual Behavior
----------------
It reports as failed

Reproducibility
---------------
100% on Debian

System Configuration
--------------------
AIO-SX (seen in any debian configuration)

Branch/Pull Time/Commit
-----------------------
After Oct 7, which was when a new version of kubernetes was added, so an upgrade could be attempted
https://review.opendev.org/c/starlingx/config/+/858833

Last Pass
---------
Based on my investigation of the root issue, this would never have passed on Debian.

Timestamp/Logs
--------------
nfv-vim logs shows this
2022-11-09T03:26:16.220 controller-0 VIM_Thread[175036] ERROR Caught exception while trying kube-upgrade-cleanup. error=[OpenStack Exception: method=DELETE, url=http://192.168.204.2:6385/v1/kube_upgrade, headers={'Content-Type': 'application/json', 'User-Agent': 'vim/1.0'}, body=None, reason=Expecting value: line 1 column 1 (char 0)]
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nfv_plugins/nfvi_plugins/nfvi_infrastructure_api.py", line 1825, in kube_upgrade_cleanup
    future.result = (yield)
nfv_plugins.nfvi_plugins.openstack.exceptions.OpenStackException: [OpenStack Exception: method=DELETE, url=http://192.168.204.2:6385/v1/kube_upgrade, headers={'Content-Type': 'application/json', 'User-Agent': 'vim/1.0'}, body=None, reason=Expecting value: line 1 column 1 (char 0)]

Test Activity
-------------
Developer Testing

Workaround
----------
 The kube upgrade completes fine, so just delete the nfv strategy.
However, tools like DC orchestration call the vim strategy so it impacts them

Revision history for this message
Al Bailey (albailey1974) wrote :

The issue is that on Debian, the no-content results are bytes, but the code that checks before doing a json load does a regular string check.

This was already addressed in https://bugs.launchpad.net/starlingx/+bug/1974475 but I needed to also update it in nfv-plugins/nfv_plugins/nfvi_plugins/openstack/rest_api.py

Changed in starlingx:
assignee: nobody → Al Bailey (albailey1974)
Changed in starlingx:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nfv (master)

Reviewed: https://review.opendev.org/c/starlingx/nfv/+/864109
Committed: https://opendev.org/starlingx/nfv/commit/60b161b9c28cb5100c9e41267678595a72068c30
Submitter: "Zuul (22348)"
Branch: master

commit 60b161b9c28cb5100c9e41267678595a72068c30
Author: Al Bailey <email address hidden>
Date: Wed Nov 9 12:27:57 2022 +0000

    Fix kube-upgrade-orchestration on debian

    During kube-upgrade orchestration, the final cleanup
    step returns an empty result body. The orchestrator
    was not handling this properly on Debian.

    In Debian, the API results are returned as bytes.
    The code that looks for an empty string needed to
    handle an empty byte string.

    Test Plan:
     PASS: Debian AIO-SX sw-manager kube-upgrade-strategy apply

    Closes-Bug: #1996057
    Related-Bug: #1974475

    Signed-off-by: Al Bailey <email address hidden>
    Change-Id: I37849f25fc908fb06b3e40cedfaa5690f45b289d

Changed in starlingx:
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Medium
tags: added: stx.8.0 stx.debian stx.nfv
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.