Attempt to change a label key value that exists will update in kubernetes but not sysinv

[Wendy Mitchell]

Brief Description
-----------------
Attempt to change a label key value that already exists will update in kubernetes but silently fail in sysinv (resulting in setting discrepancies)

Severity
--------
standard

Steps to Reproduce
------------------

1. assign a label to a locked worker node
eg, add label "apps=enabled"
2019-10-28T14:29:35.000 controller-1 -sh: info HISTORY: PID=2705646 UID=42425 system host-lock controller-0
2019-10-28T14:31:41.000 controller-1 -sh: info HISTORY: PID=2705646 UID=42425 system host-label-assign controller-0 apps=enable

2. Check that the setting is added in kubernetes
Describe the node to see if the change is there
2019-10-28T14:36:38.000 controller-1 -sh: info HISTORY: PID=2705646 UID=42425 kubectl describe node controller-0

alternatively use the kubetcl get nodes command:
kubectl get nodes --show-labels

$ kubectl get node controller-0 --show-labels
NAME STATUS ROLES AGE VERSION LABELS
controller-0 Ready master 8h v1.16.2 apps=enabled,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=controller-0,kubernetes.io/os=linux,node-role.kubernetes.io/master=

Check that the label "apps" appears in the system inventory as "enabled"

$ system host-label-list controller-0
+--------------+-----------+-------------+
| hostname | label key | label value |
+--------------+-----------+-------------+
| controller-0 | apps | enabled

3. Now try to update the existing setting

$ system host-label-assign controller-0 apps=disabled
sysadmin@controller-1 log(keystone_admin)]$

Actual Behavior
--------------

The label updated in kubernetes but unexpectedly and silently failed to update in sysinv

$ kubectl get node controller-0 --show-labels
NAME STATUS ROLES AGE VERSION LABELS
controller-0 Ready master 8h v1.16.2 apps=disabled,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=controller-0,kubernetes.io/os=linux,node-role.kubernetes.io/master=

$ kubectl describe node controller-0 | grep apps=
Labels: apps=disabled

$ system host-label-list controller-0
+--------------+-----------+-------------+
| hostname | label key | label value |
+--------------+-----------+-------------+
| controller-0 | apps | enabled |

Expected Behavior
------------------
Expect that the label value should have updated to disabled in both kubernetes and system inventory

Reproducibility
---------------
Reproducible

System Configuration
--------------------
any

Branch/Pull Time/Commit
-----------------------
2019-10-27_20-00-00

Timestamp
---------
see bash.log
2019-10-28T14:58:23.000 controller-1 -sh: info HISTORY: PID=2898052 UID=42425 system host-label-assign controller-0 apps=disabled

see ERROR sysinv.db.sqlalchemy.api in the sysinv.log

2019-10-28 14:57:35.580 426814 WARNING wsme.api [-] Client-side error: Avoiding lock action on already 'locked' host controller-0
2019-10-28 14:58:25.237 426814 INFO sysinv.api.controllers.v1.label [-] patch_data: {'apps': 'disabled'}
2019-10-28 14:58:25.262 425329 INFO sysinv.conductor.manager [req-5ab31106-6d73-44ca-9b3d-b6e2298516a9 admin admin] update_kubernetes_label: label_dict={u'apps': u'disabled'}
2019-10-28 14:58:25.329 426814 ERROR sysinv.db.sqlalchemy.api [-] Failed to add host label apps. Already exists with this uuid
2019-10-28 14:58:25.330 426814 INFO sysinv.api.controllers.v1.rest_api [-] PATCH cmd:http://localhost:30001/nfvi-plugins/v1/hosts hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"hostname": "controller-0", "uuid": "128b798c-30e8-4f1f-99af-2e8a19ccbde0"}
2019-10-28 14:58:25.332 426814 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'status': u'success'}

Last Pass
--------------
Not passed

[Ghada Khalil (gkhalil)]

stx.3.0 / medium priority - discrepancy in label setting between k8s and sysinv

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/693950

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693950
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=b2e4d43f453773e266909936520b9b62f5542503
Submitter: Zuul
Branch: master

commit b2e4d43f453773e266909936520b9b62f5542503
Author: David Sullivan <email address hidden>
Date: Tue Nov 12 18:57:53 2019 -0500

    Add overwrite option to host-label-assign

    Add an overwrite option to host-label-assign. The behavior will be
    consistent with kubectl label --overwrite.

    If a label key exists for the host and no overwrite option is given the
    operation will be rejected. With the overwrite option provided any
    existing keys will be updated to the new value.

    Change-Id: If32dfe191f002f991d8357c05ca4ed0ca4308fb6
    Closes-Bug: 1850168
    Signed-off-by: David Sullivan <email address hidden>

[Wendy Mitchell (wmitchellwr)]

verified new option to --overwrite
2019-11-22_20-00-00

$ kubectl get node compute-0 --show-labels
NAME STATUS ROLES AGE VERSION LABELS
compute-0 Ready <none> 4d18h v1.16.2 apps=enabled,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=compute-0,kubernetes.io/os=linux,openstack-compute-node=enabled,openvswitch=enabled,sriov=enabled

$ system host-label-assign compute-0 apps=disabled
Label apps exists for host compute-0. Use overwrite option to assign a new value.

$ system host-label-assign
usage: system host-label-assign [--overwrite]
                                <hostname or id> <name=value>
                                [<name=value> ...]
system host-label-assign: error: too few arguments

$ system host-label-assign --overwrite compute-0 apps=disabled
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| uuid | fe009560-8872-41e7-bdaf-3e2ae05f0fe9 |
| host_uuid | 009d6ca6-dd55-4c27-825c-73390166733f |
| label_key | apps |
| label_value | disabled