Sysinv kubernetes label audit re-adds removed node label
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
Kristine Bujold |
Bug Description
Brief Description
-----------------
Kubernetes host-label removed via system host-label-remove, yet label is still present on the kubernetes node.
Severity
--------
Minor: System/Feature is usable with minor issue
Steps to Reproduce
------------------
Race condition and may be hard to reproduce.
Expected Behavior
------------------
When a kubernetes node label is removed via system host-label-remove it should stay removed.
Actual Behavior
----------------
When a kubernetes node label is removed via system host-label-remove there is the small possibility of it being re-added by the audit.
Reproducibility
---------------
Rare
System Configuration
-------
Any
Branch/Pull Time/Commit
-------
Master 2020-03-24.
Last Pass
---------
N/A
Timestamp/Logs
--------------
Sysinv.log:
sysinv 2020-03-25 15:59:15.780 98963 INFO sysinv.
sysinv 2020-03-25 15:59:37.938 93999 INFO sysinv.
sysinv 2020-03-25 15:59:38.222 93999 INFO sysinv.
sysinv 2020-03-25 15:59:38.295 98964 INFO sysinv.
sysinv 2020-03-25 15:59:38.297 98964 INFO sysinv.
sysinv 2020-03-25 15:59:38.386 93999 INFO sysinv.
sysinv 2020-03-25 16:00:37.919 93999 INFO sysinv.
sysinv 2020-03-25 16:01:37.938 93999 INFO sysinv.
sysinv 2020-03-25 16:02:02.919 98963 INFO sysinv.
bash.log:
2020-03-
Database:
sysinv=# select * from label;
created_at | updated_at | deleted_at | id | uuid | host_id | label_key | label_value
-------
2020-03-24 18:20:09.282295 | | | 1 | b095a5cc-
2020-03-24 18:20:09.308784 | | | 2 | fa933fee-
2020-03-24 18:20:09.316646 | | | 3 | 3d0ad683-
2020-03-24 18:20:09.324641 | | | 4 | 2ff0ac28-
2020-03-24 18:20:15.868789 | | | 5 | 97cf8e49-
2020-03-24 18:20:15.879633 | | | 6 | f57a229c-
2020-03-24 18:20:15.899537 | | | 7 | 2f9100f3-
2020-03-24 18:20:15.915653 | | | 8 | 63a55a60-
2020-03-24 18:20:32.21488 | | | 9 | c764eab0-
2020-03-25 14:55:30.542282 | | | 11 | e44c1d69-
(10 rows)
Node labels:
controller-
NAME STATUS ROLES AGE VERSION LABELS
compute-0 Ready <none> 24h v1.16.2 beta.kubernetes
compute-1 Ready <none> 24h v1.16.2 beta.kubernetes
controller-0 Ready master 25h v1.16.2 beta.kubernetes
controller-1 Ready master 24h v1.16.2 beta.kubernetes
Test Activity
-------------
Developer Testing
Workaround
----------
The kubernetes node label must be manually removed via kubectl.
tags: | added: stx.containers |
Changed in starlingx: | |
status: | New → Triaged |
Changed in starlingx: | |
assignee: | nobody → Kristine Bujold (kbujold) |
Changed in starlingx: | |
status: | Triaged → In Progress |
stx.4.0 / medium priority - workaround exists, but should be investigated further