calico-kube-controllers pod sometimes moves to worker node

[Yang Liu]

Brief Description
-----------------
calico-kube-controllers pod sometimes moves to worker node upon lock/unlock standby controller

Severity
--------
Minor

Steps to Reproduce
------------------
- calico-kube-controllers pod is initially on controller-0
- system host-swact controller-0 (pod is still on controller-0)
- lock/unlock controller-0 (now active controller is controller-1)
- check calico-kube-controllers pod

Expected Behavior
------------------
- calico-kube-controllers pod should move to controller-1

Actual Behavior
----------------
- calico-kube-controllers pod moved to worker node

Reproducibility
---------------
Intermittent

System Configuration
--------------------
Multi-node system

Branch/Pull Time/Commit
-----------------------
2019-10-09_20-00-00

Last Pass
---------
Not sure

Timestamp/Logs
--------------
# Lock controller-0
[2019-10-10 19:00:34,544] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[face::2]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-lock controller-0'

# Unlock controller-0
[2019-10-10 19:02:11,611] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[face::2]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock controller-0'

# calico controller pod moved to worker:
[2019-10-10 19:08:48,602] 311 DEBUG MainThread ssh.send :: Send 'kubectl get pod -o=wide --all-namespaces'
kube-system calico-kube-controllers-7f985db75c-lz8ph 1/1 Running 0 35m dead:beef::bc1b:6533:4fd4:e140 compute-2 <none> <none>

Test Activity
-------------
Sanity

[Ghada Khalil (gkhalil)]

Minor issue as there is no system impact, but architecturally speaking, this pod should be tied to a label to run on the controller nodes only. Would be nice to fix for stx.3.0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/695588

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/695588
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=f06c354e0843173aaae9d15d0f79c33f8e29e286
Submitter: Zuul
Branch: master

commit f06c354e0843173aaae9d15d0f79c33f8e29e286
Author: Joseph Richard <email address hidden>
Date: Thu Nov 21 14:06:35 2019 -0500

    restrict calico-kube-controllers to master

    This commit adds nodeSelector field of node-role.kubernetes.io/master
    calico-kube-controllers pod, so that it is only scheduled on controller
    nodes.

    Change-Id: Ia727843f25a0dbd92c3462bd23c0c1e723a82ece
    Closes-bug: 1848773
    Signed-off-by: Joseph Richard <email address hidden>

[Yang Liu (yliu12)]

This issue is no longer seen in recent sanity.