Comment 2 for bug 1843294

Revision history for this message
Tee Ngo (teewrs) wrote :

I was informed that the platform tasks affining job is tied to the openstack-compute-node label.
After removing the openstack related labels on both controllers, the platform tasks (except for k8s infra related) seem to be affined correctly. However, some pods failed to launch due to Insufficient Memory. It turned out that Kubernetes allocatable memory is tied to the openstack labels.

When the node has openstack labels, cpu and memory reserved for platform use are "visible" to Kubernetes and thus allocatable to pods. Below is the comparison of 2 controller nodes, one with openstack labels assigned and one without

Controller-0 (without openstack labels)
============
root 117578 1 4 13:05 ? 00:15:45 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --node-ip=face::3 --cpu-manager-policy=none

Capacity:
 cpu: 36
 ephemeral-storage: 10190100Ki
 hugepages-1Gi: 60Gi
 hugepages-2Mi: 0
 intel.com/pci_sriov_net_group0_data0: 64
 memory: 97528444Ki
 pods: 110
Allocatable:
 cpu: 36 <-- platform and vswitch cpus have not been deducted
 ephemeral-storage: 9391196145
 hugepages-1Gi: 60Gi
 hugepages-2Mi: 0
 intel.com/pci_sriov_net_group0_data0: 0
 memory: 34511484Ki <--- ~32G (platform mem has not been deducted)
 pods: 110

controller-1 (without openstack labels)
============
root 117762 1 3 15:24 ? 00:06:10 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --node-ip=face::4 --cpu-manager-policy=static --system-reserved-cgroup=/system.slice --system-reserved=cpu=2,memory=16500Mi

Capacity:
 cpu: 36
 ephemeral-storage: 10190100Ki
 hugepages-1Gi: 70Gi
 hugepages-2Mi: 0
 intel.com/pci_sriov_net_group0_data0: 64
 memory: 97528444Ki
 pods: 110
Allocatable:
 cpu: 34 <--- vswitch cpus have not been deducted
 ephemeral-storage: 9391196145
 hugepages-1Gi: 70Gi
 hugepages-2Mi: 0
 intel.com/pci_sriov_net_group0_data0: 64
 memory: 7129724Ki <--- ~6G (platform mem has been deducted)
 pods: 110