Hi team,
Please check the below error.
kube-system k8s-keystone-auth-7sqhw 1/1 Running 0 7d5h
kube-system k8s-keystone-auth-857h9 1/1 Running 0 7d5h
kube-system k8s-keystone-auth-h5fkr 1/1 Running 1 7d5h
kube-system kube-dns-autoscaler-5b4c644874-flhxt 0/1 Pending 0 4d20h
kube-system kubernetes-dashboard-7f844d86d6-j54pn 1/1 Running 52 7d5h
kube-system magnum-auto-healer-6nwjq 1/1 Running 158 7d5h
kube-system magnum-auto-healer-lv8kj 1/1 Running 179 7d5h
kube-system magnum-auto-healer-p65cl 1/1 Running 177 7d5h
kube-system magnum-grafana-f5b889c6f-fdwbl 0/2 Pending 0 4d20h
kube-system magnum-kube-state-metrics-6c97c54fd5-l9kt4 0/1 Pending 0 4d20h
kube-system magnum-metrics-server-7cc4fc5c64-478gq 0/1 Pending 0 4d20h
kube-system magnum-prometheus-adapter-648584d96c-v9gcw 0/1 Pending 0 4d20h
kube-system magnum-prometheus-node-exporter-jh4jm 1/1 Running 0 7d5h
kube-system magnum-prometheus-node-exporter-rrcj2 1/1 Running 0 7d5h
kube-system magnum-prometheus-node-exporter-sb6qr 1/1 Running 0 7d5h
kube-system magnum-prometheus-operator-operator-7885b9c9d9-tpjcz 0/2 Pending 0 4d20h
kube-system openstack-cloud-controller-manager-nqmgq 0/1 CrashLoopBackOff 756 7d5h
kube-system openstack-cloud-controller-manager-q5hp8 0/1 CrashLoopBackOff 762 7d5h
kube-system openstack-cloud-controller-manager-vsp8x 0/1 CrashLoopBackOff 765 7d5h
kube-system prometheus-magnum-prometheus-operator-prometheus-0 0/3 Pending 0 4d8h
[root@k8snew-o7diy4pcuaou-master-0 core]# kubectl get nodes -o wide
Error from server: etcdserver: leader changed
[root@k8snew-o7diy4pcuaou-master-0 core]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8snew-o7diy4pcuaou-master-0 Ready master 7d5h v1.21.1 192.168.45.13 192.168.101.17 Fedora CoreOS 32.20201104.3.0 5.8.17-200.fc32.x86_64 docker://19.3.11
k8snew-o7diy4pcuaou-master-1 Ready master 7d5h v1.21.1 192.168.45.12 192.168.101.11 Fedora CoreOS 32.20201104.3.0 5.8.17-200.fc32.x86_64 docker://19.3.11
k8snew-o7diy4pcuaou-master-2 Ready master 7d5h v1.21.1 192.168.45.11 192.168.101.9 Fedora CoreOS 32.20201104.3.0 5.8.17-200.fc32.x86_64 docker://19.3.11
[root@k8snew-o7diy4pcuaou-master-0 core]# kubectl logs -n kube-system cluster-autoscaler-c6c4fc9fd-qbh48
I0411 12:31:25.450095 1 leaderelection.go:242] attempting to acquire leader lease kube-system/cluster-autoscaler...
I0411 12:31:25.474139 1 leaderelection.go:252] successfully acquired lease kube-system/cluster-autoscaler
I0411 12:31:25.527377 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0411 12:31:25.527413 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
F0411 12:31:26.253479 1 magnum_cloud_provider.go:162] Failed to create magnum manager: unable to access cluster (c8640bfb-563d-449a-b0d3-e8c55bfbe7f2): The service is currently unable to handle the request due to a temporary overloading or maintenance. This is a temporary condition. Try again later.
[root@k8snew-o7diy4pcuaou-master-0 core]#
Hi Team, Below I have mentioned the Autos calling issue in Magnum.
Issue Summary:
The nodes are scaling down but , it's failed to scaling up and shooting below error.
Template:
openstack coe cluster template create k8s--calico- cinder- auto-health- largef_ min4_max5_ lb_21_1 --image fedora-coreos-32 --keypair k8s --external-network Magnum-Test --master-lb-enabled --dns-nameserver 8.8.8.8 --master-flavor g1t1.large --flavor g1t1.large --network-driver calico --coe kubernetes --label container_ infra_prefix= "tcsmagnum. tcsecp. com/tcsmagnum/ " --label 'docker_ volume_ type=az1- stable2' --label 'boot_volume_ size=40' --label boot_volume_ type=az1- stable2 --docker- volume- size 20 --docker- storage- driver overlay2 --label kube_tag=v1.21.1 --label calico_ ipv4pool= 10.100. 0.0/24 --label flannel_ network_ subnetlen= 28 --label flannel_ backend= host-gw --fixed-network 532ebede- e9d0-4ec4- 8bf1-abab1e8d78 6f --fixed-subnet eebe853c- 70bb-48f6- 8edc-6bb8f92b18 1e --label metrics_ server_ enabled= true --label monitoring_ enabled= true --label prometheus_ adapter_ enabled= true --label cinder_ csi_enabled= true --label grafana_ admin_passwd= linux --volume-driver cinder --label 'auto_healing_ enabled= True' --label 'auto_healing_ controller= magnum- auto-healer' --label 'auto_scaling_ enabled= True' --label 'min_node_count=1' --label 'max_node_count=6' --label 'health_ status= True' --label 'health_ status_ reason= True'
Auto scaler pod logs
I0423 20:06:55.746258 1 scale_down.go:638] Can't retrieve node maynew- mln6rohb3yuf- node-3 from snapshot, removing from unremovable map, err: node not found cache.go: 156] Start refreshing cloud provider node instances cache cache.go: 168] Refresh cloud provider node instances cache finished, refresh took 7.888µs mln6rohb3yuf- master- 0 core]#
I0423 20:07:25.965144 1 node_instances_
I0423 20:07:25.965332 1 node_instances_
[root@maynew-
I0423 19:39:25.958460 1 node_instances_ cache.go: 156] Start refreshing cloud provider node instances cache cache.go: 168] Refresh cloud provider node instances cache finished, refresh took 9.151µs cache.go: 156] Start refreshing cloud provider node instances cache cache.go: 168] Refresh cloud provider node instances cache finished, refresh took 9.119µs cache.go: 156] Start refreshing cloud provider node instances cache cache.go: 168] Refresh cloud provider node instances cache finished, refresh took 9.968µs cache.go: 156] Start refreshing cloud provider node instances cache cache.go: 168] Refresh cloud provider node instances cache finished, refresh took 12.974µs cache.go: 156] Start refreshing cloud provider node instances cache cache.go: 168] Refresh cloud provider node instances cache finished, refresh took 10.614µs cache.go: 156] Start refreshing cloud provider node insta...
I0423 19:39:25.958687 1 node_instances_
I0423 19:41:25.958877 1 node_instances_
I0423 19:41:25.958919 1 node_instances_
I0423 19:43:25.959125 1 node_instances_
I0423 19:43:25.959655 1 node_instances_
I0423 19:45:25.959969 1 node_instances_
I0423 19:45:25.960640 1 node_instances_
I0423 19:47:25.961019 1 node_instances_
I0423 19:47:25.961073 1 node_instances_
I0423 19:49:25.961326 1 node_instances_