Hi Team, Please check the coe template which I have used for auto scaling and attached the logs. openstack coe cluster template create k8s--calico-cinder-auto-health-largef_min4_max5_lb --image fedora-coreos-32 --keypair k8s --external-network Magnum-Test --master-lb-enabled --dns-nameserver 8.8.8.8 --master-flavor g1t1.large --flavor g1t1.large --network-driver calico --coe kubernetes --label container_infra_prefix="tcsmagnum.tcsecp.com/tcsmagnum/" --label 'docker_volume_type=az1-stable2' --label 'boot_volume_size=40' --label boot_volume_type=az1-stable2 --docker-volume-size 20 --docker-storage-driver overlay2 --label kube_tag=v1.18.9 --label calico_ipv4pool=10.100.0.0/24 --label flannel_network_subnetlen=28 --label flannel_backend=host-gw --fixed-network 532ebede-e9d0-4ec4-8bf1-abab1e8d786f --fixed-subnet eebe853c-70bb-48f6-8edc-6bb8f92b181e --label metrics_server_enabled=true --label monitoring_enabled=true --label prometheus_adapter_enabled=true --label cinder_csi_enabled=true --label grafana_admin_passwd=linux --volume-driver cinder --label 'auto_healing_enabled=True' --label 'auto_healing_controller=magnum-auto-healer' --label 'auto_scaling_enabled=True' --label 'min_node_count=4' --label 'max_node_count=5' --label 'health_status=True' --label 'health_status_reason=True' root@maniltest:~# openstack coe cluster show cca3d284-33d5-478d-ba1f-c967285bbce9 --fit-width +----------------------+------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +----------------------+------------------------------------------------------------------------------------------------------------------------------------+ | status | CREATE_COMPLETE | | health_status | None | | cluster_template_id | 87b12a80-103a-46ea-8df8-b7ef0877abdc | | node_addresses | ['192.168.101.51', '192.168.101.65', '192.168.101.64', '192.168.101.57'] | | uuid | cca3d284-33d5-478d-ba1f-c967285bbce9 | | stack_id | 87de61cb-b949-430b-9c17-8223b3b30244 | | status_reason | None | | created_at | 2023-05-04T07:53:07+00:00 | | updated_at | 2023-05-04T08:07:59+00:00 | | coe_version | v1.18.9 | | labels | {'container_infra_prefix': 'tcsmagnum.tcsecp.com/tcsmagnum/', 'docker_volume_type': 'az1-stable2', 'boot_volume_size': '40', | | | 'boot_volume_type': 'az1-stable2', 'kube_tag': 'v1.18.9', 'calico_ipv4pool': '10.100.0.0/24', 'flannel_network_subnetlen': '28', | | | 'flannel_backend': 'host-gw', 'metrics_server_enabled': 'true', 'monitoring_enabled': 'true', 'prometheus_adapter_enabled': | | | 'true', 'cinder_csi_enabled': 'true', 'grafana_admin_passwd': 'linux', 'auto_healing_enabled': 'True', 'auto_healing_controller': | | | 'magnum-auto-healer', 'auto_scaling_enabled': 'True', 'min_node_count': '4', 'max_node_count': '5'} | | labels_overridden | {} | | labels_skipped | {} | | labels_added | {} | | faults | | | keypair | k8s | | api_address | https://192.168.101.50:6443 | | master_addresses | ['192.168.101.50'] | | create_timeout | 60 | | node_count | 4 | | discovery_url | https://discovery.etcd.io/d25a19902d2509218505f7ef75e7aad7 | | master_count | 1 | | container_version | 1.12.6 | | name | testnew | | master_flavor_id | g1t1.large | | flavor_id | g1t1.large | | health_status_reason | {} | | project_id | f97c046e99e146fea19127de7f9c63df | +----------------------+------------------------------------------------------------------------------------------------------------------------------------+ root@maniltest:~# root@maniltest:~# openstack coe nodegroup list cca3d284-33d5-478d-ba1f-c967285bbce9 +--------------------------------------+----------------+------------+------------------+------------+-----------------+--------+ | uuid | name | flavor_id | image_id | node_count | status | role | +--------------------------------------+----------------+------------+------------------+------------+-----------------+--------+ | 98491117-6ab3-463c-8d12-9937c6b5bdc4 | default-master | g1t1.large | fedora-coreos-32 | 1 | CREATE_COMPLETE | master | | 4bb80d26-d1f6-44b8-994b-1dbd25290a63 | default-worker | g1t1.large | fedora-coreos-32 | 4 | CREATE_COMPLETE | worker | +--------------------------------------+----------------+------------+------------------+------------+-----------------+--------+ root@maniltest:~# openstack coe nodegroup show cca3d284-33d5-478d-ba1f-c967285bbce9 4bb80d26-d1f6-44b8-994b-1dbd25290a63 +--------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | uuid | 4bb80d26-d1f6-44b8-994b-1dbd25290a63 | | name | default-worker | | cluster_id | cca3d284-33d5-478d-ba1f-c967285bbce9 | | project_id | f97c046e99e146fea19127de7f9c63df | | docker_volume_size | 20 | | labels | {'container_infra_prefix': 'tcsmagnum.tcsecp.com/tcsmagnum/', 'docker_volume_type': 'az1-stable2', 'boot_volume_size': '40', 'boot_volume_type': 'az1-stable2', 'kube_tag': 'v1.18.9', 'calico_ipv4pool': '10.100.0.0/24', 'flannel_network_subnetlen': '28', 'flannel_backend': 'host-gw', 'metrics_server_enabled': 'true', 'monitoring_enabled': 'true', 'prometheus_adapter_enabled': 'true', 'cinder_csi_enabled': 'true', 'grafana_admin_passwd': 'linux', 'auto_healing_enabled': 'True', 'auto_healing_controller': 'magnum-auto-healer', 'auto_scaling_enabled': 'True', 'min_node_count': '4', 'max_node_count': '5'} | | labels_overridden | {} | | labels_skipped | {} | | labels_added | {} | | flavor_id | g1t1.large | | image_id | fedora-coreos-32 | | node_addresses | ['192.168.101.51', '192.168.101.65', '192.168.101.64', '192.168.101.57'] | | node_count | 4 | | role | worker | | max_node_count | None | | min_node_count | 1 | | is_default | True | | stack_id | 87de61cb-b949-430b-9c17-8223b3b30244 | | status | CREATE_COMPLETE | | status_reason | Stack CREATE completed successfully | +--------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ root@maniltest:~# autoscaler logs W0504 08:06:09.929077 1 clusterstate.go:629] Readiness for node group default-worker not found W0504 08:06:09.929109 1 static_autoscaler.go:785] Couldn't find template for node group default-worker E0504 08:06:09.929189 1 static_autoscaler.go:415] Failed to scale up: Could not compute total resources: No node info for: default-worker W0504 08:06:09.929199 1 clusterstate.go:389] Failed to find readiness information for default-worker W0504 08:06:09.929205 1 clusterstate.go:451] Failed to find readiness information for default-worker W0504 08:06:09.929208 1 clusterstate.go:389] Failed to find readiness information for default-worker W0504 08:06:19.935182 1 clusterstate.go:451] Failed to find readiness information for default-worker W0504 08:06:19.935302 1 static_autoscaler.go:785] Couldn't find template for node group default-worker E0504 08:06:19.935469 1 static_autoscaler.go:415] Failed to scale up: Could not compute total resources: No node info for: default-worker W0504 08:06:29.943524 1 static_autoscaler.go:785] Couldn't find template for node group default-worker E0504 08:06:29.943658 1 static_autoscaler.go:415] Failed to scale up: Could not compute total resources: No node info for: default-worker W0504 08:06:39.952479 1 static_autoscaler.go:785] Couldn't find template for node group default-worker E0504 08:06:39.952610 1 static_autoscaler.go:415] Failed to scale up: Could not compute total resources: No node info for: default-worker W0504 08:06:49.959214 1 static_autoscaler.go:785] Couldn't find template for node group default-worker E0504 08:06:49.959385 1 static_autoscaler.go:415] Failed to scale up: Could not compute total resources: No node info for: default-worker I0504 08:07:48.237123 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:07:48.237772 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 50.382µs I0504 08:09:48.238007 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:09:48.238063 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 13.774µs I0504 08:11:48.238281 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:11:48.238693 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 14.355µs I0504 08:13:48.238948 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:13:48.239091 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 15.455µs I0504 08:15:48.239318 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:15:48.239515 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 14.885µs I0504 08:17:48.239766 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:17:48.239809 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 12.235µs I0504 08:19:48.240000 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:19:48.240405 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 17.217µs I0504 08:21:48.240894 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:21:48.241093 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 47.07µs I0504 08:23:48.241364 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:23:48.241408 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 11.781µs [root@testnew-wxdecq7kb4p5-master-0 core]# kubectl logs -n kube-system cluster-autoscaler-57b98d69c8-gvn82 ------------------------------------------------------------------------------------------------------------------------------ root@maniltest:~# openstack server list | grep -i testnew | efa09698-3b73-48a5-964e-abf9d5222e68 | testnew-wxdecq7kb4p5-node-0 | ACTIVE | k8sinternal=192.168.45.47, 192.168.101.51 | | g1t1.large | | 3fa5258d-2ec6-4980-a7f3-fb343db982fb | testnew-wxdecq7kb4p5-node-1 | ACTIVE | k8sinternal=192.168.45.51, 192.168.101.65 | | g1t1.large | | 344507e6-f733-4998-9fad-0908a658cdf4 | testnew-wxdecq7kb4p5-node-3 | SHUTOFF | k8sinternal=192.168.45.48, 192.168.101.57 | | g1t1.large | | d8603d5f-a495-466b-b4a0-f197b0377590 | testnew-wxdecq7kb4p5-node-2 | ACTIVE | k8sinternal=192.168.45.49, 192.168.101.64 | | g1t1.large | | 7fc31c4e-ad7c-4f5f-bba9-a850c9a5c144 | testnew-wxdecq7kb4p5-master-0 | ACTIVE | k8sinternal=192.168.45.46, 192.168.101.50 | | g1t1.large | root@maniltest:~# ------------------------------------------------------------------------------------------------------------------- I0504 08:17:48.239766 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:17:48.239809 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 12.235µs I0504 08:19:48.240000 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:19:48.240405 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 17.217µs I0504 08:21:48.240894 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:21:48.241093 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 47.07µs I0504 08:23:48.241364 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:23:48.241408 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 11.781µs I0504 08:25:48.241592 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:25:48.242042 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 12.116µs I0504 08:27:48.242311 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:27:48.242506 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 15.12µs I0504 08:29:48.242705 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:29:48.242936 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 12.089µs I0504 08:31:48.243129 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:31:48.243600 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 11.499µs I0504 08:33:48.243797 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:33:48.243991 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 12.687µs I0504 08:35:48.244181 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:35:48.244363 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 12.686µs I0504 08:37:48.244565 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:37:48.244783 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 12.946µs I0504 08:39:48.245114 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:39:48.245180 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 13.118µs I0504 08:41:48.245382 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:41:48.245743 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 13.474µs I0504 08:43:48.245956 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:43:48.245998 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 12.055µs I0504 08:45:48.246208 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:45:48.246459 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 14.325µs I0504 08:45:54.594580 1 static_autoscaler.go:559] Decreasing size of default-worker, expected=4 current=3 delta=-1 I0504 08:45:54.594630 1 magnum_nodegroup.go:255] Decreasing target size by -1, 4->3 I0504 08:45:56.872199 1 static_autoscaler.go:342] Some node group target size was fixed, skipping the iteration I0504 08:47:48.247550 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I0504 08:47:48.247855 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 73.5µs Regards, Sriramu Desingh