| 2024-01-17 11:21:19 |
Thomas Bechtold |
description |
Hello everyone,
I'm encountering a problem when using the newest Ubuntu AMI EKS optimized. Specifically, I'm facing issues with EKS Cluster joining.
I attempted the workaround suggested in this link: https://bugs.launchpad.net/cloud-images/+bug/2045791, but unfortunately, it didn't resolve the problem.
AMIs causing the issue:
ami-0f3ea2eb3faa6e2b6 ubuntu-eks/k8s_1.26/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20231213.1
ami-004b4213ca29ada16 ubuntu-eks/k8s_1.26/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20231105
ami-0984af2dedae97f46 ubuntu-eks/k8s_1.27/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20231213.1
ami-0fde44ab1d7b005e8 ubuntu-eks/k8s_1.27/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20231117
These AMIs are working well:
ami-074c0d8d07da7f245 ubuntu-eks/k8s_1.26/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20230616
ami-0ccf557a5464c4733 ubuntu-eks/k8s_1.27/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20230714
I'm encountering the following error in the EKS NodeGroup log: "NodeCreationFailure Couldn't proceed with the upgrade process as new nodes are not joining the node group."
I attempted to run bootstrap.sh manually, but it seems to be stuck:
/etc/eks/bootstrap.sh bd-int-eks-cluster --kubelet-extra-args '--max-pods=110' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP --use-max-pods false
Using containerd as the container runtime
Aliasing EKS k8s snap commands
Stopping k8s daemons until configured
Stopped.
Cluster "kubernetes" set.
2024-01-17 09:33:27,092:__main__:INFO:Setting kubelet-eks key >cluster-dns< to >10.100.0.10<
2024-01-17 09:33:27,107:__main__:INFO:received '202/Accepted' from snapd for PUT on /v2/snaps/kubelet-eks/conf (change-id: 19)
Cloud-init-output.log:
Using containerd as the container runtime
Aliasing EKS k8s snap commands
Added:
- kubelet-eks.kubelet as kubelet
Added:
- kubectl-eks.kubectl as kubectl
Stopping k8s daemons until configured
Stopped.
Cluster "kubernetes" set.
2024-01-17 09:04:27,223:__main__:INFO:Setting kubelet-eks key >cluster-dns< to >10.100.0.10<
2024-01-17 09:04:27,239:__main__:INFO:received '202/Accepted' from snapd for PUT on /v2/snaps/kubelet-eks/conf (change-id: 6)
2024-01-17 09:08:27,296:__main__:ERROR:timeout while waiting for in-progress changes
2024-01-17 09:08:27,297:__main__:INFO:result for change: {'id': '6', 'kind': 'configure-snap', 'summary': 'Change configuration of "kubelet-eks" snap', 'status': 'Done', 'tasks': [{'id': '128', 'kind': 'run-hook', 'summary': 'Run configure hook of "kubelet-eks" snap', 'status': 'Done', 'progress': {'label': '', 'done': 1, 'total': 1}, 'spawn-time': '2024-01-17T09:04:27.224751922Z', 'ready-time': '2024-01-17T09:04:29.988173989Z'}], 'ready': True, 'spawn-time': '2024-01-17T09:04:27.224784346Z', 'ready-time': '2024-01-17T09:04:29.988177399Z'}
Container runtime is containerd total: 290.8 (484.0 KiB/s)
2024-01-17 09:08:31,174:__main__:INFO:Setting kubelet-eks key >container-runtime< to >remote<
2024-01-17 09:08:31,196:__main__:INFO:received '202/Accepted' from snapd for PUT on /v2/snaps/kubelet-eks/conf (change-id: 7)
2024-01-17 09:12:31,296:__main__:ERROR:timeout while waiting for in-progress changes
2024-01-17 09:12:31,298:__main__:INFO:result for change: {'id': '7', 'kind': 'configure-snap', 'summary': 'Change configuration of "kubelet-eks" snap', 'status': 'Done', 'tasks': [{'id': '129', 'kind': 'run-hook', 'summary': 'Run configure hook of "kubelet-eks" snap', 'status': 'Done', 'progress': {'label': '', 'done': 1, 'total': 1}, 'spawn-time': '2024-01-17T09:08:31.176039532Z', 'ready-time': '2024-01-17T09:08:33.89372478Z'}], 'ready': True, 'spawn-time': '2024-01-17T09:08:31.176065403Z', 'ready-time': '2024-01-17T09:08:33.89372832Z'}
2024-01-17 09:12:31,615:__main__:INFO:Setting kubelet-eks key >container-runtime-endpoint< to >unix:///run/containerd/containerd.sock<
2024-01-17 09:12:31,652:__main__:INFO:received '202/Accepted' from snapd for PUT on /v2/snaps/kubelet-eks/conf (change-id: 8)
2024-01-17 09:16:31,680:__main__:ERROR:timeout while waiting for in-progress changes
2024-01-17 09:16:31,688:__main__:INFO:result for change: {'id': '8', 'kind': 'configure-snap', 'summary': 'Change configuration of "kubelet-eks" snap', 'status': 'Done', 'tasks': [{'id': '130', 'kind': 'run-hook', 'summary': 'Run configure hook of "kubelet-eks" snap', 'status': 'Done', 'progress': {'label': '', 'done': 1, 'total': 1}, 'spawn-time': '2024-01-17T09:12:31.622651724Z', 'ready-time': '2024-01-17T09:12:35.823591143Z'}], 'ready': True, 'spawn-time': '2024-01-17T09:12:31.622687105Z', 'ready-time': '2024-01-17T09:12:35.823593913Z'}
cloud-provider is external
2024-01-17 09:16:35,553:__main__:INFO:Setting kubelet-eks key >hostname-override< to >ip-11-243-100-38.eu-west-1.compute.internal<
2024-01-17 09:16:35,594:__main__:INFO:received '202/Accepted' from snapd for PUT on /v2/snaps/kubelet-eks/conf (change-id: 9)
2024-01-17 09:20:35,616:__main__:ERROR:timeout while waiting for in-progress changes
2024-01-17 09:20:35,618:__main__:INFO:result for change: {'id': '9', 'kind': 'configure-snap', 'summary': 'Change configuration of "kubelet-eks" snap', 'status': 'Done', 'tasks': [{'id': '131', 'kind': 'run-hook', 'summary': 'Run configure hook of "kubelet-eks" snap', 'status': 'Done', 'progress': {'label': '', 'done': 1, 'total': 1}, 'spawn-time': '2024-01-17T09:16:35.57120716Z', 'ready-time': '2024-01-17T09:16:41.834718414Z'}], 'ready': True, 'spawn-time': '2024-01-17T09:16:35.571229331Z', 'ready-time': '2024-01-17T09:16:41.834721155Z'}
2024-01-17 09:20:35,820:__main__:INFO:Setting kubelet-eks key >image-credential-provider-config< to >/etc/eks/ecr-credential-provider/config.json<
2024-01-17 09:20:35,850:__main__:INFO:received '202/Accepted' from snapd for PUT on /v2/snaps/kubelet-eks/conf (change-id: 10)
Any assistance or suggestions would be greatly appreciated.
Thank you! |
Hello everyone,
I'm encountering a problem when using the newest Ubuntu AMI EKS optimized. Specifically, I'm facing issues with EKS Cluster joining.
I attempted the workaround suggested in this link: https://bugs.launchpad.net/cloud-images/+bug/2045791, but unfortunately, it didn't resolve the problem.
AMIs causing the issue:
ami-0f3ea2eb3faa6e2b6 ubuntu-eks/k8s_1.26/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20231213.1
ami-004b4213ca29ada16 ubuntu-eks/k8s_1.26/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20231105
ami-0984af2dedae97f46 ubuntu-eks/k8s_1.27/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20231213.1
ami-0fde44ab1d7b005e8 ubuntu-eks/k8s_1.27/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20231117
These AMIs are working well:
ami-074c0d8d07da7f245 ubuntu-eks/k8s_1.26/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20230616
ami-0ccf557a5464c4733 ubuntu-eks/k8s_1.27/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20230714
I'm encountering the following error in the EKS NodeGroup log: "NodeCreationFailure Couldn't proceed with the upgrade process as new nodes are not joining the node group."
I attempted to run bootstrap.sh manually, but it seems to be stuck:
/etc/eks/bootstrap.sh bd-int-eks-cluster --kubelet-extra-args '--max-pods=110' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP --use-max-pods false
Using containerd as the container runtime
Aliasing EKS k8s snap commands
Stopping k8s daemons until configured
Stopped.
Cluster "kubernetes" set.
2024-01-17 09:33:27,092:__main__:INFO:Setting kubelet-eks key >cluster-dns< to >10.100.0.10<
2024-01-17 09:33:27,107:__main__:INFO:received '202/Accepted' from snapd for PUT on /v2/snaps/kubelet-eks/conf (change-id: 19)
Cloud-init-output.log:
Using containerd as the container runtime
Aliasing EKS k8s snap commands
Added:
- kubelet-eks.kubelet as kubelet
Added:
- kubectl-eks.kubectl as kubectl
Stopping k8s daemons until configured
Stopped.
Cluster "kubernetes" set.
2024-01-17 09:04:27,223:__main__:INFO:Setting kubelet-eks key >cluster-dns< to >10.100.0.10<
2024-01-17 09:04:27,239:__main__:INFO:received '202/Accepted' from snapd for PUT on /v2/snaps/kubelet-eks/conf (change-id: 6)
2024-01-17 09:08:27,296:__main__:ERROR:timeout while waiting for in-progress changes
2024-01-17 09:08:27,297:__main__:INFO:result for change: {'id': '6', 'kind': 'configure-snap', 'summary': 'Change configuration of "kubelet-eks" snap', 'status': 'Done', 'tasks': [{'id': '128', 'kind': 'run-hook', 'summary': 'Run configure hook of "kubelet-eks" snap', 'status': 'Done', 'progress': {'label': '', 'done': 1, 'total': 1}, 'spawn-time': '2024-01-17T09:04:27.224751922Z', 'ready-time': '2024-01-17T09:04:29.988173989Z'}], 'ready': True, 'spawn-time': '2024-01-17T09:04:27.224784346Z', 'ready-time': '2024-01-17T09:04:29.988177399Z'}
Container runtime is containerd total: 290.8 (484.0 KiB/s)
2024-01-17 09:08:31,174:__main__:INFO:Setting kubelet-eks key >container-runtime< to >remote<
2024-01-17 09:08:31,196:__main__:INFO:received '202/Accepted' from snapd for PUT on /v2/snaps/kubelet-eks/conf (change-id: 7)
2024-01-17 09:12:31,296:__main__:ERROR:timeout while waiting for in-progress changes
2024-01-17 09:12:31,298:__main__:INFO:result for change: {'id': '7', 'kind': 'configure-snap', 'summary': 'Change configuration of "kubelet-eks" snap', 'status': 'Done', 'tasks': [{'id': '129', 'kind': 'run-hook', 'summary': 'Run configure hook of "kubelet-eks" snap', 'status': 'Done', 'progress': {'label': '', 'done': 1, 'total': 1}, 'spawn-time': '2024-01-17T09:08:31.176039532Z', 'ready-time': '2024-01-17T09:08:33.89372478Z'}], 'ready': True, 'spawn-time': '2024-01-17T09:08:31.176065403Z', 'ready-time': '2024-01-17T09:08:33.89372832Z'}
2024-01-17 09:12:31,615:__main__:INFO:Setting kubelet-eks key >container-runtime-endpoint< to >unix:///run/containerd/containerd.sock<
2024-01-17 09:12:31,652:__main__:INFO:received '202/Accepted' from snapd for PUT on /v2/snaps/kubelet-eks/conf (change-id: 8)
2024-01-17 09:16:31,680:__main__:ERROR:timeout while waiting for in-progress changes
2024-01-17 09:16:31,688:__main__:INFO:result for change: {'id': '8', 'kind': 'configure-snap', 'summary': 'Change configuration of "kubelet-eks" snap', 'status': 'Done', 'tasks': [{'id': '130', 'kind': 'run-hook', 'summary': 'Run configure hook of "kubelet-eks" snap', 'status': 'Done', 'progress': {'label': '', 'done': 1, 'total': 1}, 'spawn-time': '2024-01-17T09:12:31.622651724Z', 'ready-time': '2024-01-17T09:12:35.823591143Z'}], 'ready': True, 'spawn-time': '2024-01-17T09:12:31.622687105Z', 'ready-time': '2024-01-17T09:12:35.823593913Z'}
cloud-provider is external
2024-01-17 09:16:35,553:__main__:INFO:Setting kubelet-eks key >hostname-override< to >ip-11-243-100-38.eu-west-1.compute.internal<
2024-01-17 09:16:35,594:__main__:INFO:received '202/Accepted' from snapd for PUT on /v2/snaps/kubelet-eks/conf (change-id: 9)
2024-01-17 09:20:35,616:__main__:ERROR:timeout while waiting for in-progress changes
2024-01-17 09:20:35,618:__main__:INFO:result for change: {'id': '9', 'kind': 'configure-snap', 'summary': 'Change configuration of "kubelet-eks" snap', 'status': 'Done', 'tasks': [{'id': '131', 'kind': 'run-hook', 'summary': 'Run configure hook of "kubelet-eks" snap', 'status': 'Done', 'progress': {'label': '', 'done': 1, 'total': 1}, 'spawn-time': '2024-01-17T09:16:35.57120716Z', 'ready-time': '2024-01-17T09:16:41.834718414Z'}], 'ready': True, 'spawn-time': '2024-01-17T09:16:35.571229331Z', 'ready-time': '2024-01-17T09:16:41.834721155Z'}
2024-01-17 09:20:35,820:__main__:INFO:Setting kubelet-eks key >image-credential-provider-config< to >/etc/eks/ecr-credential-provider/config.json<
2024-01-17 09:20:35,850:__main__:INFO:received '202/Accepted' from snapd for PUT on /v2/snaps/kubelet-eks/conf (change-id: 10)
Any assistance or suggestions would be greatly appreciated.
Thank you!
---
External link: https://warthogs.atlassian.net/browse/CPC-3731 |
|
