Ubuntu EKS AMIs 1.21-1.23 do not have AWS_REGION in the initial kubeconfig file

[Jiaping Zeng]

Hello,

For Ubuntu EKS AMIs 1.21-1.23, the initial /var/lib/kubelet/kubeconfig file does not contain the '--region' flag and the initial 'AWS_REGION' value. This 'AWS_REGION' value is used by EKS Bootstrap script at /etc/eks/bootstrap.sh to set the correct regional endpoint: [1], which uses the linked sed command to replace 'AWS_REGION' with the correct local region.

For comparison between version 1.23 and 1.24,

1.24:
ubuntu@ip-xxx:~$ cat /var/lib/kubelet/kubeconfig
...
      args:
        - "token"
        - "-i"
        - "CLUSTER_NAME"
        - --region
        - "AWS_REGION"

1.23 (1.21 and 1.22 are also missing the region flag):
ubuntu@ip-xxx:~$ cat /var/lib/kubelet/kubeconfig
...
      args:
        - eks
        - get-token
        - --cluster-name
        - "CLUSTER_NAME"

For most regions, missing the region flag does not cause any issue as the default global endpoint is used by default (albeit slightly slower performance by not using the local endpoint). However, when these AMIs are used in China region, authenticate calls fail due to the global endpoint not being accessible in the region. This then causes new nodes to not be able to join their cluster, which can be worked around by manually adding the region information in kubeconfig and restarting kubelet.

Could the AMIs be updated to include the --region AWS_REGION flag in kubeconfig as is done in 1.24?

Please let me know if any additional information is needed from AWS side. Thank you.

[1] https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh#L343

Tested AWS AMI IDs for reference:

1.21: ami-0d8500d0848382782
1.22: ami-00259561cd0a03d14
1.23: ami-005efc21ad9dd856f
1.24 (working): ami-0bd12c633e73201a0

[Robby Pocase (rpocase)]

Thanks for the report! We will start working this ASAP and except to have new AMIs available within the next week or so.

We currently plan to scope this to EKS 1.22 - 1.23 since Amazon EKS 1.21 support ends on February 15th [1]. Please let us know if there are any major concerns here.

[1] https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar

[Thomas Bechtold (toabctl)]

The latest images (serial 20230131) do contain the fix. please let us know if you still see problems.

[Jiaping Zeng (jpzaws)]

Thanks for the quick fix! I have verified from my side that the updated 1.21-1.23 AMIs now set the region correctly.

Tested AMIs:
1.21: ami-0ffbf5890c10d2ebc
1.22: ami-00898231a9ed08b01
1.23: ami-02f9fe43b6941564b