Ubuntu EKS AMIs 1.21-1.23 do not have AWS_REGION in the initial kubeconfig file

Bug #2003926 reported by Jiaping Zeng
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fix Released
Thomas Bechtold

Bug Description


For Ubuntu EKS AMIs 1.21-1.23, the initial /var/lib/kubelet/kubeconfig file does not contain the '--region' flag and the initial 'AWS_REGION' value. This 'AWS_REGION' value is used by EKS Bootstrap script at /etc/eks/bootstrap.sh to set the correct regional endpoint: [1], which uses the linked sed command to replace 'AWS_REGION' with the correct local region.

For comparison between version 1.23 and 1.24,

ubuntu@ip-xxx:~$ cat /var/lib/kubelet/kubeconfig
        - "token"
        - "-i"
        - "CLUSTER_NAME"
        - --region
        - "AWS_REGION"

1.23 (1.21 and 1.22 are also missing the region flag):
ubuntu@ip-xxx:~$ cat /var/lib/kubelet/kubeconfig
        - eks
        - get-token
        - --cluster-name
        - "CLUSTER_NAME"

For most regions, missing the region flag does not cause any issue as the default global endpoint is used by default (albeit slightly slower performance by not using the local endpoint). However, when these AMIs are used in China region, authenticate calls fail due to the global endpoint not being accessible in the region. This then causes new nodes to not be able to join their cluster, which can be worked around by manually adding the region information in kubeconfig and restarting kubelet.

Could the AMIs be updated to include the --region AWS_REGION flag in kubeconfig as is done in 1.24?

Please let me know if any additional information is needed from AWS side. Thank you.

[1] https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh#L343

Tested AWS AMI IDs for reference:

1.21: ami-0d8500d0848382782
1.22: ami-00259561cd0a03d14
1.23: ami-005efc21ad9dd856f
1.24 (working): ami-0bd12c633e73201a0

Revision history for this message
Robby Pocase (rpocase) wrote :

Thanks for the report! We will start working this ASAP and except to have new AMIs available within the next week or so.

We currently plan to scope this to EKS 1.22 - 1.23 since Amazon EKS 1.21 support ends on February 15th [1]. Please let us know if there are any major concerns here.

[1] https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar

Robby Pocase (rpocase)
Changed in cloud-images:
importance: Undecided → High
Changed in cloud-images:
status: New → In Progress
assignee: nobody → Thomas Bechtold (toabctl)
Revision history for this message
Thomas Bechtold (toabctl) wrote :

The latest images (serial 20230131) do contain the fix. please let us know if you still see problems.

Changed in cloud-images:
status: In Progress → Fix Released
Revision history for this message
Jiaping Zeng (jpzaws) wrote :

Thanks for the quick fix! I have verified from my side that the updated 1.21-1.23 AMIs now set the region correctly.

Tested AMIs:
1.21: ami-0ffbf5890c10d2ebc
1.22: ami-00898231a9ed08b01
1.23: ami-02f9fe43b6941564b

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.