[juju 3.1.1] kubernetes-worker failed: "aws-relation-joined" hook, unauthorized

[Bas de Bruijne]

In testrun https://solutions.qa.canonical.com/v2/testruns/7dadff54-c601-4f0c-a9fb-7e9f8fd0ebc9, which is ck8s 1.26 on aws with juju 3.1 candidate (3.1.1), the k8s-worker fails in the aws-relation-joined hook:

-------------------------
2023-03-17 13:31:09 INFO unit.kubernetes-worker/1.juju-log server.go:325 aws:10: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kube-api-endpoint
2023-03-17 13:31:09 INFO unit.kubernetes-worker/1.juju-log server.go:325 aws:10: Invoking reactive handler: hooks/relations/http/provides.py:15:broken:ingress-proxy
2023-03-17 13:31:09 INFO unit.kubernetes-worker/1.juju-log server.go:325 aws:10: Invoking reactive handler: hooks/relations/aws-integration/requires.py:91:send_instance_info:aws
2023-03-17 13:31:09 ERROR unit.kubernetes-worker/1.juju-log server.go:325 aws:10: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kubernetes-worker-1/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-kubernetes-worker-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-kubernetes-worker-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-kubernetes-worker-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-kubernetes-worker-1/charm/hooks/relations/aws-integration/requires.py", line 93, in send_instance_info
    self._to_publish['instance-id'] = self.instance_id
  File "/var/lib/juju/agents/unit-kubernetes-worker-1/charm/hooks/relations/aws-integration/requires.py", line 121, in instance_id
    with urlopen(self._instance_id_url) as fd:
  File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized
-------------------------

Further analysis will follow.

Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/7dadff54-c601-4f0c-a9fb-7e9f8fd0ebc9/index.html

[George Kraft (cynerva)]

The failing HTTP call is in interface-aws-integration[1], which is trying to reach the AWS instance metadata service[2].

It looks like interface-aws-integration is trying to use IMDSv1, but a change in Juju[3] for 3.1.1 makes it so that AWS instances created by Juju are no longer allowed to use IMDSv1. We will need to update interface-aws-integration to support IMDSv2.

[1]: https://github.com/juju-solutions/interface-aws-integration/blob/d8d8c7ef17c99ad53383f3cabf4cf5c8191d16f7/requires.py#L121
[2]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
[3]: https://github.com/juju/juju/pull/15166

[Ian Booth (wallyworld)]

Yeah, this was a necessary security related change.

https://bugs.launchpad.net/bugs/1960568

[Konstantinos Kaskavelis (kaskavel)]

Is the support for IMDSv2 something we should expect for 3.1.1 release testing?
This issue currently blocks 50% of our testing for this release.

[George Kraft (cynerva)]

The Kubernetes charms don't have official support for Juju 3.x yet. Support for Juju 3.x will be added with the release of Charmed Kubernetes 1.27, targeted for release sometime between April 11-18. We will make sure to add support for IMDSv2 to the charms as part of that release.

[Thomas Miller (tlmiller)]

Thanks guys, We have tested this from the Juju side this morning and it is indeed the interface charm failing to get the instance id from the client.

It's a relatively easy fix as per: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-metadata-v2-how-it-works.html

I have left a diff of what I think the changes should be here: https://github.com/juju-solutions/interface-aws-integration/issues/13

[George Kraft (cynerva)]

Thanks Thomas.

SQA folks, would it unblock you if we landed the fix in an edge build of the kubernetes charms?

[Alexander Balderson (asbalderson)]

Hi George,

An edge charm would work perfectly fine for us, we can use a 1.27/edge charm with 1.26 k8s no problem as well

[George Kraft (cynerva)]

Cool, I'll see if I can get this going then.

[George Kraft (cynerva)]

PR: https://github.com/juju-solutions/interface-aws-integration/pull/14