Can't bootstrap controller on AWS using instance roles

[Diko Parvanov]

Trying to bootstrap a controller on AWS with personal credentials (using my access-key and secert-key) following the documentation here: https://juju.is/docs/juju/amazon-ec2 the bootstrap fails with the following error, although I am an administrator with full access:

juju bootstrap aws
ERROR
The provided credentials could not be validated and
may not be authorized to carry out the request.
Ensure that your account is authorized to use the Amazon EC2 service and
that you are using the correct access keys.
These keys are obtained via the "Security Credentials"
page in the AWS console: operation error EC2: DescribeAccountAttributes, https response error StatusCode: 401, RequestID: af2496c5-d990-44b5-98b4-2cbf454ae70c, api error AuthFailure: AWS was not able to validate the provided access credentials

I am running this command from my workstation, outside of AWS. Same happens if I run with with an IAM role with full admin privileges in (on EC2 instance) or outside of AWS.

Any version of juju (2.9, 3.1, 3.3)

[Joseph Phillips (manadart)]

Is there anything else other than the access key and secret key in your local ~/.aws/credentials?

Does it work if you try a different region?

[Diko Parvanov (dparv)]

It doesn't work in any region. In ~/.aws/credentials there is nothing - I am using the juju credentials added with add-credential.

[John A Meinel (jameinel)]

Talking with Diko, this is trying to use the managed role from AWS (as provided from the metadata service and an assigned role to the instance).

I believe what is needed is to use `--bootstrap-constraints "instance-role=ROLEID"`
We should make sure this is clear in our documentation.

[John A Meinel (jameinel)]

The attempt to get SECRET_KEY and ACCESS_KEY was using:

ROLE_NAME=XXXX
METADATA_URL="http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE_NAME"
AWS_ACCESS_KEY_ID=$(curl -s $METADATA_URL | jq -r .AccessKeyId)
AWS_SECRET_KEY_ID=$(curl -s $METADATA_URL | jq -r .SecretAccessKey)
AWS_SESSION_TOKEN=$(curl -s $METADATA_URL | jq -r .Token)
REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)
mkdir -p ~/.local/share
juju bootstrap aws

I don't think juju pays attention to AWS_SESSION_TOKEN without --constraints "instance-role=XXXX" being set (either to 'auto' or to an explicit name)

[Diko Parvanov (dparv)]

Following on the autoload-credentials help prompt:

EC2
  Credentials and regions:
    1. On Linux, $HOME/.aws/credentials and $HOME/.aws/config
    2. Environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

So.. this is what happens why getting those credentials (changed for random chars for this paste ofc):

ubuntu@ip-172-31-85-11:~$ echo $AWS_ACCESS_KEY_ID
ASI14A2UC3BH25a62E7N3LB
ubuntu@ip-172-31-85-11:~$ echo $AWS_SECRET_ACCESS_KEY
QPuWjF02gqVfa2jOx4TmkTOSP81uRaYVCORpf430hLhU5
ubuntu@ip-172-31-85-11:~$ juju autoload-credentials aws
This operation can be applied to both a copy on this client and to the one on a controller.
No current controller was detected and there are no registered controllers on this client: either bootstrap one or register one.

Looking for cloud and credential information on local client...
No cloud credentials found.

I tried using the --constraints or --bootstrap-constraints with instance-role=XXX neither works for me.

Btw. this doesn't work either if you have the credentials in ~/.aws/credentials

[Diko Parvanov (dparv)]

Adding the credentials manually with juju add-credential aws and trying to bootstrap:

ubuntu@ip-172-31-85-11:~$ juju bootstrap aws --bootstrap-constraints instance-role=EC2Role
ERROR
The provided credentials could not be validated and
may not be authorized to carry out the request.
Ensure that your account is authorized to use the Amazon EC2 service and
that you are using the correct access keys.
These keys are obtained via the "Security Credentials"
page in the AWS console: operation error EC2: DescribeAccountAttributes, https response error StatusCode: 401, RequestID: 9dbbde87-8649-4a64-a054-6ed09dc7bbfa, api error AuthFailure: AWS was not able to validate the provided access credentials

ubuntu@ip-172-31-85-11:~$ juju bootstrap aws --constraints instance-role=EC2Role
ERROR
The provided credentials could not be validated and
may not be authorized to carry out the request.
Ensure that your account is authorized to use the Amazon EC2 service and
that you are using the correct access keys.
These keys are obtained via the "Security Credentials"
page in the AWS console: operation error EC2: DescribeAccountAttributes, https response error StatusCode: 401, RequestID: 239fb59e-433b-405f-a19d-d52d54edaf99, api error AuthFailure: AWS was not able to validate the provided access credentials

ubuntu@ip-172-31-85-11:~$ juju bootstrap aws --bootstrap-constraints instance-role=auto
ERROR
The provided credentials could not be validated and
may not be authorized to carry out the request.
Ensure that your account is authorized to use the Amazon EC2 service and
that you are using the correct access keys.
These keys are obtained via the "Security Credentials"
page in the AWS console: operation error EC2: DescribeAccountAttributes, https response error StatusCode: 401, RequestID: 62725b29-ec61-49c9-8bb7-2ed117f9962d, api error AuthFailure: AWS was not able to validate the provided access credentials

[Simon Richardson (simonrichardson)]

The source of the error from the Juju side comes from[1]. At the very least we should improve the logging to improve diagnosing the root cause.

 1. https://github.com/juju/juju/blob/501fcd8100874ed52072c9e7248c6af6f156f92e/provider/ec2/provider.go#L174-L201

[John A Meinel (jameinel)]

I wasn't able to find a CI test that we have to confirm that we are working correctly with AWS Roles.

Reading the error message, this looks like we are failing while trying to check whether the supplied role can do the things we need. (Which might be an optional role which we could progress without.)

Either way, this is functionality that we do want to support, and we should make sure that we know it works on our end.

[Thomas Miller (tlmiller)]

Thank you for letting me play catch up on this bug.

There is a lot of different information above all of which is in conflict with each other. Hopefully the below can sort out some of the information.

If you are bootstrapping a Juju controller to AWS the AWS credentials located on the client where the juju cli command is being run for is what is used to create all the infrastructure needed for the controller run. We only support AWS secret key credentials at the moment from client. This means that if you are trying to bootstrap off of a jump host that has a role attached to it we don't support this. We don't support assuming a role from the client either.

The --bootstrap-constraints instance-role tells the client to bootstrap a controller that once established uses the credentials tied to the ec2 machine to interact with AWS. i.e once the controller is running and communication the controller needs to perform with cloud is done via the EC2 role attached. This is instead of your local AWS credentials on the client being uploaded to the controller.

As this looks to be just an issue isolated with bootstrapping and not the running controller are you able to confirm if you are using secret key credentials as there is conflicting information above. Are you also able to confirm if you client keys don't need to also assume a role to get their required permissions needed?

Ta
tlm

[Diko Parvanov (dparv)]

Based on that I need both:
1. personal/account AWS credentials
2. IAM role that has access

It really doesn't make sense to be using the IAM Role in this scenario. Juju can just build whatever resources it wants with the personal/account AWS credentials provided, so why do we need the IAM role in the first place?

The whole idea of an IAM role is to be able to automate things (e.g. deployments) without the need for any user input for credentials.

[Thomas Miller (tlmiller)]

Hey Diko,

I am still not sure we are on the same page at the moment. IAM Roles are very useful in a controller context as it allows a controller to use credentials from the IAM metadata service and assume a roll to perform provisioning tasks (automation) on behalf of a Juju deployments.

We don't support assuming roles from the CLI yet. But a common design pattern for multiple AWS accounts is to have one central account holding all of the IAM users and allowing them to assume a role into the other org accounts to get the needed permissions etc. This is a case we don't yet support with the Juju cli.

To track back to this bug. If we just run a vanilla `juju bootstrap aws` using local credentials with a secret key from the IAM console we are seeing the bad credentials error message pop up? In this case what w e are poorly trying to convey is that AWS have rejected the credentials the user has provided to Juju.

Assuming the above is correct we need to dig into the AWS side of things and figure out what about the credentials AWS is rejecting. I have seen AWS IAM access policies that restrict access keys being used if there is no MFA information provided. Have you put these keys into the aws cli and see if you are able to describe all ec2 machines for a region?

[Diko Parvanov (dparv)]

So with credentials from the UI it works just fine. It didn't work with the access key and secret that I got for the IAM role.

However, I seem to have worked around this via the following workflow inside the EC2 instance, that has the IAM role assigned:

aws iam create-user --user-name juju-bootstrap
aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess --user-name juju-bootstrap
ACCESS_KEY=$(aws iam create-access-key --user-name juju-bootstrap)
AWS_ACCESS_KEY_ID=$(echo $ACCESS_KEY | jq -r '.AccessKey.AccessKeyId')
AWS_SECRET_ACCESS_KEY=$(echo $ACCESS_KEY | jq -r '.AccessKey.SecretAccessKey')
REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)
CONTROLLER_NAME=aws-$REGION
cat > /home/ubuntu/.aws/credentials <<EOL
[default]
aws_access_key_id = $AWS_ACCESS_KEY_ID
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
EOL
juju default-region aws $REGION
juju bootstrap aws

[Thomas Miller (tlmiller)]

Thanks Diko,

Interesting work around. Obviously not ideal with what you are trying to achieve. Going to consider this a feature request at the moment.

The basic workflow would be to get the AWS go client to load credentials from the env and use these for bootstrap. The juju client would need to make the controller with an instance role as we are unable to transfer credentials in this setup.

[Peter Jose De Sousa (pjds)]

Hitting this issue now and cannot bootstrap from my workstation. Tried my restricted IAM juju user, refreshing my secrets, and then finally tried my root account. All of the above hit the error provided by Diko.

[Peter Jose De Sousa (pjds)]

I was able to workaround this by supplying --bootstrap-constraints.