A reanimated cloud account leaves the client in limbo

[Peter Matulis]

When I deactivate an AWS (IAM) user and then make it active again Juju is not able to make use of that account. Restarting the Juju client host doesn't help. I attached logs from the controller model. There are some errors in there.

$ juju bootstrap --credential jlaurin aws aws

Creating Juju controller "aws" on aws/us-east-1
Looking for packaged Juju agent version 2.5.3 for amd64
Launching controller instance(s) on aws/us-east-1...
 - i-0f6fcc7977733e8ac (arch=amd64 mem=4G cores=2)us-east-1a"
Installing Juju agent on bootstrap instance
Fetching Juju GUI 2.14.0
Waiting for address
Attempting to connect to 54.90.164.210:22
Attempting to connect to 172.31.17.219:22
Connected to 54.90.164.210
Running machine configuration script...
Bootstrap agent now started
Contacting Juju controller at 54.90.164.210 to verify accessibility...
Bootstrap complete, "aws" controller now available
Controller machines are in the "controller" model
Initial model "default" added

<< DEACTIVATE jlaurin IN AWS CONSOLE >>

$ juju add-machine

failed to create 1 machine
ERROR cannot add a new machine:
The provided credentials could not be validated and
may not be authorized to carry out the request.
Ensure that your account is authorized to use the Amazon EC2 service and
that you are using the correct access keys.
These keys are obtained via the "Security Credentials"
page in the AWS console.
: AWS was not able to validate the provided access credentials (AuthFailure)

<< REACTIVATE jlaurin IN AWS CONSOLE >>

$ juju add-machine

created machine 0

$ juju status

Model Controller Cloud/Region Version SLA Timestamp
default aws aws/us-east-1 2.5.3 unsupported 15:19:29Z

Machine State DNS Inst id Series AZ Message
0 pending pending bionic

$ juju debug-log -m controller --replay > controller.log

[Anastasia (anastasia-macmood)]

This is a normal behavior - Juju will not use the same credential once it is marked as invalid.

You have enabled and disabled and re-enabled credential in a cloud console. Juju paused. But you did not tell Juju that the credential is ok again. This is mostly the result of your testing scenario. Under normal circumstance, in a natural flow of events, the details of a credential will change, so you need to tell Juju to mark that credential as valid. Please use 'update-credential' after re-activating it.

[Peter Matulis (petermatulis)]

I feel the user is left adrift here.

> you did not tell Juju that the credential is ok again

I did not tell Juju that the credential was *not* ok.

Juju performed an internal action so the onus is on Juju to guide the user somehow ("This credential has been marked as invalid. To use it again, ensure the corresponding cloud account is active and your credential is valid. Then use the `update-credential` command").

Anyway, the command output suggests everything is working as expected, but that's not what happens. The machine stays in a pending state forever.

https://paste.ubuntu.com/p/Y3prD7rdNd/

I can confirm that this gets things working again:

juju update-credential aws jlaurin

[Anastasia (anastasia-macmood)]

Yes, Juju will internally figure out that a credential is invalid. Once this happens, Juju will not try this credential again unless a user tells it that the credential is ok to try again. This happens via 'update-credential' command. I agree that messaging could be better and I feel that it is covered by the bug you've previously filed - https://bugs.launchpad.net/bugs/1822117

[John A Meinel (jameinel)]

I agree that there should be a message on the pending machine about "no
credential available" or "credential suspended due to invalid responses",
or something along those lines.
It seems that when the provisioner goes to create a new instance, it can
see why it is unable to do so, and report that to the user. It may not give
them the immediate understanding that 'update-credential' is the way
forward, but it would at least give them an idea that the machine isn't
just slow to provision.

On Tue, Apr 2, 2019 at 3:40 AM Peter Matulis <email address hidden>
wrote:

> I feel the user is left adrift here.
>
> > you did not tell Juju that the credential is ok again
>
> I did not tell Juju that the credential was *not* ok.
>
> Juju performed an internal action so the onus is on Juju to guide the
> user somehow ("This credential has been marked as invalid. To use it
> again, ensure the corresponding cloud account is active and your
> credential is valid. Then use the `update-credential` command").
>
> Anyway, the command output suggests everything is working as expected,
> but that's not what happens. The machine stays in a pending state
> forever.
>
> https://paste.ubuntu.com/p/Y3prD7rdNd/
>
> I can confirm that this gets things working again:
>
> juju update-credential aws jlaurin
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1822637
>
> Title:
> Operations no longer possible with a reanimated account on AWS
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1822637/+subscriptions
>