Bad cloud credentials prevent model destruction

[Junien F]

Hi,

We have a model with 0 machine in it (openstack provider), in the "destroying" status. It's using an openstack tenant that has been removed, and apparently this is what prevents the model destruction (this log comes up often) :

2017-10-11 19:45:42 ERROR juju.worker.dependency engine.go:546 "undertaker" manifold worker returned unexpected error: destroying instances: failed to get list of server details
caused by: authentication failed
caused by: Unauthorised URL http://<IP>:5000/v2.0/tokens
caused by: request (http://<IP>:5000/v2.0/tokens) returned unexpected status: 401; error info: Failed: 401 error: The request you have made requires authentication.

I had to tcpdump to find the tenant used, and search in the DB which model was using that tenant - it would be nice to have this info (model UUID, and openstack tenant) in the error message (see bug 1690004).

How can we get this model to be removed ?

This is juju 2.2.4 (but perhaps the deletion happened with another version).

Thanks

[Anastasia (anastasia-macmood)]

@Junien Fridrick (axino),

Please try to update model credentials to a valid one using 'update-credential' command.
I am hoping the update will proceed irrespective of a model status, well, at least 'destroying' != 'dead'.

We will try to address this scenario in the future cloud credential work. At lest, I agree, the error message could be more helpful.

Also, the bug you've linked to in the description goes to "driver for epson c59" :) I am pretty sure it's not related...

[Junien F (axino)]

I guess I can create a new user and update the credentials. I'll let you know how it goes. Also, I have updated the linked bug.

[Junien F (axino)]

Adding the tenant/user with the correct password allowed the model to be deleted. I still wish this would be a smoother process :)

[Anastasia (anastasia-macmood)]

We are looking at the requirements for cloud credentials management on a long-lived models. As part of this work, we will consider what to do / how to flag to the user(s) invalid or invalidated credentials.

[Anastasia (anastasia-macmood)]

There are 2 parts to the fix here.

1. At least from 2.6, we now prefix log messages with model UUID. For example, this is what I'd get if my AWS credential is invalid:

9ba9ae36-e76c-441c-8093-6dac3df27d36: machine-0 2019-05-07 05:14:26 DEBUG juju.worker.dependency engine.go:583 "undertaker" manifold worker stopped: destroying instances:
The provided credentials could not be validated.....

Based on this, I'll mark this report as Fix Committed.

2. For easier UX, we have added '--force' to destroy model so that the users that know what they are doing have an option to blow the model away ignoring errors. I think that when cloud credential become invalid, 'force' option should definitely remove a model. It does not at the moment. I'll create a separate a bug report (bug # 1827988) and triage it to 2.6.1.

[John A Meinel (jameinel)]

Isn't the model UUID that you see there the model that the controller is
running in, and not the model that is being processed? (we historically
have had a significant problem here, and its on the roadmap to work on
this.)

On Tue, May 7, 2019 at 9:41 AM Anastasia <email address hidden>
wrote:

> There are 2 parts to the fix here.
>
>
> 1. At least from 2.6, we now prefix log messages with model UUID. For
> example, this is what I'd get if my AWS credential is invalid:
>
> 9ba9ae36-e76c-441c-8093-6dac3df27d36: machine-0 2019-05-07 05:14:26 DEBUG
> juju.worker.dependency engine.go:583 "undertaker" manifold worker stopped:
> destroying instances:
> The provided credentials could not be validated.....
>
> Based on this, I'll mark this report as Fix Committed.
>
>
> 2. For easier UX, we have added '--force' to destroy model so that the
> users that know what they are doing have an option to blow the model away
> ignoring errors. I think that when cloud credential become invalid, 'force'
> option should definitely remove a model. It does not at the moment. I'll
> create a separate a bug report (bug # 1827988) and triage it to 2.6.1.
>
> ** Changed in: juju
> Status: Triaged => Fix Committed
>
> ** Changed in: juju
> Milestone: 2.5.6 => 2.6-rc2
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1722896
>
> Title:
> Bad cloud credentials prevent model destruction
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1722896/+subscriptions
>