juju add-unit often complains of "inconsistent state" when you need to replace a failed unit

[Nick Moffitt]

One of the most common use cases for juju add-unit is when an existing unit is in an unrecoverable state (such as a problem with the compute node it is hosted on), and you want to take advantage of the cloud to just spin up a replacement. Unfortunately juju often refuses to add one for you because it's not happy with the state of the environment at that point:

    ERROR cannot add unit 1/1 to service "example": cannot add unit to service "example": inconsistent state

Typically this operation is one you're performing under time pressure while a production service is down, and you are taking this action in order to repair a fault.

For a start, the error message suggests to the uninitiated that juju wants you to fix the unfixable unit before adding any more. It may be helpful to clarify what exactly it is that is inconsistent, and if waiting and trying again may help (it seems to in practice).

Second, it would be nice if this could be removed as a stumbling block to repairing production environments during crises.

[Anastasia (anastasia-macmood)]

@Nick Moffitt,

Could you please provide more information?
Reproducible scenario will go a long way too \o/

Were you running any commands before add-unit? If yes, what were they?
Does waiting and trying again help? If yes, for how long do you usually wait to try again?
Does it happen all the time? i.e. you run add-unit first time and it fails, you wait and run it again and it succeeds consistently?

[Nick Moffitt (nick-moffitt)]

I can't really say how to build up a reproducible situation, but this doesn't need a particularly large environment to have this. But if you just start up an environment in 1.25.6 or so and shut down a unit from outside juju, usually this happens (particularly if a hook failed earlier).

Yeah, usually if you keep retrying eventually it works. But we didn't know this until we got really desperate.

[Anastasia (anastasia-macmood)]

@Nick,

As you have discovered this on Juju 1.x which is currently only open to Critical bugs, we will not be fixing it there. It's great that you have a workaround \o/

We will, however, fix it in Juju 2.x. Thank you for the additional info!

[Anastasia (anastasia-macmood)]

@Nick Moffitt (nick-moffitt),
Could you please re-test with Juju 2.1?
We have addressed an issue with removing units/applications in error, in 2.1-rc1.

I think your experience would improve too \o/

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]

[Paul Goins (vultaire)]

On 2019-06-07, we encountered this when working on a customer's cloud, trying to add a new mysql unit to replace one which had failed. We got the same "inconsistent state" issues, and we were able to force it to go through by just continuing to retry. This was with juju client 2.6.3-trusty-amd64, juju model at 2.4.3.

That environment has since been upgraded, so I can't readily re-test this; just wanted to provide a data point that this issue might still be persisting to some degree.

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]

[David O Neill (dmzoneill)]

Same

juju add-unit cinder --to lxd:1
ERROR cannot add unit 1/1 to application "cinder": cannot add unit to application "cinder": inconsistent state
juju add-unit cinder --to lxd:1
ERROR cannot add unit 1/1 to application "cinder": cannot add unit to application "cinder": inconsistent state
juju add-unit cinder --to lxd:1

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.