Can't remove down controller from HA cluster, even though cluster has minimum of three required nodes

[Craig Bender]

Lost a controller, added a new controller, but cannot remove the down controller.

Machine State DNS Inst id Series AZ
0 down 10.1x4.xx.50 xdatne xenial default
1 started 10.xx44.xx.56 n3asas xenial default
2 started 10.1x4.xx.62 gxa7wk xenial default
4 started 10.1x4.xx.176 ph4b8x xenial default

ubuntu@osinfra-ch2-g01:~$ juju remove-machine 0 --force
ERROR no machines were destroyed: machine is required by the model

Remaining nodes spend a lot of time trying to talk to down node.

[Craig Bender (craig-bender)]

Have to give it a juju enable-ha the -n3 flag, then down controller can be removed.
Doc is missing that bit.

Recovery section: https://jujucharms.com/docs/2.1/controllers-ha

Also the doc should probably mention that constraints used on initial HA enablement need to be used, otherwise it will grab any machine.

Perhaps that's a separate bug.

[Witold Krecicki (wpk)]

Could you paste output of juju show-controller and juju show-machines?

13.05.2017 20:15 "Craig Bender" <email address hidden> napisał(a):

> Public bug reported:
>
> Lost a controller, added a new controller, but cannot remove the down
> controller.
>
> Machine State DNS Inst id Series AZ
> 0 down 10.1x4.xx.50 xdatne xenial default
> 1 started 10.xx44.xx.56 n3asas xenial default
> 2 started 10.1x4.xx.62 gxa7wk xenial default
> 4 started 10.1x4.xx.176 ph4b8x xenial default
>
> ubuntu@osinfra-ch2-g01:~$ juju remove-machine 0 --force
> ERROR no machines were destroyed: machine is required by the model
>
> Remaining nodes spend a lot of time trying to talk to down node.
>
> ** Affects: juju
> Importance: Undecided
> Status: New
>
>
> ** Tags: enable-ha juju
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju-bugs
> https://bugs.launchpad.net/bugs/1690573
>
> Title:
> Can't remove down controller from HA cluster, even though cluster has
> minimum of three required nodes
>
> Status in juju:
> New
>
> Bug description:
> Lost a controller, added a new controller, but cannot remove the down
> controller.
>
> Machine State DNS Inst id Series AZ
> 0 down 10.1x4.xx.50 xdatne xenial default
> 1 started 10.xx44.xx.56 n3asas xenial default
> 2 started 10.1x4.xx.62 gxa7wk xenial default
> 4 started 10.1x4.xx.176 ph4b8x xenial default
>
> ubuntu@osinfra-ch2-g01:~$ juju remove-machine 0 --force
> ERROR no machines were destroyed: machine is required by the model
>
> Remaining nodes spend a lot of time trying to talk to down node.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1690573/+subscriptions
>

[Witold Krecicki (wpk)]

Also, try juju enable-ha -n 3 and then removing the down machine.

13.05.2017 20:15 "Craig Bender" <email address hidden> napisał(a):

> Public bug reported:
>
> Lost a controller, added a new controller, but cannot remove the down
> controller.
>
> Machine State DNS Inst id Series AZ
> 0 down 10.1x4.xx.50 xdatne xenial default
> 1 started 10.xx44.xx.56 n3asas xenial default
> 2 started 10.1x4.xx.62 gxa7wk xenial default
> 4 started 10.1x4.xx.176 ph4b8x xenial default
>
> ubuntu@osinfra-ch2-g01:~$ juju remove-machine 0 --force
> ERROR no machines were destroyed: machine is required by the model
>
> Remaining nodes spend a lot of time trying to talk to down node.
>
> ** Affects: juju
> Importance: Undecided
> Status: New
>
>
> ** Tags: enable-ha juju
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju-bugs
> https://bugs.launchpad.net/bugs/1690573
>
> Title:
> Can't remove down controller from HA cluster, even though cluster has
> minimum of three required nodes
>
> Status in juju:
> New
>
> Bug description:
> Lost a controller, added a new controller, but cannot remove the down
> controller.
>
> Machine State DNS Inst id Series AZ
> 0 down 10.1x4.xx.50 xdatne xenial default
> 1 started 10.xx44.xx.56 n3asas xenial default
> 2 started 10.1x4.xx.62 gxa7wk xenial default
> 4 started 10.1x4.xx.176 ph4b8x xenial default
>
> ubuntu@osinfra-ch2-g01:~$ juju remove-machine 0 --force
> ERROR no machines were destroyed: machine is required by the model
>
> Remaining nodes spend a lot of time trying to talk to down node.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1690573/+subscriptions
>

[John A Meinel (jameinel)]

I walked him through that last night. This bug is essentially that the docs
don't guide you through that well.

John
=:->

On May 13, 2017 22:25, "Witold Krecicki" <email address hidden> wrote:

> Also, try juju enable-ha -n 3 and then removing the down machine.
>
> 13.05.2017 20:15 "Craig Bender" <email address hidden> napisał(a):
>
> > Public bug reported:
> >
> > Lost a controller, added a new controller, but cannot remove the down
> > controller.
> >
> > Machine State DNS Inst id Series AZ
> > 0 down 10.1x4.xx.50 xdatne xenial default
> > 1 started 10.xx44.xx.56 n3asas xenial default
> > 2 started 10.1x4.xx.62 gxa7wk xenial default
> > 4 started 10.1x4.xx.176 ph4b8x xenial default
> >
> > ubuntu@osinfra-ch2-g01:~$ juju remove-machine 0 --force
> > ERROR no machines were destroyed: machine is required by the model
> >
> > Remaining nodes spend a lot of time trying to talk to down node.
> >
> > ** Affects: juju
> > Importance: Undecided
> > Status: New
> >
> >
> > ** Tags: enable-ha juju
> >
> > --
> > You received this bug notification because you are subscribed to juju.
> > Matching subscriptions: juju-bugs
> > https://bugs.launchpad.net/bugs/1690573
> >
> > Title:
> > Can't remove down controller from HA cluster, even though cluster has
> > minimum of three required nodes
> >
> > Status in juju:
> > New
> >
> > Bug description:
> > Lost a controller, added a new controller, but cannot remove the down
> > controller.
> >
> > Machine State DNS Inst id Series AZ
> > 0 down 10.1x4.xx.50 xdatne xenial default
> > 1 started 10.xx44.xx.56 n3asas xenial default
> > 2 started 10.1x4.xx.62 gxa7wk xenial default
> > 4 started 10.1x4.xx.176 ph4b8x xenial default
> >
> > ubuntu@osinfra-ch2-g01:~$ juju remove-machine 0 --force
> > ERROR no machines were destroyed: machine is required by the model
> >
> > Remaining nodes spend a lot of time trying to talk to down node.
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/juju/+bug/1690573/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1690573
>
> Title:
> Can't remove down controller from HA cluster, even though cluster has
> minimum of three required nodes
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1690573/+subscriptions
>

[Tim Penhey (thumper)]

Well, if enable-ha got the system into a situation where there were three active working HA machines in the cluster, then 'remove-machine 0' *should* just do the right thing.

[Joseph Phillips (manadart)]

Mongo peer-group logic has been overhauled since this report.