harvest mode setting not honoured by destroy-model

[Frode Nordahl]

$ juju --version
2.2-alpha1-xenial-amd64

Steps to reproduce:
1) juju add-model test
2) juju add-machine
3) juju model-config provisioner-harvest-mode=none
4) juju remove-machine 0
5) Verify that machine has been removed from model
6) Verify that machine has NOT been harvested on the provider
7) juju destroy-model test
8) Watch that regardless of model-config provisioner-harvest-mode setting being set to none the machine has indeed been harvested on the provider

[John A Meinel (jameinel)]

The lifetimes of machines deployed by Juju are intentionally tied to the
lifetime of the model. We don't support identifying already running
machines which means that those machines would just become orphaned and it
would not be possible to control them under Juju again.

Harvest mode was never intended to support this. Is there a concrete reason
why you want to deploy but never operate those machines afterward?

Tempted to flag this as invalid/won't fix

John
=:->

On Mar 10, 2017 09:55, "Frode Nordahl" <email address hidden> wrote:

> ** Tags added: sts
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> https://bugs.launchpad.net/bugs/1671733
>
> Title:
> harvest mode setting not honoured by destroy-model
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1671733/+subscriptions
>

[Felipe Reyes (freyes)]

Hi John,

On Fri, Mar 10, 2017 at 05:00:21PM -0000, John A Meinel wrote:
> The lifetimes of machines deployed by Juju are intentionally tied to the
> lifetime of the model. We don't support identifying already running
> machines which means that those machines would just become orphaned and it
> would not be possible to control them under Juju again.

Something that I found strange on this behavior is that after you ran
"juju remove-machine X" that machine won't be displayed to the user in
"juju status" which it's consistent with the operation remove-machine, so
if remove-machine did release the resource, why later (could be minutes,
days, months) juju will release the machine in maas during
destroy-model?.

> Harvest mode was never intended to support this. Is there a concrete reason
> why you want to deploy but never operate those machines afterward?

Frode found this during the exploration of possible workarounds for the
bug https://bugs.launchpad.net/juju/+bug/1671588

Best,

[Anastasia (anastasia-macmood)]

I am removing this from "juju-core" as we do not track Juju 2.x bugs there. Also, this is not a bug but is an intended behavior. So, I am marking this as Invalid for "juju" in agreement with comment # 1.

[John A Meinel (jameinel)]

I'm not as sure. If "juju remove-machine" really does leave the machine
running, then I would expect "juju destroy-model" to not touch the machine.

John
=:->

On Mon, Mar 13, 2017 at 5:00 PM, Anastasia <email address hidden>
wrote:

> I am removing this from "juju-core" as we do not track Juju 2.x bugs
> there. Also, this is not a bug but is an intended behavior. So, I am
> marking this as Invalid for "juju" in agreement with comment # 1.
>
> ** No longer affects: juju-core
>
> ** Changed in: juju
> Status: New => Invalid
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> https://bugs.launchpad.net/bugs/1671733
>
> Title:
> harvest mode setting not honoured by destroy-model
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1671733/+subscriptions
>

[Frode Nordahl (fnordahl)]

Anastasia, thank you for removing juju-core from the bug. Adding that was my mistake.

John, thank you for your supporting comment. juju remove-machine does indeed leave the machine running when provisioner-harvest-mode is set to none.

I think the current behaviour is problematic. If an operator, for whatever reason, at some point decides to use that setting to remove a machine from a juju model without having it destroyed, and then juju later decides to destroy it anyway without any visibility to the operator is going to cause harm.

[Felipe Reyes (freyes)]

On Mon, Mar 13, 2017 at 10:00:45PM -0000, Anastasia wrote:
> I am removing this from "juju-core" as we do not track Juju 2.x bugs
> there. Also, this is not a bug but is an intended behavior. So, I am
> marking this as Invalid for "juju" in agreement with comment # 1.

From the docs[0]:

"""
Juju can be in one of several harvesting modes, in order of most
conservative to most aggressive:

* none: Machines will never be harvested. This is a good choice if
  machines are managed via a process outside of Juju.
...
"""

I understand from this that if it's set to 'none', then it's up to the
user to release the resources, probably (haven't checked) the only thing
that juju should do is remove itself from the machine (uninstall the
agent, remove /var/lib/juju)

[0] https://jujucharms.com/docs/stable/models-config#juju-lifecycle-and-harvesting

[John A Meinel (jameinel)]

I have a feeling that implementation wise we need to remove whatever tag on the machine that indicates it is in the model which is evaluated during model tearsown.

[Frode Nordahl (fnordahl)]

Yes, that or just remove it from the database altogether.

From a operational perspective orphaned machines is not a critical problem, erroneously killed ones is.

[Anastasia (anastasia-macmood)]

Unfortunately, with 2.6 this is still the case. I'll add it to 2.6.1 milestone.

[Anastasia (anastasia-macmood)]

This is a very interesting case and is highlighting a few issues with the implementation.

For one, when removing machines 'destroy-model' actually force removes them (unless the machines are manual). This, in itself, may cause all sort of undesirable issues since the machines and their units (and consequently relations, applications, etc) are not given a chance to terminate properly.

The fix here has 2 parts.

Part 1:
Now that we do have --force option for 'destroy-model', we should only force remove non-manual machines when --force is desired for the model. Manual machine treatment will remain the same. The implications of this change would be a bit of a double-edged sword: on one hand, the machines will be given a chance to be terminated properly and harvest mode will be respected; on the other hand, this will definitely impact on how quickly models will be destroyed.

Part 2:
This all implies that force removal of machines does not respect harvest mode. We should rectify that.

[Anastasia (anastasia-macmood)]

After some discussions, we'll keep the model destruction to always force remove non-manual machines - there is no benefit in going through a normal, lengthy removal if we are destroying a model anyway. However, we will ensure that force removing a machine respects harvest mode.

[Anastasia (anastasia-macmood)]

Actually, I am seeing the same behavior with 'remove-machine --force'.. At least from the command line, when the removal is forced, the machine on the model with provisioner-harvest-mode=none is untouched on the cloud but is terminated when its model is destroyed. I'll dig into model destruction exclusively.

[Anastasia (anastasia-macmood)]

I think the problem is that the models cloud machines are terminated by the undertaker when a model is destroyed, not a provisioner unlike in remove-machine case. And the undertaker is not harvest-mode savvy.

[Anastasia (anastasia-macmood)]

The same would apply to if a controller has models with harvest mode set to leave cloud instances alone. When Juju destroys it, harvest mode will not be respected... Need to fix this too.

[Anastasia (anastasia-macmood)]

Due to current priorities, I am not currently working on this bug, so I am un-assigning myself from it.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.