Bootstrap prefers .jenv over environments.yaml

[Aaron Bentley]

I am using the manual provider, so .jenv files are left behind.
When I start a new instance, I get a new IP and write it into the environments.yaml as the bootstrap-host (see also bug #1282217).
When I bootstrap, juju ignores the value I deliberately changed in environments.yaml, and instead uses the stale value from the .jenv

This is a bug because juju is ignoring the user's explicit choice to change the value in environments.yaml, where users are supposed to change the value. Users should not have to hand-hack the .jenv or delete it for a new environment to use the values in environments.yaml.

It is a regression because in previous versions, bootstrap always used the values from environments.yaml

[John A Meinel (jameinel)]

So these are left behind because we aren't using "juju destroy-environment" to clean it up?

[John A Meinel (jameinel)]

What if you use "juju destroy-environment --force" and having that ensure the .jenv file is gone. Which means you can ensure that if the environment is gone, you've told juju to actually forget about it.

Another option is to have some sort of "--forget" or "juju forget-environment" which tells juju to stop remembering an environment that was destroyed out of band.

[John A Meinel (jameinel)]

juju forget-environment has the nice property that it also works for shared environments, where you might not even want to destroy the actual environment, you just want to get rid of your local information about the environment.

[Aaron Bentley (abentley)]

I really don't want any of that. If I wanted to persist the .jenv settings across bootstraps, I would copy them into environments.yaml. I don't want dirty shutdowns breaking future start-ups.

The way I see it, environments.yaml is the configuration and the jenv files are the state. When you're starting, the *configuration* should be honoured, not the last known state of the previous attempt. Juju itself seems to recognize this principle-- by deleting the .jenvs, it ensures that in most cases, the previous state does not affect bootstrapping. It simply needs to handle the corner case where the .jenvs were not actually deleted.

[Dimiter Naydenov (dimitern)]

The .jenv files were introduced recently (r1822 - around 1.15 time) and the whole point of them is to cache api endpoints and credentials, so you don't have to connect to the provider's storage to get the api/state endpoints. It brought performance improvements to CLI connections (it's assumed you want to use your newly bootstrapped environment efficiently and not have to fetch the bootstrap state file from provider storage to get the api endpoints for every CLI command, don't you?)

.jenv files are created after bootstrapping and are used instead of environments.yaml *by design*. Some settings stored there are immutable after bootstrap, one example being the bootstrap-host you're talking about. environments.yaml *might have been* the source of truth for configuration, but that's before you bootstrap, not after.

What you're describing is an unusual scenario. I agree that bootstrap-host should probably be configurable for the manual provider, and there could be a CLI command to "forget" and environment (erase its .jenv). The typical practice is to bootstrap manual environment, work with it and finally destroy it before trying again with a different host. That's what a user will do, but I agree it's probably slow (to do bootstrap+destroy each time) and more convoluted in a CI context, where you need to run the same live tests over and over.

From the UX perspective, we should probably implement some form of checking if environments.yaml changed after bootstrap and notify the user the changes won't take effect, or silently update the .jenv to reflect those changes (if at all possible - settings deemed immutable can't be changed after bootstrap anyway). This will allow us to continue reaping the benefits of .jenv files without surprising the user their changes do not take effect.

[Aaron Bentley (abentley)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 14-02-28 07:16 AM, Dimiter Naydenov wrote:
> The .jenv files were introduced recently (r1822 - around 1.15
> time)

I know. I experienced the transition.

> and the whole point of them is to cache api endpoints and
> credentials, so you don't have to connect to the provider's
> storage to get the api/state endpoints. It brought performance
> improvements to CLI connections (it's assumed you want to use your
> newly bootstrapped environment efficiently and not have to fetch
> the bootstrap state file from provider storage to get the api
> endpoints for every CLI command, don't you?)

I'm not saying that jenv files are a bad idea in general. As you
note, they are supposed to represent the runtime state of a
bootstrapped environment. I am saying the stale state of
no-longer-bootstrapped environment should not interfere with
bootstrapping future environments.

> .jenv files are created after bootstrapping and are used instead of
> environments.yaml *by design*.

Since bootstrap creates them and destroy-environment deletes them,
it's pretty clear they are not supposed to exist at bootstrap time. I
find it hard to understand why the bootstrap command would be designed
to prefer files that are not supposed to exist. I think it is more
likely that *other commands* are designed to prefer .jenvs and that
bootstrap only does because it uses the same code.

Let me be clear, I'm only talking about bootstrap. Other commands
that assume a bootstrapped environment should of course prefer the .jenvs.

> environments.yaml *might have been* the source of truth for
> configuration, but that's before you bootstrap, not after.

All I want is for environments.yaml to be the source of truth before I
bootstrap.

> What you're describing is an unusual scenario. I agree that
> bootstrap- host should probably be configurable for the manual
> provider, and there could be a CLI command to "forget" and
> environment (erase its .jenv). The typical practice is to
> bootstrap manual environment, work with it and finally destroy it
> before trying again with a different host. That's what a user will
> do

According to the docs, destroy-environment does not work for the
manual provider. Are the docs in error?

> but I agree it's probably slow (to do bootstrap+destroy each time)
> and more convoluted in a CI context

All of our other provider tests bootstrap and destroy. The only
reason we don't do it for the manual provider is because it's
documented to not work for the manual provider.

>> From the UX perspective, we should probably implement some form
>> of
> checking if environments.yaml changed after bootstrap and notify
> the user the changes won't take effect, or silently update the
> .jenv to reflect those changes (if at all possible - settings
> deemed immutable can't be changed after bootstrap anyway).

Silently updating the .jenv with the values from environments.yaml is
close to what I want, but I want the much simpler approach of reading
environments.yaml and overwriting the .jenv with those values. To
implement this, all you have to do is tell bootstrap not to look at
the .jenv.

> This will allo...

[Aaron Bentley (abentley)]

Reusing stale data also has security implications. Say my admin-secret in gets compromised. If I don't have admin-secret in my environments.yaml, I usually can re-bootstrap to make Juju generate a new admin-secret. If you only update the .jenv with new settings and there's no admin-secret in my environments.yaml, then the old admin-secret will be retained and I am vulnerable to compromise. The same applies to any credentials like certificates.

[John A Meinel (jameinel)]

Doing things like "juju sync-tools" or a couple of other commands cause the .jenv to be created before the environment is actually bootstrapped. eg sync-tools needs to put the tools into the control-bucket that will be used by bootstrap to download the tools, which means we need to record what the control-bucket will be. control-bucket is optional, and we will supply a random one for you, which means that we can't just read environments.yaml (same for admin-secret). Though admin-secret isn't *needed* until bootstrap.

[Aaron Bentley (abentley)]

Marked as triaged because the linked branch mitigates the bug, but does not fix it.

[William Reade (fwereade)]

So, to be clear, a .jenv is the canonical description of an environment: environments.yaml is a somewhat icky mechanism for feeding data into a new .jenv file at or before bootstrap time. Failing to destroy a .jenv when the environment is destroyed is *certainly* a bug (eg lp:1273295); and it's also important to have a "forget-environment" command (to go with an "add-environment" command, such that both will deal with .jenvs exclusively), where "forget-environment" will be a viable path for resolving this bug, lp:1291165, and future instances in which a shared .jenv refers to an environment that no longer exists; but the only way the core of this issue will be resolved is when we eliminate sync-tools entirely, at which point the existence of a .jenv will be sufficient reason to block bootstrap entirely.

[Tim Penhey (thumper)]

We aren't going to change this. Environments.yaml is going away (perhaps in 2.0).