add-k8s requires a model

[Cory Johns]

Is there a reason that add-k8s requires a current or specified model? As far as I can tell, it doesn't use it.

[John A Meinel (jameinel)]

It should not. It probably is using the common "ModelCmdBase" and shouldn't. You don't need a model to add a model. You certainly don't need a model to add a cloud. (you should be able to add-k8s even without having bootstrapped, even if it wouldn't be a target for bootstrap)

[Christopher Lee (veebers)]

The command add-k8s currently stores data on the controller when run.

This means that the command adds details in the local config file as well as the currently selected controller.

Is the expectation of this command that it needs to be run on any controller to enable it with that k9s cluster (as a cloud)?

It requires "add-k8s" be run on every controller (i.e. if you boostrap, add-k8s test1, bootstrap another controller the test1 k8s cloud won't be populated on the new controller).

Would it be possible for the cloud and cred populating to happen later on in the process (specifically add-model)?
This would allow for a work flow of:
  - add-k8s test1
  - bootstrap c1, bootstrap c2
  - juju add-model -c c2 newmodel test1 (check here and update controller with cloud details if not already existing).

[John A Meinel (jameinel)]

We have a small issue that you cannot bootstrap *onto* a k8s cloud, so
there is always somewhere else that you have to bootstrap.
Which is a bit different from a lot of our other assumptions that the
controller sits on the cloud where the models are, but honestly we want to
move away from that anyway. (eg, as an Ideal you could have 1 Juju
Controller, that could create a model on MAAS that would build an Openstack
that would allow you to build a Kubernetes on top of Openstack, which would
then let you deploy an AI workflow on top of Kubernetes.)

As for whether we should track local k8s credentials separately from the
values on the controllers...

I think it would make the experience consistent with how we do credentials
for other clouds. We don't forget about them when you destroy a controller,
and you can upload your local credentials at any point to a remote
controller (think JAAS) to have it start controlling instances with those
credentials.

Now, I'm not 100% sure what should happen with 'remove-k8s' vs an
equivalent 'remove-credential' or 'remove-cloud'.
(k8s seems to blur the distinction between cloud definition and credential
for that cloud, as well). I know it is generally nicer for users to be able
to supply them combined (that was certainly the feedback from MAAS that it
was cumbersome to add the cloud and then add the credential for your user
in that cloud.)

I think ideally, add-k8s can be effectively 'add-cloud && add-credential'
but I'm wondering if we want to end up removing them with remove-cloud,
remove-credential instead of combining them...

We should probably have a chat with a few people about expectations and
what the user stories are.

On Tue, May 8, 2018 at 5:48 AM, Christopher Lee <email address hidden>
wrote:

> The command add-k8s currently stores data on the controller when run.
>
> This means that the command adds details in the local config file as
> well as the currently selected controller.
>
> Is the expectation of this command that it needs to be run on any
> controller to enable it with that k9s cluster (as a cloud)?
>
> It requires "add-k8s" be run on every controller (i.e. if you boostrap,
> add-k8s test1, bootstrap another controller the test1 k8s cloud won't be
> populated on the new controller).
>
> Would it be possible for the cloud and cred populating to happen later on
> in the process (specifically add-model)?
> This would allow for a work flow of:
> - add-k8s test1
> - bootstrap c1, bootstrap c2
> - juju add-model -c c2 newmodel test1 (check here and update controller
> with cloud details if not already existing).
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1768847
>
> Title:
> add-k8s requires a model
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1768847/+subscriptions
>

[Christopher Lee (veebers)]

Perhaps "add-k8s" should be split into 2 commands, "add-k8s" like add-cloud adds details to the local config so juju knows about it and "enable-k8s -c <controller>" (or activate-k8s or something) that puts the details onto a controller so that you can now 'add-model <k8s cloud name>).

To then remove the details from the controller you can deactivate-k8s (remove from the controller) and remove-k8s (like current remove-cloud) removes details from local config files (maybe removing from the controllers too?).

[John A Meinel (jameinel)]

The main caveat is that I think we *do* want a single-step "import the
cloud and credentials" somewhere in the system, because feedback has been
that forcing users to use 2 commands is not very nice.

On Tue, May 8, 2018 at 12:28 PM, Christopher Lee <<email address hidden>
> wrote:

> Perhaps "add-k8s" should be split into 2 commands, "add-k8s" like add-
> cloud adds details to the local config so juju knows about it and
> "enable-k8s -c <controller>" (or activate-k8s or something) that puts
> the details onto a controller so that you can now 'add-model <k8s cloud
> name>).
>
> To then remove the details from the controller you can deactivate-k8s
> (remove from the controller) and remove-k8s (like current remove-cloud)
> removes details from local config files (maybe removing from the
> controllers too?).
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1768847
>
> Title:
> add-k8s requires a model
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1768847/+subscriptions
>

[Ian Booth (wallyworld)]

Agree with John - there needs to be a single add-k8s command (or add-cloud in general) that does both things, register the cloud definition / endpoint etc plus the credential. We have that now with add-k8s for the kubernetes case, and should look to grow that to support maas and other non-public clouds whose definitions do not ship out of the box.

What needs work is the clean up step. remove-cloud only currently removes the local definition. We need to consider having that command inspect the controller and if there are any models currently using the cloud, refuse to do the removal; if the cloud stored in the controller is not used, it can be removed, followed by the local removal from the cloud yaml. Or something like that.

[Christopher Lee (veebers)]

Ok, so my understanding from this convo is that:
  - add-k8s is expected to update the local config *and* the current controller with details
  - Meaning add-k8s requires a bootstrapped controller
  - Adding the added k8s cloud to another controller would require one to re-run the 'add-k8s' command with the new controller (this doesn't error if it exists in the local config but would update the new controller)

  - The cleanup details are for bug LP:1768845 and will move the discussion there (namely, what does it mean to clean it up from the controller).

Assuming my understanding is correct there are some improvements available here I think:
  - Flesh out the summary to mention the controllers involvement and that it updates local config but just the controller in question, maybe mention that to do other controllers to re-run the command.

[Cory Johns (johnsca)]

I think it's reasonable to have add-k8s be a combination of add-cloud and add-credential, but I don't think it should require a controller until you actually add-model. This would match how we currently handle cloud and credential info, where it's stored in the local data and the credentials are uploaded automatically to the controller when add-model is invoked. That makes sense, because the credentials are relevant to the cloud and not the specific controller running on that cloud.

If you add endpoint and credential info for an OpenStack cloud, you don't have to re-add that for every new controller you bootstrap into that cloud.

[Ian Booth (wallyworld)]

add-cloud and add-k8s are subtly different, and we are discussing changing add-cloud to also add to the controller if there is one running.

add-cloud at the moment is meant to set up a cloud definition in the local yaml file to seed the bootstrap of a new controller. All models in that controller must run on the cloud. add-cloud after the controller is bootstrapped is meaningless.

add-k8s is only useful in conjunction with a running controller. You don't bootstrap a controller onto k8s but add a k8s cloud as an option for an existing controller to use. Just adding the k8s definition to local yaml is kinda meaningless; the openstack analogy is not that relevant as you bootstrap onto openstack but you can't onto k8s.

What we will be adding is a remove-k8s command. This will remove a k8s cloud definition from the controller so long as no models currently use it.