cannot deploy any charm using ost provider on 2.7-rc2

[Edward Hope-Morley]

Since my client upgraded from 2.7-rc1 to 2.7-rc2 I am no longer able to deploy anything using the Openstack provider:

ubuntu@hopem-bastion:~/stsstack-bundles/openstack$ juju add-model foo
Added 'foo' model on stsstack/stsstack with credential 'hopem' for user 'admin'

ubuntu@hopem-bastion:~/stsstack-bundles/openstack$ juju deploy cs:ubuntu
ERROR default space name "_default" not valid

ubuntu@hopem-bastion:~/stsstack-bundles/openstack$ juju list-spaces
cannot list spaces: spaces not supported (not supported)
ERROR cannot list spaces: spaces not supported (not supported)

ubuntu@hopem-bastion:~/stsstack-bundles/openstack$ juju --version
2.7-rc2-bionic-amd64

[Edward Hope-Morley (hopem)]

Upgrading to rc3 didn't fix it:

ubuntu@hopem-bastion:~$ sudo snap refresh juju --channel 2.7/candidate
juju (2.7/candidate) 2.7-rc3+2.7-2e69ca6 from Canonical✓ refreshed
ubuntu@hopem-bastion:~$ juju deploy cs:ubuntu
ERROR default space name "_default" not valid
1 ubuntu@hopem-bastion:~$ juju --version
2.7-rc3-bionic-amd64

[Edward Hope-Morley (hopem)]

fwiw dropping back to 2.6.10 fixed it so this is a regression in 2.7.

[Edward Hope-Morley (hopem)]

A fresh bootstrap using rc3 seems to work well so I will close this out for now.

[Richard Harding (rharding)]

Yes, this was a known issue in rc2 and addressed in rc3. I'm surprised the update to rc3 didn't fix it.

[John A Meinel (jameinel)]

Given the string "_default", I think that only existed in rc1, (it is now
called "alpha") which is why rc1 => rc3 broke.

However, our statement is that if you go to rc1 you'll be able to upgrade
to anything post rc1, means we do need a fix for this (IMO).

On Mon, Nov 11, 2019 at 9:50 AM Edward Hope-Morley <
<email address hidden>> wrote:

> A fresh bootstrap using rc3 seems to work well so I will close this out
> for now.
>
> ** Changed in: juju
> Status: New => Invalid
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1852098
>
> Title:
> cannot deploy any charm using ost provider on 2.7-rc2
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1852098/+subscriptions
>

[Richard Harding (rharding)]

Sorry, maybe not. Looking.

[John A Meinel (jameinel)]

This could be something wrt mixed versions (controller is 2.7rc3 but agents
are still 2.7rc1), but it sounds like something where rc1 was setting a
value that isn't valid in rc3 and we didn't have an upgrade step post rc1.
(default-model as model configuration comes to mind as an obvious place)
Which means a user workaround of "juju model-config default-model=''" or
something like that is probably fine vs an upgrade step. The key is that
Juju needs to provide a way for someone going from 2.6 to test our rc, and
still be able to upgrade to 2.7.0 final. If we can just make it happen in
an upgrade step that is better, but a manual workaround would also be
acceptable, IMO.

On Mon, Nov 11, 2019 at 10:50 AM Richard Harding <email address hidden>
wrote:

> Sorry, maybe not. Looking.
>
> ** Changed in: juju
> Status: Invalid => Triaged
>
> ** Changed in: juju
> Importance: Undecided => High
>
> ** Changed in: juju
> Assignee: (unassigned) => Joseph Phillips (manadart)
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1852098
>
> Title:
> cannot deploy any charm using ost provider on 2.7-rc2
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1852098/+subscriptions
>

[Richard Harding (rharding)]

Discussing this, we need to validate if you can work around this with a manual setting of the model-config default-space=alpha

If that works and can be noted as a workaround for users that went through the rc1 release that's ok for now.

If that's not sufficient we'll need to make sure we have a proper upgrade-step and put out an rc-4 with it.

[Joseph Phillips (manadart)]

This is a *client* incompatibility between the latest versions and the earlier beta that entertained the "_default" space.

If you have "_default" as model configuration for default-space, a later client does not tolerate this as a valid space name. Anything that instantiates an environ client-side throws this error.

One option is to allow "_default" as valid default-space when loading config in environs/config/config.go so that a later client can be used to upgrade. Then we just remove it post 2.7.0.

[Achilleas Anagnostopoulos (achilleasa)]

I have attempted a sequence of upgrades from 2.6.10 -> 2.7-beta1 -> 2.7-rcX. In each step I first updated the client, ran remove-application and deploy, then upgraded the *controller* and repeated the same steps. I can reproduce the problem on lxd when upgrading the controller from 2.7-beta1 -> 2.7-rc1.

I can see that after upgrading from 2.7-beta1 -> 2.7-rc1 the "default-space" model config is set to "_default". This prevents deploys and upgrading (both the controller and the model which in my case was still in 2.6.10).

However, running "juju model-config default-space=alpha" as per the above suggestions resolves the issue and makes everything work again.

[James Hebden (ec0)]

Also saw this on an existing LXD provider which was bootstrapped on 2.7-rc1, now trying to use either the controller or any of the deployed models with the 2.8-beta1 client. This looks to be an upgrade blocker, as without the workaround, it is impossible to run any operations against the controller, controller model, or deployed models.

With the above workaround applied, i.e. -
juju model-config -m controller default-space=alpha
juju model-config default-space=alpha

I am able to access the controller and deployed model again on the LXD provider.
I was then able to upgrade from 2.7-rc1 to 2.8-beta1 without issue.

[John A Meinel (jameinel)]

Since there is a way to get from rc1 to newer releases (with a manual step), and upgrading from anything to rc2+ doesn't run into this, I don't think it is a blocker for 2.7. (2.7.0 doesn't have this, it only happens if you go to rc1, and we have a way to get you off rc1.)

[Tim Penhey (thumper)]

We aren't going to add an automated upgrade step for this, given that the manual commands unblock this.

I'm going to mark this as fix committed, since it isn't either Invalid, nor Won't Fix as that implies that there is no way forwards.