[vsphere 6.5] root-disk-source is failing to select from available datastores

[Peter Jose De Sousa]

Hi,

When creating a controller on vSphere 6.5 I am able to specify a datastore with the root-disk-source constraint:

08:59:16 DEBUG juju.provider.vmware createvm.go:453 Selecting datastore
08:59:16 DEBUG juju.provider.vmware createvm.go:483 desired datastore "TAM-VXR-K8S-PRD-DS01"
08:59:16 DEBUG juju.provider.vmware createvm.go:484 accessible datastores ["TAM-VXR-K8S-PRD-DS01" "TAM-VXR-K8S-PRD-DS02" "TAM-VXR-K8S-PRD-DS03"]
08:59:16 INFO juju.provider.vmware createvm.go:487 selecting datastore TAM-VXR-K8S-PRD-DS01

Although, when specifying one of these datastores in the bundle I cannot select any of these datastores:

"ERROR cannot deploy bundle: cannot create machine for holding kubeapi-load-balancer, kubernetes-master and vsphere-integrator units: cannot add a new machine datastore "TAM-VXR-K8S-PRD-DS01" not found"

Interestingly, leaving the datastore unspecified in the constraints, but set in the model defaults allows me to select one datastore.

Here's output from add-machine:

ubuntu@maas:~/$ juju add-machine --constraints="root-disk-source=TAM-VXR-K8S-PRD-DS02"
failed to create 1 machine
ERROR cannot add a new machine: datastore "TAM-VXR-K8S-PRD-DS02" not found
ubuntu@maas:~/$ juju add-machine --constraints="root-disk-source=TAM-VXR-K8S-PRD-DS01"
failed to create 1 machine
ERROR cannot add a new machine: datastore "TAM-VXR-K8S-PRD-DS01" not found
ubuntu@maas:~/$ juju add-machine --constraints="root-disk-source=TAM-VXR-K8S-PRD-DS03"
failed to create 1 machine
ERROR cannot add a new machine: datastore "TAM-VXR-K8S-PRD-DS03" not found

Any help would be greatly appreciated,

Cheers
Peter

[Peter Jose De Sousa (pjds)]

Subscribing field high

[Peter Jose De Sousa (pjds)]

Adding bootstrap log

[Christian Muirhead (2-xtian)]

Hi Peter -
I've just tried bootstrapping with a root-disk-source constraint and then deploying a bundle with a different root-disk-source constraint, which worked fine.
Can you give more information about the datacenter (clusters/hosts and datastores) and the bundle (ideally a cut-down one that exhibits the same behaviour)?
Is the constraint in the bundle on the application or on the machine?

Also logging output from the failed add-machine commands would be useful.

The fact that the error is from add-machine rather a bad status on the machine suggests that the problem is in the precheck instance call, but I can't see how that would happen yet.

[Peter Jose De Sousa (pjds)]

Hi Christian

Can you give more information about the datacenter (clusters/hosts and datastores) and the bundle (ideally a cut-down one that exhibits the same behaviour)?

There is one cluster, two hosts and three datastores.

Is the constraint in the bundle on the application or on the machine?

The constraint is on a machine, although it will exhibit this issue if I put the constraints onto charms.

Also logging output from the failed add-machine commands would be useful.

Logging and bundle attached.

Thanks!

[Peter Jose De Sousa (pjds)]

Apologies, the bundle was clipped.

Here's the bundle:

[Christian Muirhead (2-xtian)]

Ok, I tried again with the constraint on the machine and could still deploy. :/

Could you send the output of `govc ls -json /<your-datacenter>/datastore`?

The stack trace definitely shows that the error is in the instance precheck, but that just gets all accessible datastores and checks for one with the right name.

[Peter Jose De Sousa (pjds)]

Hi Christian,

Sure, attached. Thank you so much for your help so far.

Peter

[Christian Muirhead (2-xtian)]

Thanks for that Peter, it looks like the datastores are in a folder, rather than at the top level. I feel silly saying this because it seems so obvious in retrospect, but it didn't occur to me that you could put datastores in folders.

The code to get the accessible datastores won't traverse a folder tree at the moment. We can change that. As a workaround in the meantime, is it feasible to move the datastores up to the top-level?

There are some wrinkles to how we handle datastores in folders - I think it would be better to keep the paths absolute for consistency with resource pools, but it's interesting that bootstrapping was able to find the datastore by unqualified name. I think that's because it finds the datastore by links from the host at that point in the code, rather than from the root datastores folder.

[Peter Jose De Sousa (pjds)]

Hi Christian,

Great catch it does indeed seem the datastores in are in a folder. I will request that these be moved to the top level in the meantime.

Again, thank you for your work

Peter

[Christian Muirhead (2-xtian)]

PR for fix on 2.7 branch: https://github.com/juju/juju/pull/11528
(I'll merge it forward to the 2.8 branch once this is landed.)