[vsphere 6.5] root-disk-source is failing to select from available datastores

Bug #1874031 reported by Peter De Sousa
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju
High
Christian Muirhead

Bug Description

Hi,

When creating a controller on vSphere 6.5 I am able to specify a datastore with the root-disk-source constraint:

08:59:16 DEBUG juju.provider.vmware createvm.go:453 Selecting datastore
08:59:16 DEBUG juju.provider.vmware createvm.go:483 desired datastore "TAM-VXR-K8S-PRD-DS01"
08:59:16 DEBUG juju.provider.vmware createvm.go:484 accessible datastores ["TAM-VXR-K8S-PRD-DS01" "TAM-VXR-K8S-PRD-DS02" "TAM-VXR-K8S-PRD-DS03"]
08:59:16 INFO juju.provider.vmware createvm.go:487 selecting datastore TAM-VXR-K8S-PRD-DS01

Although, when specifying one of these datastores in the bundle I cannot select any of these datastores:

"ERROR cannot deploy bundle: cannot create machine for holding kubeapi-load-balancer, kubernetes-master and vsphere-integrator units: cannot add a new machine datastore "TAM-VXR-K8S-PRD-DS01" not found"

Interestingly, leaving the datastore unspecified in the constraints, but set in the model defaults allows me to select one datastore.

Here's output from add-machine:

ubuntu@maas:~/$ juju add-machine --constraints="root-disk-source=TAM-VXR-K8S-PRD-DS02"
failed to create 1 machine
ERROR cannot add a new machine: datastore "TAM-VXR-K8S-PRD-DS02" not found
ubuntu@maas:~/$ juju add-machine --constraints="root-disk-source=TAM-VXR-K8S-PRD-DS01"
failed to create 1 machine
ERROR cannot add a new machine: datastore "TAM-VXR-K8S-PRD-DS01" not found
ubuntu@maas:~/$ juju add-machine --constraints="root-disk-source=TAM-VXR-K8S-PRD-DS03"
failed to create 1 machine
ERROR cannot add a new machine: datastore "TAM-VXR-K8S-PRD-DS03" not found

Any help would be greatly appreciated,

Cheers
Peter

Peter De Sousa (pjds)
description: updated
description: updated
Peter De Sousa (pjds)
description: updated
Peter De Sousa (pjds)
description: updated
Revision history for this message
Peter De Sousa (pjds) wrote :

Subscribing field high

Revision history for this message
Peter De Sousa (pjds) wrote :

Adding bootstrap log

Peter De Sousa (pjds)
summary: - [vsphere 6.5] root-disk-source failing to select from available
+ [vsphere 6.5] root-disk-source is failing to select from available
datastores
Revision history for this message
Christian Muirhead (2-xtian) wrote :

Hi Peter -
I've just tried bootstrapping with a root-disk-source constraint and then deploying a bundle with a different root-disk-source constraint, which worked fine.
Can you give more information about the datacenter (clusters/hosts and datastores) and the bundle (ideally a cut-down one that exhibits the same behaviour)?
Is the constraint in the bundle on the application or on the machine?

Also logging output from the failed add-machine commands would be useful.

The fact that the error is from add-machine rather a bad status on the machine suggests that the problem is in the precheck instance call, but I can't see how that would happen yet.

Changed in juju:
status: New → Incomplete
Revision history for this message
Peter De Sousa (pjds) wrote :

Hi Christian

Can you give more information about the datacenter (clusters/hosts and datastores) and the bundle (ideally a cut-down one that exhibits the same behaviour)?

There is one cluster, two hosts and three datastores.

Is the constraint in the bundle on the application or on the machine?

The constraint is on a machine, although it will exhibit this issue if I put the constraints onto charms.

Also logging output from the failed add-machine commands would be useful.

Logging and bundle attached.

Thanks!

Revision history for this message
Peter De Sousa (pjds) wrote :

Apologies, the bundle was clipped.

Here's the bundle:

Revision history for this message
Peter De Sousa (pjds) wrote :
Revision history for this message
Christian Muirhead (2-xtian) wrote :

Ok, I tried again with the constraint on the machine and could still deploy. :/

Could you send the output of `govc ls -json /<your-datacenter>/datastore`?

The stack trace definitely shows that the error is in the instance precheck, but that just gets all accessible datastores and checks for one with the right name.

Revision history for this message
Peter De Sousa (pjds) wrote :

Hi Christian,

Sure, attached. Thank you so much for your help so far.

Peter

Changed in juju:
status: Incomplete → Confirmed
Revision history for this message
Christian Muirhead (2-xtian) wrote :

Thanks for that Peter, it looks like the datastores are in a folder, rather than at the top level. I feel silly saying this because it seems so obvious in retrospect, but it didn't occur to me that you could put datastores in folders.

The code to get the accessible datastores won't traverse a folder tree at the moment. We can change that. As a workaround in the meantime, is it feasible to move the datastores up to the top-level?

There are some wrinkles to how we handle datastores in folders - I think it would be better to keep the paths absolute for consistency with resource pools, but it's interesting that bootstrapping was able to find the datastore by unqualified name. I think that's because it finds the datastore by links from the host at that point in the code, rather than from the root datastores folder.

Changed in juju:
milestone: none → 2.8-rc1
importance: Undecided → High
assignee: nobody → Christian Muirhead (2-xtian)
Revision history for this message
Peter De Sousa (pjds) wrote :

Hi Christian,

Great catch it does indeed seem the datastores in are in a folder. I will request that these be moved to the top level in the meantime.

Again, thank you for your work

Peter

Ian Booth (wallyworld)
Changed in juju:
status: Confirmed → In Progress
Revision history for this message
Christian Muirhead (2-xtian) wrote :

PR for fix on 2.7 branch: https://github.com/juju/juju/pull/11528
(I'll merge it forward to the 2.8 branch once this is landed.)

Ian Booth (wallyworld)
Changed in juju:
status: In Progress → Fix Committed
Harry Pidcock (hpidcock)
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers