[vsphere] Juju fails to bootstrap controller in case a cluster is in the folder

[Przemyslaw Hausman]

I'm using vSphere provider and I can't bootstrap juju controller to a specific cluster or a host in the cluster. Juju fails to bootstrap controller in case a cluster is in the folder.

This is the structure in vSphere: (or see attached screenshot)

Datacenter
+- Folder
   +- Cluster

I'm getting "ERROR failed to bootstrap model: cannot start bootstrap instance: availability zone "FolderName/ClusterName" not found".

Here's how I bootstrap the controller:
```
juju bootstrap \
  --config bootstrap-timeout=1800 \
  --to zone=FolderName/ClusterName \
  --constraints 'zone=FolderName/ClusterName cores=8 mem=24G root-disk=100G' \
  --model-default /home/ubuntu/deploy/generated/model_defaults.yaml \
  vsphere foundation_vsphere
```

$ juju --version
2.8.2-bionic-amd64

$ ubuntu@infra-1:~/deploy$ juju bootstrap \
  --debug \
  --config bootstrap-timeout=1800 \
  --to zone=FolderName/ClusterName \
  --constraints 'zones=FolderName/ClusterName cores=8 mem=24G root-disk=100G' \
  --model-default /home/ubuntu/deploy/generated/model_defaults.yaml \
  vsphere foundation_vsphere

14:01:50 INFO juju.cmd supercommand.go:54 running juju [2.8.2 0 a44e6eb38430da695737f5e9f37819478b9587c3 gc go1.14.7]
14:01:50 DEBUG juju.cmd supercommand.go:55 args: []string{"/snap/juju/14127/bin/juju", "bootstrap", "--debug", "--config", "bootstrap-timeout=1800", "--to", "zone=FolderName/ClusterName", "--constraints", "zones=FolderName/ClusterName cores=8 mem=24G root-disk=100G", "--model-default", "/home/ubuntu/deploy/generated/model_defaults.yaml", "vsphere", "foundation_vsphere"}
14:01:50 DEBUG juju.cmd.juju.commands bootstrap.go:1165 authenticating with region "" and credential "vsphere" ()
14:01:50 DEBUG juju.cmd.juju.commands bootstrap.go:1293 provider attrs: map[]
14:01:52 INFO cmd authkeys.go:114 Adding contents of "/home/ubuntu/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
14:01:52 INFO cmd authkeys.go:114 Adding contents of "/home/ubuntu/.ssh/id_rsa.pub" to authorized-keys
14:01:52 DEBUG juju.cmd.juju.commands bootstrap.go:1363 preparing controller with config: map[agent-metadata-url: agent-stream:released apt-ftp-proxy: apt-http-proxy: apt-https-proxy: apt-mirror: apt-no-proxy: authorized-keys:ssh-rsa [REDACTED]
ssh-rsa [REDACTED]
 automatically-retry-hooks:true backup-dir: cloudinit-userdata: container-image-metadata-url: container-image-stream:released container-inherit-properties: container-networking-method: datastore:FC1660_MSG-SEC_04 default-series:bionic default-space: development:false disable-network-management:false egress-subnets: enable-os-refresh-update:true enable-os-upgrade:true fan-config: firewall-mode:instance ftp-proxy: http-proxy: https-proxy: ignore-machine-addresses:false image-metadata-url: image-stream:released juju-ftp-proxy: juju-http-proxy: juju-https-proxy: juju-no-proxy:127.0.0.1,localhost,::1 logforward-enabled:false logging-config: lxd-snap-channel:latest/stable max-action-results-age:336h max-action-results-size:5G max-status-history-age:336h max-status-history-size:5G name:controller net-bond-reconfigure-delay:17 no-proxy:127.0.0.1,localhost,::1 primary-network:NetworkName provisioner-harvest-mode:destroyed proxy-ssh:false resource-tags: snap-http-proxy: snap-https-proxy: snap-store-assertions: snap-store-proxy: snap-store-proxy-url: ssl-hostname-verification:true test-mode:false transmit-vendor-metrics:true type:vsphere update-status-hook-interval:5m uuid:7a81e787-3997-4689-8f52-1de76493b793]
14:01:52 INFO cmd bootstrap.go:775 Creating Juju controller "foundation_vsphere" on vsphere/DatacenterName
14:01:52 INFO juju.cmd.juju.commands bootstrap.go:844 combined bootstrap constraints: cores=8 mem=24576M root-disk=102400M zones=FolderName/ClusterName
14:01:52 DEBUG juju.environs.bootstrap bootstrap.go:308 model "controller" supports application/machine networks: false
14:01:52 DEBUG juju.environs.bootstrap bootstrap.go:310 network management by juju enabled: true
14:01:52 INFO cmd bootstrap.go:376 Loading image metadata
14:01:52 INFO cmd bootstrap.go:448 Looking for packaged Juju agent version 2.8.2 for amd64
14:01:52 INFO juju.environs.bootstrap tools.go:81 looking for bootstrap agent binaries: version=2.8.2
14:01:52 DEBUG juju.environs.tools tools.go:102 finding agent binaries in stream: "released"
14:01:52 DEBUG juju.environs.tools tools.go:104 reading agent binaries with major.minor version 2.8
14:01:52 DEBUG juju.environs.tools tools.go:112 filtering agent binaries by version: 2.8.2
14:01:52 DEBUG juju.environs.tools tools.go:115 filtering agent binaries by series: bionic
14:01:52 DEBUG juju.environs.tools tools.go:118 filtering agent binaries by architecture: amd64
14:01:52 DEBUG juju.environs.tools urls.go:138 trying datasource "keystone catalog"
14:01:57 DEBUG juju.environs.simplestreams simplestreams.go:692 using default candidate for content id "com.ubuntu.juju:released:tools" are {20161007 mirrors:1.0 content-download streams/v1/cpc-mirrors.sjson []}
14:01:57 INFO juju.environs.bootstrap tools.go:83 found 0 packaged agent binaries
14:01:57 INFO cmd bootstrap.go:472 No packaged binary found, preparing local Juju agent binary
14:01:57 DEBUG juju.environs.sync sync.go:311 Making agent binary tarball
14:01:57 DEBUG juju.environs.tools build.go:117 looking for: /snap/juju/14127/bin/juju
14:01:57 DEBUG juju.environs.tools build.go:194 checking: /snap/juju/14127/bin/jujud
14:01:57 INFO juju.environs.tools build.go:200 Found agent binary to upload (/snap/juju/14127/bin/jujud)
14:01:57 INFO juju.environs.tools build.go:202 target: /tmp/juju-tools884417937/jujud
14:01:57 INFO juju.environs.tools build.go:214 target jujuc: /tmp/juju-tools884417937/jujuc
14:01:57 INFO juju.environs.tools build.go:235 including versions file "/snap/juju/14127/bin/jujud-versions.yaml"
14:01:58 DEBUG juju.environs.tools versionfile.go:55 looking for sha256 36f8206be093103d8d03bb1d3ad8b2e030aca58f58d1fb7627e4727bd2fcd185
14:01:58 DEBUG juju.environs.tools build.go:325 using official version 2.8.2-bionic-amd64
14:01:58 DEBUG juju.environs.tools build.go:44 adding entry: &tar.Header{Typeflag:0x30, Name:"jujuc", Linkname:"", Size:8826880, Mode:493, Uid:0, Gid:0, Uname:"ubuntu", Gname:"ubuntu", ModTime:time.Time{wall:0x2c6f40be, ext:63734824917, loc:(*time.Location)(0x6480080)}, AccessTime:time.Time{wall:0x2c6f40be, ext:63734824917, loc:(*time.Location)(0x6480080)}, ChangeTime:time.Time{wall:0x2c6f40be, ext:63734824917, loc:(*time.Location)(0x6480080)}, Devmajor:0, Devminor:0, Xattrs:map[string]string(nil), PAXRecords:map[string]string(nil), Format:0}
14:01:58 DEBUG juju.environs.tools build.go:44 adding entry: &tar.Header{Typeflag:0x30, Name:"jujud", Linkname:"", Size:125026304, Mode:493, Uid:0, Gid:0, Uname:"ubuntu", Gname:"ubuntu", ModTime:time.Time{wall:0x2bf52e8a, ext:63734824917, loc:(*time.Location)(0x6480080)}, AccessTime:time.Time{wall:0x2bf52e8a, ext:63734824917, loc:(*time.Location)(0x6480080)}, ChangeTime:time.Time{wall:0x2bf52e8a, ext:63734824917, loc:(*time.Location)(0x6480080)}, Devmajor:0, Devminor:0, Xattrs:map[string]string(nil), PAXRecords:map[string]string(nil), Format:0}
14:02:05 DEBUG juju.environs.tools build.go:44 adding entry: &tar.Header{Typeflag:0x30, Name:"jujud-versions.yaml", Linkname:"", Size:119, Mode:420, Uid:0, Gid:0, Uname:"ubuntu", Gname:"ubuntu", ModTime:time.Time{wall:0x2c6f40be, ext:63734824917, loc:(*time.Location)(0x6480080)}, AccessTime:time.Time{wall:0x2c6f40be, ext:63734824917, loc:(*time.Location)(0x6480080)}, ChangeTime:time.Time{wall:0x2c6f40be, ext:63734824917, loc:(*time.Location)(0x6480080)}, Devmajor:0, Devminor:0, Xattrs:map[string]string(nil), PAXRecords:map[string]string(nil), Format:0}
14:02:05 INFO juju.environs.sync sync.go:343 using official agent binary 2.8.2-bionic-amd64 (37807kB)
14:02:05 INFO cmd bootstrap.go:535 Starting new instance for initial controller
14:02:05 DEBUG juju.provider.vmware client.go:125 relative folderPath "cloudname" found, join with DC vm folder "/DatacenterName/vm" now
14:02:06 INFO cmd bootstrap.go:177 Launching controller instance(s) on vsphere/DatacenterName...
14:02:06 ERROR juju.cmd.juju.commands bootstrap.go:795 failed to bootstrap model: cannot start bootstrap instance: availability zone "FolderName/ClusterName" not found
14:02:06 DEBUG juju.cmd.juju.commands bootstrap.go:796 (error details: [{/build/snapcraft-juju-35d6cf/parts/juju/src/cmd/juju/commands/bootstrap.go:889: failed to bootstrap model} {/build/snapcraft-juju-35d6cf/parts/juju/src/environs/bootstrap/bootstrap.go:633: } {/build/snapcraft-juju-35d6cf/parts/juju/src/environs/bootstrap/bootstrap.go:539: } {/build/snapcraft-juju-35d6cf/parts/juju/src/provider/common/bootstrap.go:57: } {/build/snapcraft-juju-35d6cf/parts/juju/src/provider/common/bootstrap.go:220: cannot start bootstrap instance} {/build/snapcraft-juju-35d6cf/parts/juju/src/provider/common/bootstrap.go:307: } {/build/snapcraft-juju-35d6cf/parts/juju/src/provider/vsphere/environ_availzones.go:154: } {/build/snapcraft-juju-35d6cf/parts/juju/src/provider/vsphere/environ_instance.go:120: } {/build/snapcraft-juju-35d6cf/parts/juju/src/provider/vsphere/environ_availzones.go:173: availability zone "FolderName/ClusterName" not found}])
14:02:06 DEBUG juju.cmd.juju.commands bootstrap.go:1471 cleaning up after failed bootstrap
14:02:06 INFO juju.provider.common destroy.go:21 destroying model "controller"
14:02:06 INFO juju.provider.common destroy.go:32 destroying instances
14:02:06 INFO juju.provider.common destroy.go:56 destroying storage
14:02:06 DEBUG juju.provider.vmware client.go:125 relative folderPath "cloudname/Juju Controller (*)/Model \"controller\" (7a81e787-3997-4689-8f52-1de76493b793)" found, join with DC vm folder "/DatacenterName/vm" now
14:02:06 DEBUG juju.provider.vmware client.go:160 no VMs matching path "cloudname/Juju Controller (25b5fae0-d10c-4c05-8819-6bd2728297ec)/Model \"*\" (*)/*"
14:02:06 DEBUG juju.provider.vmware client.go:125 relative folderPath "cloudname/Juju Controller (25b5fae0-d10c-4c05-8819-6bd2728297ec)" found, join with DC vm folder "/DatacenterName/vm" now
14:02:06 INFO cmd supercommand.go:544 command finished

FWIW, I have added some traces in the locally compiled juju client and found out that env.AvailabilityZones(ctx) in [1] returns empty 'zones'. It's not nil, it's just an empty array. Similarily env.zones in [2] is also an empty array.

1. https://github.com/juju/juju/blob/92301e2ff1ef993e35886c5f34a1b85edc7135c9/provider/vsphere/environ_availzones.go#L164
2. https://github.com/juju/juju/blob/92301e2ff1ef993e35886c5f34a1b85edc7135c9/provider/vsphere/environ_availzones.go#L66

[Przemyslaw Hausman (phausman)]

Subscribed field-critical as this is blocking customer deployment.

[Pen Gale (pengale)]

Marked as duplicate of #1884490, and available in the edge channel

Can you confirm that the edge snap fixes your issue?

[Pen Gale (pengale)]

Whoops. Above should read "Marked as duplicate of #1884490, which is fix committed, and available in the edge channel."

[Pedro Guimarães (pguimaraes)]

Hi petevg, we are actually compiling juju's directly from develop and that does not resolve this issue.

Besides, the #1884490 is related to Datacenter Folders, whereas this is related to cluster folders:

/<dc-folder>/dc-name/...
/<dc-folder>/dc-name/<cluster-folder>/cluster-name

That means that former will appear at bootstrap, the later (cluster folders) can be seen on placement directive.

Therefore, I am removing the duplicate.

[Pedro Guimarães (pguimaraes)]

The main issue here is that env.zones on the files cited by Przemyslaw return as empty array, not "nil" pointer. Therefore, availZone code is never ran.

[Pedro Guimarães (pguimaraes)]

Digging more into this issue, I can see that env.zones is declared initially on: https://github.com/juju/juju/blob/develop/provider/vsphere/session.go#L33

On core/network, type AvailabiliyZones is a list: https://github.com/juju/juju/blob/346cd856873ac33c093993c423829b8bfc3517c7/core/network/zone.go#L19

Which will be initialized to (an empty) list of AvailabilityZone. Therefore, this value will never be nil. Proposed a quick fix where we check for both either way on:
https://github.com/phvalguima/juju/blob/vsphere-cluster-folder

Also, on method ComputeResources (internal/vsphereclient) I am trying to add some logic to deal with mo.Folder type (a clusterfolder) by appending Folder's content to the list we are iterating over:
https://github.com/phvalguima/juju/blob/vsphere-cluster-folder/provider/vsphere/internal/vsphereclient/client.go#L212

The extra logging will go away once I am sure the loop is doing what I expect it to do.

One weird thing, automated tests broke on:
 Install Mongo Dependencies: ubuntu-latest
Don't think this is related to my code...

[Ian Booth (wallyworld)]

The availability zone fix just needs a one line change

https://github.com/juju/juju/blob/develop/provider/vsphere/environ_availzones.go#L66

replace

if env.zones == nil {

with

if len(env.zones) == 0 {

[Przemyslaw Hausman (phausman)]

Hi @wallyworld, the update you've suggested does not work for me. Please see how I adjusted the code and what was the resulting output below:

 65 // AvailabilityZones is part of the common.ZonedEnviron interface.
 66 func (env *sessionEnviron) AvailabilityZones(ctx context.ProviderCallContext) (network.AvailabilityZones, error) {
 67 logger.Debugf("LP #1894236: env: %+v", env)
 68 // if env.zones == nil { // LP #1894236
 69 if len(env.zones) == 0 {
 70 computeResources, err := env.client.ComputeResources(env.ctx)
 71 logger.Debugf("LP #1894236: computeResources: %+v", computeResources)
 72 if err != nil {
 73 HandleCredentialError(err, env, ctx)
 74 return nil, errors.Trace(err)
 75 }
 76 var zones network.AvailabilityZones
 77 logger.Debugf("LP #1894236: zones: %+v", zones)
 78 for _, cr := range computeResources {
 79 if cr.Summary.GetComputeResourceSummary().EffectiveCpu == 0 {
 80 logger.Debugf("skipping empty compute resource %q", cr.Name)
 81 continue
 82 }
 83 pools, err := env.client.ResourcePools(env.ctx, cr.Name+"/...")
 84 logger.Debugf("LP #1894236: pools: %+v", pools)
 85 if err != nil {
 86 HandleCredentialError(err, env, ctx)
 87 return nil, errors.Trace(err)
 88 }
 89 for _, pool := range pools {
 90 zone := &vmwareAvailZone{
 91 r: *cr,
 92 pool: pool,
 93 }
 94 logger.Tracef("zone: %q", zone.Name())
 95 zones = append(zones, zone)
 96 }
 97 }
 98 env.zones = zones
 99 }
100 logger.Debugf("LP #1894236: AvailabilityZones returns: %+v", env.zones)
101 return env.zones, nil
102 }

JUJU_LOGGING_CONFIG="<root>=DEBUG;unit=DEBUG;juju.provider.vsphere=TRACE" _build/linux_amd64/bin/juju bootstrap \
  --debug \
  --build-agent \
  --config bootstrap-timeout=1800 \
  --to zone=FolderName/ClusterName \
  --constraints 'zones=FolderName/ClusterName cores=8 mem=24G root-disk=100G' \
  --model-default /home/ubuntu/deploy/generated/model_defaults.yaml \
  vsphere foundation_vsphere
[...]
06:07:00 DEBUG juju.provider.vmware environ_availzones.go:67 LP #1894236: env: &{environ:0xc000524a80 ctx:0xc00005c100 client:0xc000593380 zones:[]}
06:07:00 DEBUG juju.provider.vmware environ_availzones.go:71 LP #1894236: computeResources: []
06:07:00 DEBUG juju.provider.vmware environ_availzones.go:77 LP #1894236: zones: []
06:07:00 DEBUG juju.provider.vmware environ_availzones.go:100 LP #1894236: AvailabilityZones returns: []

[Przemyslaw Hausman (phausman)]

The code with the above comment (#9) with a better formatting:
https://pastebin.ubuntu.com/p/6VJZDzSTz6/

[Przemyslaw Hausman (phausman)]

This is the structure reported by govc:

$ govc find | grep ClusterName
/DatacenterName/host/FolderName/ClusterName
/DatacenterName/host/FolderName/ClusterName/Resources
/DatacenterName/host/FolderName/ClusterName/host1
/DatacenterName/host/FolderName/ClusterName/host2

[Ian Booth (wallyworld)]

 My suggested fix wasn't for the whole issue, just this code change made in comment #7

        invalidZones := false
        if env.zones == nil {
  invalidZones = true
        // else if to avoid any nil pointer errors
        } else if len(env.zones) == 0 {
  invalidZones = true
        }
 if invalidZones {

All the above can be replaced with

        if len(env.zones) == 0 {

[Ian Booth (wallyworld)]

Is it possible to turn on TRACE logging for "juju.provider.vmware"

That will print the exact folder names juju is trying to match on and will make it easier to see how to adjust the code.

[Przemyslaw Hausman (phausman)]

Hi Ian, thanks for helping. I added some more traces in the code and came up with a quick and dirty handling of mo.Folder. Please see this commit: https://github.com/phausman/juju/commit/f1248d917edd849f8c83ab4c0fd141d9a4d086c0

The log from running `juju bootstrap` with locally compiled binary (having the above patch applied) is here: https://pastebin.ubuntu.com/p/5yyWFG59ff/

It still does not work because ResourcePools() function is not able to find compute resources if the path is relative. By making the path absolute (I temporarily hardcoded it) I was able to finally bootstrap controllers.

Here's the comment on making the path absolute: https://github.com/phausman/juju/commit/f1248d917edd849f8c83ab4c0fd141d9a4d086c0#diff-cd4692c0c22f02147a7473d15a4f1ca8R88

What do you think?

[Ian Booth (wallyworld)]

The changes seem to be heading in the right direction.
We'll have to work out what API(s) to use to get the correct parent folder path to turn the relative path into an absolute one.
The issue right now is we might not really have a fast vsphere we can easily set up the correct folder structure on to test with and figure out the APIs required. we're also a little stretched resources wise. What's the required time frame for a fix? Can we get access to the vsphere you are using as that will make things a lot easier to work with.

[Przemyslaw Hausman (phausman)]

Hi Ian, the fix is needed for the ongoing customer deployment. The plan is to finish the deployment by 22nd of September. During this time we have access to vsphere and we can test potential patches.

As to the patch, I added some more logic to pull the absolute path for compute resource if its parent is a Folder. Please see here: https://github.com/juju/juju/commit/bd8c3db17822b29e4df8fb45ac503c95119422b3

It actually solves my corner case and I can now bootstrap the controller. However it's far from tested, production-ready, generic solution.

For example, this patch probably won't work if I had multiple folders with same name clusters, like:

/DatacenterName/host/Folder-1/ClusterName
/DatacenterName/host/Folder-2/ClusterName

This is because availZone function tests z.Name(), that in my case is always "ClusterName/Resources". It does not take into account parent folder's name. (https://github.com/phausman/juju/blob/develop/provider/vsphere/environ_availzones.go#L169)

And if I run juju bootstrap with `--to FolderName/ClusterName/Resources --constraints zones=FolderName/ClusterName/Resources`, bootstrapping fails because "FolderName/ClusterName/Resources" zone does not exist (but "ClusterName/Resources" does).

I'm happy to help making this patch production ready. Just let me know how can I help.

[Ian Booth (wallyworld)]

The end goal is to have a PR with unit tests, preferably validated by on site testing. There may be enough info on this bug for us to make the necessary changes. We can look when there's a free resource, hopefully soon. If you need something immediately, we'd be happy if you were to take what you have done and make it more generic and useful for your case as well as the default case. We can write tests and help clean up any issues if the core logic is in place.

[Przemyslaw Hausman (phausman)]

Hi Ian, do you have enough input data to work on a potential permanent patch when you have a chance / free resource? I'm happy to collect any specific information that could help fixing it as long as I have access to the target environment. I'm thinking eg. juju traces, govc outputs, vSphere screenshots, etc.

[Przemyslaw Hausman (phausman)]

FYI, I've created a PR on the initial patch, so that we can work on it together directly on github.

https://github.com/juju/juju/pull/11991

[Ian Booth (wallyworld)]

Thank you for taking the time to do the PR. Much appreciated. We will take a look asap next week and help get it ready to land.

[Michael Skalka (mskalka)]

Moving this down to high as a workaround was found.

[Ben Hoyt (benhoyt)]

Just an update that we've now merged the fix for this into the 2.8 branch (https://github.com/juju/juju/pull/12012), and it should be on the 2.8 edge snap in a couple of hours. Thanks Przemyslaw Hausman for your initial patch -- that was really helpful.

This fixes the issue in our tests (host in a folder), though our Vsphere datacenter is probably not set up quite the same as the customer's, so it'd be great if you can pull the 2.8 edge snap and test this for us. If you find anything or have any concerns, please let us know ASAP. Hopefully this will get into the 2.8.4 final later this week.

[Przemyslaw Hausman (phausman)]

hi @benhoyt, thank you for the patch. I have just tested it on the target environment and can confirm that it solves the issue. Thank you!

FYI, I did use the following options for bootstrapping the controller:

--to zone=SomeFolder/ClusterName
--constraints 'zones=SomeFolder/ClusterName cores=8 mem=24G root-disk=100G'

[Ben Hoyt (benhoyt)]

That's excellent Przemyslaw - thanks for the confirmation.

[Erlon R. Cruz (sombrafam)]

Hi folks,

Changing this to private as there's tons of details on the customer deployment.

[Erlon R. Cruz (sombrafam)]

Sorry, changed the wrong bug.