Cannot bootstrap with vSphere data center under a folder

[Nobuto Murata]

The bottom line of the error is:
failed to bootstrap model: folder path "/MyDC/vm/juju-root" not found
when we have a directory structure as "/MyFolder/MyDC/vm/juju-root".

[Environment]
vSphere 6.5

$ juju version
2.8.0-xenial-amd64

$ govc ls
/MyFolder/MyDC/vm
/MyFolder/MyDC/network
/MyFolder/MyDC/host
/MyFolder/MyDC/datastore

$ govc find | grep juju-root
/MyFolder/MyDC/vm/juju-root

[Juju config]
$ juju show-cloud vsp-cloud
Client cloud "vsp-cloud":

defined: local

type: vsphere
auth-types: [userpass]
endpoint: myendpoint.example.com
credential-count: 3
regions:
  MyFolder/MyDC: {}

$ juju show-credentials vsp-cloud vsp-cloud-creds
credentials:
  vsp-cloud:
    vsp-cloud-creds:
      auth-type: userpass
      password: ********
      user: ********
      vmfolder: juju-root

[Bootstrap]
$ juju bootstrap vsp-cloud vsp-controller \
    --config primary-network=MyNetwork \
    --config datastore=MyDatastore \
    --credential vsp-cloud-creds \
    --debug

Full output: https://pastebin.canonical.com/p/QmWpNnCZjK/

excerpt:
...
17:43:51 INFO cmd bootstrap.go:762 Creating Juju controller "vsp-controller" on vsp-cloud/MyFolder/MyDC
...
17:43:54 INFO cmd bootstrap.go:513 Starting new instance for initial controller
17:43:55 DEBUG juju.provider.vmware client.go:121 relative folderPath "juju-root" found, join with DC vm folder "/MyDC/vm" now
17:43:55 DEBUG juju.provider.vmware client.go:121 relative folderPath "juju-root" found, join with DC vm folder "/MyDC/vm" now
17:43:55 ERROR juju.cmd.juju.commands bootstrap.go:782 failed to bootstrap model: folder path "/MyDC/vm/juju-root" not found
17:43:55 DEBUG juju.cmd.juju.commands bootstrap.go:783 (error details: [{/build/snapcraft-juju-03af7d/parts/juju/src/cmd/juju/commands/bootstrap.go:857: failed to bootstrap model} {/build/snapcraft-juju-03af7d/parts/juju/src/environs/bootstrap/bootstrap.go:609: } {/build/snapcraft-juju-03af7d/parts/juju/src/environs/bootstrap/bootstrap.go:517: } {/build/snapcraft-juju-03af7d/parts/juju/src/provider/vsphere/environ.go:143: } {/build/snapcraft-juju-03af7d/parts/juju/src/provider/vsphere/environ.go:162: } {/build/snapcraft-juju-03af7d/parts/juju/src/provider/vsphere/internal/vsphereclient/client.go:349: } {/build/snapcraft-juju-03af7d/parts/juju/src/provider/vsphere/internal/vsphereclient/client.go:130: folder path "/MyDC/vm/juju-root" not found}])
17:43:55 DEBUG juju.cmd.juju.commands bootstrap.go:1428 cleaning up after failed bootstrap

[Nobuto Murata (nobuto)]

I cannot say how common having one folder above a data center in vSphere setup, but I saw the structure with two customers so far.

[Pedro Guimarães (pguimaraes)]

So, the interesting part here is that govc ls is returning the DC folder, while FindFolder does not find it.

I wonder if this is an issue with "finder" method:
https://github.com/juju/juju/blob/b911edb7a026e0d97c66dc0b9f15f53c71a1450d/provider/vsphere/internal/vsphereclient/client.go#L137

Or datacenter.Folders() is not returning the entire path:
https://github.com/juju/juju/blob/develop/provider/vsphere/internal/vsphereclient/client.go#L110

Maybe we should be using another method to recover DC's absolute path, such as:
https://github.com/vmware/govmomi/blob/master/find/finder.go#L162

[Pedro Guimarães (pguimaraes)]

Going a little deeper, FindFolder seems to be the right place to run any changes since createvm.go also uses it to recover VMs' RootVMFolder before running the actual creation command:
https://github.com/juju/juju/blob/develop/provider/vsphere/internal/vsphereclient/createvm.go#L319

It seems govmomi is not returning the entire absolute path with:
https://github.com/juju/juju/blob/develop/provider/vsphere/internal/vsphereclient/client.go#L120

My suggestion is to add an extra check, as such (after L120 check):
if !strings.HasPrefix(folderPath, fi.datacenterPath(ctx, datacenter)) {
 c.logger.Debugf("Discovered a DC folder, adding it tot he begining of the path")
        // ADD IT TO THE BEGINNING OF THE PATH
}

I am not sure that datacenterPath will return the entire root path, with DC folder.

[Nobuto Murata (nobuto)]

We've set "vmfolder" as /MyFolder/MyDC/vm/juju-root to try to workaround the issue.

Then, we got:

19:32:40 DEBUG juju.provider.vmware client.go:121 relative folderPath "/MyFolder/MyDC/vm/juju-root" found, join with DC vm folder "/MyDC/vm" now
19:32:40 ERROR juju.cmd.juju.commands bootstrap.go:782 failed to bootstrap model: folder path "/MyDC/vm/MyFolder/MyDC/vm/juju-root" not found

So "/MyFolder/MyDC/vm/juju-root" wasn't recognized as an absolute path.

[Nobuto Murata (nobuto)]

> We've set "vmfolder" as /MyFolder/MyDC/vm/juju-root to try to workaround the issue.
>
> Then, we got:
>
> 19:32:40 DEBUG juju.provider.vmware client.go:121 relative folderPath "/MyFolder/MyDC/vm/juju-root" found, join with DC vm folder "/MyDC/vm" now
> 19:32:40 ERROR juju.cmd.juju.commands bootstrap.go:782 failed to bootstrap model: folder path "/MyDC/vm/MyFolder/MyDC/vm/juju-root" not found
>
> So "/MyFolder/MyDC/vm/juju-root" wasn't recognized as an absolute path.

We tried a custom build of Juju locally with the following diff.
https://github.com/nobuto-m/juju/commit/26ab58c38fb3bc41905f8085c6939bc3628181b5
to ignore/by-pass the condition of "if !strings.HasPrefix(folderPath, dcfolders.VmFolder.InventoryPath)".

Then, we got an expected output as:
WARN juju.provider.vmware client.go:122 forcibly using absolute path "/MyFolder/MyDC/vm/juju-root" even though not matching with DC vm folder "/MyDC/vm"

It's not a permanent fix, but at least it unblocked the bootstrap process.

[Ian Booth (wallyworld)]

I think the intent of the fix is good - if the "vmfolder" attribute is set to an absolute folder path, it's valid to try and use that. If not, then interpret as a path relative to the DC folder.

We can go 2 ways - enforce a relative path by requiring a "./" prefix, or signal an absolute path with a "/" prefix. The latter seems best.

We'll get a fix landed.

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/11800

[Ian Booth (wallyworld)]

This fix is now in the 2.8 edge snap, would be great to have it tested on site and get feedback that it solves the problem.

[Nobuto Murata (nobuto)]

Moving back to new since the original issue reported here hasn't been fixed. We've implemented a workaround to specify a full path instead though.

[John A Meinel (jameinel)]

So the original issue was "cannot bootstrap if the DC is under a folder" which you can now bootstrap when the DC is under a folder by specifying an absolute path.

What is the original issue that hasn't been fixed?

[Nobuto Murata (nobuto)]

I had a second thought, and I'm closing this bug since the bootstrap itself works with the absolute path workaround implemented as a part of this bug.

I will file a separate bug explaining the UX issue with vSphere datacenter under a folder.

[Nobuto Murata (nobuto)]

Commented to the vsphere provider doc as the first step:
https://discourse.juju.is/t/using-vmware-vsphere-with-juju/1099/13