Juju bootstrap vmware vsphere not working with vsan datastore

Bug #1800940 reported by Kent Williams on 2018-10-31
46
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Kubernetes Master Charm
Undecided
Unassigned
juju
High
Tim McNamara

Bug Description

I am unable to successfully bootstrap a vmware vsphere controller using vsan as the datastore. Bootstrapping works fine if vsan is not used as datastore.

I came across another person that encountered the exact same issue here: https://askubuntu.com/questions/1020107/juju-bootstrap-not-working-with-vsan?newreg=d3c6766987f74e15a59866e2621f6c5f

When the VM attempts to boot, network interfaces attempt a PXE boot, but it is never able to boot from the disk file.

-Kent

Kent Williams (k3nt) on 2018-10-31
summary: - Juju bootstrap not working with vsan
+ Juju bootstrap vmware vsphere not working with vsan datastore
Ian Booth (wallyworld) on 2018-11-01
Changed in juju:
milestone: none → 2.5.0
status: New → Triaged
importance: Undecided → High
Kent Williams (k3nt) on 2018-11-01
description: updated
Kent Williams (k3nt) wrote :

Is there any more information I can provide to assist in this bug resolution?

Changed in juju:
milestone: 2.5.0 → 2.5-beta2
Ian Booth (wallyworld) on 2018-11-30
Changed in juju:
milestone: 2.5-beta2 → 2.5-beta3
Changed in juju:
milestone: 2.5-beta3 → 2.5-rc1
Watteel (piwi3910) wrote :

same issue here, you are not alone

Calvin Hartwell (calvinh) wrote :

Blocking some customer deployments (https://bugs.launchpad.net/juju/+bug/1807953).

Changed in juju:
milestone: 2.5-rc1 → 2.5.1
Kris Applegate (kris-applegate) wrote :

I'd love to assist with resolving this bug. I'm relatively new to juju, but this is blocking some of the work we're doing in the Dell EMC Customer Solution Centers. I have access to environments to assist in troubleshooting, but I'll need help. Please let me know what I can do.

Kent Williams (k3nt) wrote :

Is there any indication that this will actually make it into the 2.5.1 release?

Ian Booth (wallyworld) on 2019-01-09
tags: added: vsphere-provider
Ian Booth (wallyworld) on 2019-01-28
Changed in juju:
milestone: 2.5.1 → 2.5.2
Changed in juju:
milestone: 2.5.2 → 2.5.3
Daniel Bidwell (bidwell) wrote :

juju bootstrap downloads a *.vmdk file. The cdrom points to an ovfenv-juju-*.iso but fails to boot from it and drops down to the vmware pxe boot which of course times out and failes.

Could the fact that it is using vmware hardware version 10 instead of version 14 have anything to do with it?

Daniel Bidwell (bidwell) wrote :

upgrading the vmware hardware version to 11,13, or 14 does not make any difference in it's behaviour.

Changed in juju:
milestone: 2.5.3 → 2.5.4
Changed in juju:
milestone: 2.5.4 → 2.5.5
Changed in juju:
milestone: 2.5.6 → 2.5.8
Changed in juju:
milestone: 2.5.8 → 2.5.9
Tim McNamara (tim-clicks) wrote :

Adding kubernetes-master-charm as it is marked as affecting a duplicate bug report.

George Kraft (cynerva) wrote :

I'm marking this as invalid for kubernetes-master, since the issue occurs during Juju bootstrap or machine deployment, before kubernetes-master ever runs.

Changed in charm-kubernetes-master:
status: New → Incomplete
status: Incomplete → Invalid
Kent Williams (k3nt) wrote :

Agreed, don't see the kubernetes-master charm being relevant for this bug.

Thanks for pointing me to the duplicate issue.

Is there anyway I can get the definitions for the VMs that juju tries to start? Because then I could possible help with debugging the issue from the VMware end.

Is there anything else I can do to help out with debugging this?

Changed in juju:
status: Triaged → In Progress
Tim McNamara (tim-clicks) wrote :

Thanks for the offer of assistance Frederik. I missed the email notification when it came through and apologise for not updating this ticket sooner.

I have been able to identify the change that has introduced this problem[0]. As context, Juju does quite a complicated dance when creating a virtual machine. We want to make sure that we're using a Ubuntu certified cloud image[1], but we also wish to enable some parameters that may be specified by configuration parameters. So we download the image, tweak some parameters in its metadata file, then upload the disk along with the new metadata to VSAN.

It looks as though the metadata file that Juju generates is incorrect and the VMDK uploaded into the vSAN does not get imported correctly. I'll update the ticket when I've discovered where the fault lies.

[0] https://github.com/juju/juju/pull/7856
[1] https://cloud-images.ubuntu.com/

Brilliant, let me know if there's anything I can assist you with, testing maybe?

Tim McNamara (tim-clicks) wrote :

Good news. I was able to make some progress late last week and was able to bootstrap Juju onto a vSphere/ESXi instance backed by vSAN.

I'm cautiously optimistic that we'll be able to release this fix in the next few weeks. There is still some clean up work to do before it is merged.

Thanks for the offer to test Frederik. If you're still interested, you can build my development branch[0] and experiment. Things should just work.

Here were the two commands that successfully ran for me:

    juju bootstrap vsphere
    juju deploy ubuntu

[0] https://github.com/timClicks/juju/tree/develop-vsphere--vmdk-vsan

Changed in juju:
assignee: nobody → Tim McNamara (tim-clicks)
milestone: 2.5.9 → 2.7-beta1

This is wonderful news. Is there any chance that they can update the
vsphere images from hardware version 10 to 14?

On Mon, 2019-07-22 at 01:06 +0000, Tim McNamara wrote:
> Good news. I was able to make some progress late last week and was
> able
> to bootstrap Juju onto a vSphere/ESXi instance backed by vSAN.
>
> I'm cautiously optimistic that we'll be able to release this fix in
> the
> next few weeks. There is still some clean up work to do before it is
> merged.
>
> Thanks for the offer to test Frederik. If you're still interested,
> you
> can build my development branch[0] and experiment. Things should just
> work.
>
> Here were the two commands that successfully ran for me:
>
> juju bootstrap vsphere
> juju deploy ubuntu
>
> [0] https://github.com/timClicks/juju/tree/develop-vsphere--vmdk-vsan
>
> ** Changed in: juju
> Assignee: (unassigned) => Tim McNamara (tim-clicks)
>
> ** Changed in: juju
> Milestone: 2.5.9 => 2.7-beta1
>
--
Daniel R. Bidwell | <email address hidden>
Sr. Systems Architect | chief Information Security Officer
Andrews University | Information Technology Services
If two always agree, one of them is unnecessary.
Karma is getting what you deserve,
mercy is not getting what you deserve
grace is getting what you do not deserve.
In theory, theory and practice are the same.
In practice, they are not.

Fantastic! I'll give it a try tomorrow!

Thank you so much for looking in to this problem.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers