VMWare Guest OS Customization will fail for Ubuntu 18.04 Server LiveCD

[vmware-gos-Yuhua]

This issue occurs when:
1) There is conflict between cloud-init and VMware guest OS customization.
   Customization fails if cloud-init present for Ubuntu 18.04 server liveCD edition. This affects only the liveCD server edition, but not the server/desktop edition if installed into disk.

   And guest customization package is cleaned after powering on the customized virtual machine, customization process can NOT complete correctly.

2) Sometimes systemd services execution sequence on booting virtual machine causes failure.

[vmware-gos-Yuhua (yhzou)]

Currently we have workaround for it.

Workaround:

1. set cloud-init or perl scripts as the customization engine
   1) if you want to set cloud-init as the customization engine by:
      Set “disable_vmware_customization: false" in "/etc/cloud.cfg"

   2) if you want to set perl script as customization engine, you should disable or remove
      cloud-init

      Disable cloud-init service by running this command:
      sudo touch /etc/cloud/cloud-init.disabled

      Remove cloud-init package and purge the config files by running these commands:
      sudo apt-get purge cloud-init

2. Open the /usr/lib/tmpfiles.d/tmp.conf file.
   Go to the line 11 and add the prefix #.

   or example:
   #D /tmp 1777 root root -

3. If you have open-vm-tools installed, open the /lib/systemd/system/open-vm-tools.service file.
   Add “After=dbus.service” under [Unit].

[Joshua Powers (powersj)]

Can you provide additional details as to what caused the need for step 3 of the workaround (e.g. modifying open-vm-tools service file)?

What issue was seen?
Any logs that show the issue?

[Christian Ehrhardt  (paelzer)]

I did not feel the discussion was over, why did you close that asfak?
It the KB article updated, is there anything than can be done in Ubuntu (see Joshs questions in comment #2).

[Pengpeng Sun (pengpengs)]

As to Joshua's question:
Customization of hostname is using "hostnamectl set-hostname theNewHostName" which requires 'dbus.service' is running. We found the dbus.service might not be running when perl/cloud-init executes the hostnamectl command on Ubuntu18.04 (Open-vm-tools v10.2.0).
As the workaround, We publish this KB to tell customer about this issue. Here is a thread that customer met and workaround this issue: https://github.com/vmware/open-vm-tools/issues/240

[Joshua Powers (powersj)]

Per the GitHub issue [1] and the published workaround [2] does open-vm-tools in Bionic and newer releases need to add the following to the open-vm-tools.service file?

"After=dbus.service"

[1] https://github.com/vmware/open-vm-tools/issues/240
[2] https://kb.vmware.com/s/article/56409

[Pengpeng Sun (pengpengs)]

Hi Joshua,

Sorry for the late response.
Yes, adding "After=dbus.service" to open-vm-tools.service file is the workaround.

[Christian Ehrhardt  (paelzer)]

Thanks for the info Pengpeng.
The problem with that is that it will make cloud-init rather late.

Currently (this is the latest state after many fixup for just those reasons of being too early/too late):

Note: Disambiguation of service names used below:

vgauth - open-vm-tools.vgauth.service
vmtoolsd - open-vm-tools.service
fs - systemd-remount-fs.service
tmp - systemd-tmpfiles-setup.service
cloud-init-l - cloud-init-local.service
cloud-init - cloud-init.service

 fs -> vmtoolsd -> cloud-init-l
               ^
 tmp --|
 vgauth --|
 apparmor --|

The problem is that all those services are meant to be super early and therefore have DefaultDependencies=No.

Dbus on the other hand is not that super early and adding
  After=dbus.service
Would move it way back in the initialization order which also you don't wan't AFAIK (bug 1667831 to have open-vm-tools before cloud-init-local was by Sankar from VMWare).

To be sure about it let me ask two questions:
- are your scripts triggered by cloud init
- if so in which config module/phase as I'm sure I have done cloud-init driven hostname changed?
- OR are your scripts triggered by open-vm-tools directly?

@cloud-init Team:
- if we'd end up to make open-vm-tools after dbus.service that would move cloud-init-local.service much later, could you outline the problems with that OR if you are ok with it state so?

Furthermore lets talk in the call today to make sure we are on the same page for this.

[Christian Ehrhardt  (paelzer)]

I was discussing with Ryan about it and as expected just adding "After=dbus.service" to open-vm-tools.service will for sure trigger random behavior.
This is will create a unresolvable dependency because:
- dbus has DefaultDependencies which makes it after sysinit.target
- Cloud-init-local needs to be early to make changes required before
  certain targets start e.g. Before=sysinit.target
- But open-vm-tools is before cloud-init-local
That is a conflicting requirement and systemd will kick one out of the loop at best.

We would be interested if you could share how to set up VMware to make use of that hostnamectl so that we can see and debug the case on our own as well.
And furthermore the interactions with cloud-init.

Worth a try might be to try making open-vm-tools "After=dbus.socket" which would provide the socket that hostnamectl needs but NOT needing it to fully complete.
If it does not work let us know and we can give up that approach.
If it works for your current needs we still need to ensure it does not inherit the dependencies, after a boot with that modification please check the journal if you can find any message like:
  "Breaking ordering cycle by deleting"
Example from another case
 "Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with var-lib-mysql.mount/start"

In general it seems vmtoolsd is doing many things.
It provides services e.g. through vmware-rpctool which some tools need to use early (so it needs to be early)
And it tries to do configuration e.g. hostnamectl which needs to be late.
Cloud init was split into stages for the same reason and it appears that the current issue vmtoolsd being torn apart having to be early & late at the same time is due to similar issues.
You might consider (surely a longer effort) to split things up as well to provide early services early and later functions late without those conflicts.

[Pengpeng Sun (pengpengs)]

Thanks a lot for the info, Christian.
Let me answer your questions firstly:
- are your scripts triggered by cloud init
NO.
- if so in which config module/phase as I'm sure I have done cloud-init driven hostname changed?
- OR are your scripts triggered by open-vm-tools directly?
VMware Guest OS Customization works with either pure 'perl' scripts or cloud-init as the customization engine.
For 'perl' scripts customization, scripts are triggered by open-vm-tools.service, the 'hostnamectl' is the cmd in scripts to set hostname. And 'hostnamectl' has dependency on 'dbus.service'.
For 'cloud-init' customization, only 2 config files are copied to a folder by open-vm-tools.service, and then cloud-init-local.service read the config file from that folder to do the customization. This is the reason of "bug 1667831 to have open-vm-tools before cloud-init-local was by Sankar from VMWare".
And AFAIK, cloud-init is not using 'hostnamectl' to set hostname for Ubuntu distro. so when do the 'cloud-init' customization on Ubuntu, there is no dependency on 'dbus.service'.

And Yes, I also noticed the cycle dependency was introduced if add 'After=dbus.service' to 'open-vm-tools.service' when cloud-init*.services are also enabled.(https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1796875/comments/8)
So the 'after=dbus.service' should only be added when using perl scripts as customization engine.

see details in the workaround KB for Ubuntu18.04 Live Server https://kb.vmware.com/s/article/54986
For Set cloud-init as the customization engine:
   no need add 'after=dbus.service' to open-vm-tools.service
For Set perl script as the customization engine:
   need add 'after=dbus.service' to open-vm-tools.service
   and also disable or remove cloud-init

[Christian Ehrhardt  (paelzer)]

Thanks for the confirmations!

The service can only have one set of dependencies, since it is the same package no matter if some component on your deployment chain chooses between:
 A) For Set cloud-init as the customization engine:
 B) For Set perl script as the customization engine:

IMHO all paths should use (A) through cloud-init, but I'm sure you had reasons to also have (B).
Therefore since you likely want/need to keep (B) still I'd think that the right solution for that is (as suggested before) to split the functionality of vmtoolsd/open-vm-tools.service.

Split it into:
- one providing services e.g. rpc used by other components that runs early (essentially as we have it today)
- one executing the (B) customization scripts late as you'd need it to make the guarantees for (B)

As I have mentioned before a lessons learned from cloud-init that you might use is that some changes have to be early (unable to change later) and some late (need some components up).
Therefore an even better split is to break (B) into early and late stages. E.g. considering all your legacy (B) late but one might want to add early script in the future.

Or to re-iterate just use the already existing path through cloud-init if you can.

[Pengpeng Sun (pengpengs)]

Thanks for the suggestion!
Actually we are working on using (A) only and by default. Before that happens, (B) needs be kept.
Instead of splitting vmtoolsd/open-vm-tools services, we might fix this issue in scripts, make sure when 'hostnamectl' cmd executes, the dbus.service is running.

[Christian Ehrhardt  (paelzer)]

Sounds good.
In the meantime have you tried:
  After=dbus.socket
Would that be enough for your needs?
But at the same time not trigger the "Breaking ordering cycle"?

[Pengpeng Sun (pengpengs)]

Yes, I just tried 'After=dbus.socket' on my Ubuntu18.10, it works.
We need do some verifications and then update our KB if it works on the other Ubuntu versions.

[Christian Ehrhardt  (paelzer)]

Ok, if 'After=dbus.socket' works for you that is great.
If this does NOT add the ordering cycle issue we could even maybe make that part of the package.
We'd need to do some testing but first you could check and confirm - with 'After=dbus.socket' do you still not see the "Breaking ordering cycle" warning?

[vmware-gos-Yuhua (yhzou)]

GOS Customazition does't work with cloud-init engine when add "After=dbus.socket" in open-vm-tools.service

[Pengpeng Sun (pengpengs)]

Hi Christian,

According to #15, please do NOT make "After=dbus.socket" as a part of open-vm-tools package.
While we will update VMware KBs to change from "After=dbus.service" to "After=dbus.socket" for (B).
 A) For Set cloud-init as the customization engine:
 B) For Set perl script as the customization engine:

[Christian Ehrhardt  (paelzer)]

Ok,
but mid term I'd love to see you to split open-vm-tools into early and late stages which would allow the VMWare KB to be removed and just work for everyone.

OTOH you might consider deprecating and then removing (B) in favor of only (A) a solution to the same problem and needing less development effort.

[Christian Ehrhardt  (paelzer)]

To summarize the discussion and close this for now:

- as outlined in comment #16 there are two paths in vmtools
  (A) being the new code using cloud-init is NOT affected
  (B) being the old code path being affected with own customization code
  The problem is that for some customizations this has to run early, and for some others things
  it has to run late - that is the Dilemma here.

- Path (A) does not have a problem and is the way to go in the future
  Cloud-init provides all that is needed to configure the system

- (B) is there for compatibility with older setups

- To fix (B) there are two options:
  - modify the config (that is what the KB article is about)
    That is the least nice option, but the benefit is that it costs no effort
  - open-vm-tools could split vmtools customization code into two
    (or more) pieces like cloud-init already does
    Those pieces could then run early/late as needed
    This would be a major dev effort for a code path that is considered
    "the old one" and probably not worth the effort.

So from a Ubuntu perspective all is discussed and done.
I'm marking the bug tasks as Invalid.

From a VMWare perspective the options are:
- keep the KB article up (we can bikeshed about wording)
- spend dev time on (B) to be split
- put time on further deprecating and dropping path (B) (recommended)

[Maher AlAsfar (malasfar)]

what's the update on this as this continue to be an issue for customers using Ubuntu and cloud-init on vmware ? have we like reached a solution in terms of outlining the steps needed to be done for this to work ?

Thank you

[Maher AlAsfar (malasfar)]

I have been spending so many cycle on this, here are the Testing Tasks around Ubuntu 16.04 Customization with Cloud-init
and its expected behaviour for vSphere Provisioning

Using After=dbus.socket doesn't work so stick the KB i mentioned below

if you want to do it yourself its Test 2 with DHCP and Test 4 for Static IP
Please let me know if you were successful or have any comments, questions or suggestions
-----
- Create an Ubuntu 16.04 Virtual Machine on vSphere 6.7 U2.
- Apply the latest updates and upgrades -> sudo apt-get update && sudo apt-get upgrades

Environment Facts
- DHCP is available on the network when preparing the template and later for provisioning.
- Cloud Assembly Static IP range also available for vSphere deployments.

Test 1
=======
behavior for deploying VMs using DHCP from CAS before installing cloud-init

Tasks Invoked by vCenter

- Clone Virtual Machine from Template
- Reconfigure Virtual Machine
- Customize Virtual Machine Guest OS
- Power on Virtual Machine
 During the boot time the network state is Disconnected
- Customization Starts and Succeeds, listed in the VM Events
- VM Reboots
    During the boot time the network state is Connected
- IP provided from DHCP
- Host-name Updated

Note: No Customization Specification used here.

Test 2
======
behavior for deploying VMs using DHCP from CAS After installing cloud-init

Tasks Invoked by vCenter

- Clone Virtual Machine from Template
- Reconfigure Virtual Machine
- Customize Virtual Machine Guest OS
- Power on Virtual Machine
 During the boot time the network state is Connected
- Customization doesn't Start at all, so its not listed in the VM Events
- IP provided from DHCP
- Host-name is not Updated
- Cloud Config Code executes successfully

Note: No Customization Specification used here.

Test 3
======
behavior for deploying VMs using Static IP from CAS before installing cloud-init

Tasks Invoked by vCenter

- Clone Virtual Machine from Template
- Reconfigure Virtual Machine
- Customize Virtual Machine Guest OS
- Power on Virtual Machine
 During the boot time the network state is disconnected
- Customization does start, listed in the VM Events and successful
- VM reboots and network shows as connected
- IP provided from Static IP Range
- Host-name is Updated

Note: No Customization Specification used here.

Test 4
======
behavior for deploying VMs using Static IP from CAS After installing cloud-init
following KB https://kb.vmware.com/s/article/59687

Tasks Invoked by vCenter

- Clone Virtual Machine from Template
- Reconfigure Virtual Machine
- Customize Virtual Machine Guest OS
- Power on Virtual Machine
 During the boot time the network state is disconnected

- Customization does start and successful, listed in the VM Events
- VM reboots and the network shows as connected
- IP provided from Static IP Range
- Host-name is Updated
- Cloud config executes fine

Note: No Customization Specification used here.

[Pengpeng Sun (pengpengs)]

@Maher AlAsfar (malasfar)
What's cloud-init version on the Ubuntu16.04? The version need be greater or equal to 18.2 to make customization work.
What's open-vm-tools version on the Ubuntu16.04?

If you customize Ubuntu with cloud-init, please do NOT add any "After=dbus.xxxxx" to open-vm-tools.service file, since it will create dependency cycle during booting.

And If it's possible, I suggest use Ubuntu18.04, not Ubuntu16.04 to work with cloud-init

[Maher AlAsfar (malasfar)]

Hi @Pengpeng Sun (pengpengs)

This is where i am with Ubuntu 16.04

Behaviour for deploying Ubuntu 16.04 VMs using Static IP from Automation tool After installing cloud-init

Template Tasks Taken
====================
Deploy Ubuntu 16.04 from ISO

sudo apt-get update && sudo apt-get upgrade

sudo apt-get install cloud-init

sudo dpkg-reconfigure cloud-init -> Selecting only OVF and NONE as data sources. since the automation tool maps an iso image to pass the user data to cloud-init

Following KB https://kb.vmware.com/s/article/59687
But using After=dbus.socket instead of After=dbus.service

Using Open-VM-Tools 10304 (10.2.0) Using ESXi 6.7 Update 2 and later (VM version 15) for the VM Hardware

Tasks Invoked by vCenter when provisioning from Automation tool
===============================================================
Clone Virtual Machine from Template

Reconfigure Virutal Machine

Customize Virutal Machine Guest OS

Power on Virtual Machine During the boot time the network state is disconnected

Customization does start and successful, listed in the VM Events

VM reboots and the network shows as connected

IP provied from Static IP Range

Hostname is Updated

Cloud config fail to executes in the right order ( Before Network Settings are Applied ) most of the time but magically sometimes it does execute fine after the network is up

When it doesn't work, looking at the cloud-init logs that Cloud-init start way too early when the network isn't setup / started yet when it executes.

Looking for a way to make cloud-init executes after VMware Customization .. i thought the KB i mentioned above would do that but it only help getting the customization to complete successfully because if you don't add the After=dbus.socket to the open-vm-tools.service , everything else fails which is way worse since the network will be never connected and the customization will error out in the VM Events where the logs shows the same exact error mentioned in the KB above.

Ofcourse this has a completely different dynamics when your testing with Ubuntu 18.04 which i will get into once i figure Ubuntu 16.04 out.

[Maher AlAsfar (malasfar)]

@Pengpeng Sun (pengpengs)

Just tested Ubuntu 18.04 ( same issue ) logs attached

Behaviour for deploying Ubuntu 18 VMs using Static IP from Automation tool
Automation tool maps an image iso to pass cloud-init user data

Template work done
--------------

Deploy Ubuntu 18.04 from iso
sudo apt-get udpate && sudo apt-get upgrade

Cloud-init installed by default

following KB https://kb.vmware.com/s/article/56409
But used After=dbus.socket instead of After=dbus.service

sudo cloud-init clean --log

Open-VM-Tools 10346 (10.3.10)
Compatibility: ESXi 6.7 Update 2 and later (VM version 15) for the VM Hardware

Tasks Invoked by vCenter when profivsioning from CAS

- Clone Virtual Machine from Template
- Reconfigure Virutal Machine
- Customize Virutal Machine Guest OS
- Power on Virtual Machine
 During the boot time the network state is disconnected

- Customization does start and successful, listed in the VM Events
- VM reboots and the network now is connected
- IP provied from Static IP Range
- Hostname is Updated
- Cloud config code fail to executes in the right order

i have attached the logs

[Maher AlAsfar (malasfar)]

anyone can look at the logs i provided.. here we have a successful customization where the VM gets IP-ed , hostname is updated after the customization reboot. the problem now is that Cloud-init execute i think before the reboot when there is no network ..

please if you have the time . take a look at the log provided or recommend a workaround where it forces cloud-init to execute after the customization .. Thanks again.

[Christian Ehrhardt  (paelzer)]

Since this case came up again in another bug, and also after the discussion we here saw Maher chime in still I wanted to ping VMWare again on this.
Therefore I'm assigning John who is responsible for open-vm-tools these days.

@John - is there any upstream development planned for this to further resolve. it - e.g. via the (B) option that I outlined in comment #18? Or will this stay in the "the new way works, if your old customization is affected you'll need to add the workaround" until the old customization path is fully removed?

@John - furthermore we wondered that the KB article (https://kb.vmware.com/s/article/59687) says "resolved in VMware vSphere 6.7 Update 3". Is there anything more than these words that we can give Ubuntu users to know a) what was actually changed there and b) how they can make use of it?

[vmware-gos-Yuhua (yhzou)]

@Christian Ehrhardt

I am working on KB for the following situations:
1) There is conflict between cloud-init and VMware guest OS customization.
   Customization fails if cloud-init present.
2) It will fail for customers to deploy cloud image(.ova) and then customize the VM directly. The password of default user "ubuntu" with cloud image is random. We need KB to tell the users the right steps to customize the VM with cloud image.

Best regards
Yuhua Zou

[Florian Bergmann (bergmann-f-h)]

As I opened the duplicate (and accidentaly commented there), let me just add the comment here as well:

Thanks for the background information - that's really interesting.

Yes I also saw that it is fixed in 6.7U3, however maybe that means they have regression, because we are seeing this issue in vSphere 7.

The version are: VMware ESXi, 7.0.1, 17551050
vSphere Client version 7.0.1.00200

I am not sure - maybe the workflow I am using is also at fault?

Right now I am using the OVA to deploy a initial machine I never power on but immediately turn into a template for further use (cloning a template is just a lot faster for spawning new VMs and we use the template in our ansible playbooks).

What I can see concerning the workflows:

- vSphere is generating the netplan configuration with the configuration set in ansible, so the data propagates correctly and the file is created.
- I tried to strip everything marked as used for guest customization (I only set the network name), but it still fails unfortunately.

If there is any more information I can provide just tell me, I'll happily try to get it.

I can also say it *used* to work with vSphere 7, but I don't know the versions or the image versions we were using at the time (and now we can't find them out anymore).

I attached some logs from a booted VM that show "Executing traditional GOSC workflow." maybe that's the wrong workflow and there is a way to trigger the new one?

[Pengpeng Sun (pengpengs)]

@Florian

The issue is a bit of different with the one reported at the beginning of this bug. Let me explain the details and provide a workaround for you.

The key is when dbus.service is running at booting.

With ova, cloud-init will be enabled and run, when it's running, it blocks dbus, see reason at comment #8. So dbus is running too late to make customization work even with the fix in 6.7U3.

In your case, I suggest you create a new template but turn ova into template directly.
Steps:
1. Power-on VM which deploy from ova.
2. Disable cloud-init by by running the command “sudo touch /etc/cloud/cloud-init.disabled”
3. Power-off VM and convert it to template.

[Florian Bergmann (bergmann-f-h)]

In that case I cannot use cloud-init at all though right?

The problem is I still want to attach an iso file with guest customization (well I should have pointed that out before - sorry), that sets up the user, ssh-key asf for the machine being booted.

That's something I need to configure the ssh-keys that should be allowed to access the machine.

[Pengpeng Sun (pengpengs)]

Yes, let me understand your case more clearly, I thought cloud-init is only needed at step1.

Attach an iso to set up user, ssk-key, is this a one time job or you want it happen for all cloned new VMs?

[Florian Bergmann (bergmann-f-h)]

Exactly so our current workflow looks like this:

- We deploy the ova (not using any cloud-init at all - it never gets powered on)
- We mark this machine directly as a template for reuse

Now in our playbooks we use this template to deploy new machines.

For every machine we generate the cloud-init configurations for guest-config and network-config dynamically and create a cloud-init.iso file.

The network-config file contains:

- static ip address
- netmask
- nameservers

(I think the network config could be done with the guest-customizations now - this apparently was different in the past, when no netplan config was generated by the open-vm-tools. so we might skip this now, if the customizations work)

This guest-config file contains:

- user accounts (that might differ between VMs)
- ssh-keys (that might also differ between VMs)
- Some special runcmds depending on the machine

This setup is VM specific, so we can not do it during the ova deployment but only for every machine itself.

I hope this clears up the situation a bit of how we currently use it.
If there is anything blatantly wrong about this workflow I am happy to hear it, we setup this workflow only about 8 months ago, so we are not sure if this is the way it should be used.

[Pengpeng Sun (pengpengs)]

Understood. I think you are using vSphere7, guest-customization does support netplan from vSphere 6.7U3g, so network-config can be set.
Guest-customization has 2 engines: one is perl scripts, the other is cloud-init. Unfortunately, neither of them can work with cloud-init to set userdata guest-config at the same boot.
For perl script based guest-customization, it has dependency on dbus, due to the reason in comment #28, it fails with cloud-init running.
For cloud-init based guest-customization, it uses the same cloud-init datasource OVF with ISO file, but different seed. Cloud-init can only apply either guest-customization or ISO userdata guest-config.

Good news is if you are using vSphere7, you can use customization post-customization script to finish your workflow, need below steps:

1. Power-on VM which deploy from ova.
2. Disable cloud-init by running the command “sudo touch /etc/cloud/cloud-init.disabled”
3. Enable customization script by running the command "vmware-toolbox-cmd config set deployPkg enable-custom-scripts true"
3. Power-off VM and convert it to template.
4. Create a new guest-customization specification which includes network-configs and customization script, the script looks like:
   #!/bin/sh
   if [ x$1 == x"postcustomization" ]; then
   rm /etc/cloud/cloud-init.disabled
   systemctl reboot
   fi
5. Generate the cloud-init configurations for guest-config dynamically and create a cloud-init.iso file
6. The rest steps you did

Related document and KB:
a. Create a Customization Specification for Linux https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-9A5093A5-C54F-4502-941B-3F9C0F573A39.html
b. Setting the customization script for virtual machines in vSphere 7.0
   https://kb.vmware.com/s/article/74880

[Florian Bergmann (bergmann-f-h)]

This seems to work to run the user-data part of cloud-init, so that's great.

However the problem seems to be that the Customization script must now include the network configuration as well - so either I have to find a way to generate the script dynamically or maybe find a way that *all* customization is done through cloud-init.

Is there a way to just completely turn off any customization?

I tried to use:

vmware-toolbox-cmd config set deployPkg enable-custom-scripts false
vmware-toolbox-cmd config set deployPkg enable-customization false

But it seems that even with those 2 commands it is still trying to run the customizations.

[Pengpeng Sun (pengpengs)]

Yes, exactly. I wonder how the network-config file is applied in your current workflow.

Yes, guest customization can be turned off if VM's open-vm-tools version is 11.1.0 or above, see KB https://kb.vmware.com/s/article/78903.
I just checked open-vm-tools version in https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.ova, it's 11.0.5. So you have to upgrade to 11.1.0+ to disable guest-customization.

[Florian Bergmann (bergmann-f-h)]

Just as an update - I can confirm that disabling works with 20.10 (which uses an up-to-date enough version of open-vm-tools).

I still didn't attach the network automatically, but I have the suspicion that this is because of the ansible collection, because after the machine is booted I can attach the network using govc using:

govc device.connect -vm=myvm ethernet-0

[Pengpeng Sun (pengpengs)]

I think the network disconnection is a result of guest-customization task, although the vm side customization has been disabled according to the tools configuration.
You can try connect the vm network right before power-on VM by gosv, then the vm network can be available automatically.

[Lukáš Vasek (bilak)]

Hello,
Sorry for hijacking this thread but I've recently also faced same issue. My intention was to use focal-server-cloudimg-amd64.ova (20.04LTS) on esxi 7.0 (no vSphere, just free version) and initiate it with cloud-init (userdata + metadata). However this seems not to be working as it's described in previous comments even when guestinfo.userdata and guestinfo.metadata are correctly set.

My question is how can I disable vmware tools, to use just cloud-init? I see those commands (vmware-toolbox-cmd ...), but I'm not quite sure from where can I execute them, because afaik it's not possible to get into cloudimg while it's not somehow initialized. I believe there's no user is it?

Thanks

[Pengpeng Sun (pengpengs)]

Hello Lukáš,

You mentioned "initiate it with cloud-init (userdata + metadata)", may I know how did you initiate userdata and metadata? also you mentioned guestinfo.userdata and guestinfo.metadata, did you use the way described here: https://github.com/vmware/cloud-init-vmware-guestinfo ?

Another question: did you trigger guest customization on the VM? if not, you needn't disable guest customization from the vmware-toolbox-cmd.

There should be a default user 'ubuntu' which is configured in /etc/cloud/cloud.cfg, I think you will be asked to set password for this user when you deploy focal-server-cloudimg-amd64.ova on ESXi(I'm not sure what free version means, were you using vCenter server?)

At last, we need cloud-init logs to see what really happened in VM, command 'sudo cloud-init collect-logs' should help to collect them.

[Lukáš Vasek (bilak)]

Hello Pengpeng
I'm using terraform + this provider https://github.com/josenk/terraform-provider-esxi and afaik that provider is using also https://github.com/vmware/cloud-init-vmware-guestinfo to create guestinfo.

I can't find the way how to trigger the disable action with vmware-toolbox-cmd. If I nuderstand that correclty I need to start the VM, but isn't there some tool to do that automatically? when I was trying to start that OVA in vmware esxi it didn't started at all.

Free version of esxi is just some basic functionalit where event vSphere api is not provided. So only some UI console and ssh to host is provided. You even can't create a template from some VM, it's only possible to copy it. So there fore I just want to have some empty OVA file which I will customizes with cloud init.

The problem which I see here is that I'm not able to get into the VM that is created from OVA cloud image so I'm not able to execute any action inside it.

Yesterday I was also trying to modify the OVA file with virt-customize but unsuccessfully...more on that here https://askubuntu.com/questions/1339169/cloudimage-20-04-disable-vmware-custom-scripts

I hope there's something that can be done even with free esxi and other tools.

[Pengpeng Sun (pengpengs)]

terraform is new to me, but if it's using https://github.com/vmware/cloud-init-vmware-guestinfo, it won't work unless the installation has been done in VM. See https://github.com/vmware/cloud-init-vmware-guestinfo/blob/master/README.md

So you have to power-on the VM firstly, have you tried to modify OVF file extracted from OVA file, it contains a password property which is "Default User's password", the default user is ubuntu.

Once you get into the VM, you can install cloud-init-vmware-guestinfo and then save the VM as source VM.

[Lukáš Vasek (bilak)]

Ok, so I'm trying with manual steps, no automation involved. I've converted ova to ovf, then in ovf modified password to ubuntu

<Property ovf:key="password" ovf:type="string" ovf:userConfigurable="true" ovf:value="ubuntu">
</Property>

And the uploaded vdmk + ovf through UI.

However when I start the VM I'm not able to login with ubuntu/ubuntu. Is there something what am I missing?

I've attached vmware.log, if you can spot something wrong in that please let me know.