open-vm-tools "hwclock" needed for VM guest customization not available

[John Wolfe]

The VMware guest customization uses "hwclock" to configure the hardware clock for "utc" or "localtime". Apparently starting with Ubuntu 23.10 (mantic), the "hwclock" command is no longer part of the base release.

See:
    https://bugs.launchpad.net/ubuntu/+bug/2037412
    https://answers.launchpad.net/ubuntu/+question/708039

The "hwclock" command is available in the "util-linux-extra" package.

Please add the "hwclock" or the "util-linux-extra" package as a dependency of open-vm-tools for Ubuntu 23.10 and later releases.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Pengpeng Sun (pengpengs)]

Hi Team,

Please take a look at this bug, thanks.

[Pengpeng Sun (pengpengs)]

Tracking by VMware bug 3293381

[Sergio Durigan Junior (sergiodj)]

Thanks for taking the time to file a bug.

It is not possible to make open-vm-tools depend on fake-hwclock nor util-linux-extra because both packages are in universe, and open-vm-tools is in main. If we were to do it, we would have to promote one of the packages to main first. In theory this could be done on Noble, but Mantic has been released already so it's harder to make such change there.

[Pengpeng Sun (pengpengs)]

Many thanks for the update.

So it's harder on Mantic, any luck for the future Mantic point releases, ex: 23.10.2 or 23.10.3?
Besides this adding util-linux-extra as dependency of open-vm-tools, I was also proposing another adding hwclock or util-linux-extra back to default installation packages, see https://bugs.launchpad.net/ubuntu-seeds/+bug/2037412. Wondering if setting hardware clock is not needed for most of Ubuntu users?

Please do bring back hwclock on Noble, in the meanwhile, we will skip hardware clock customization if hwclock command is not available in VM from vSphere side, but Noble will be widely supported from released vSphere versions, so please kindly fix this.

[Mitchell Dzurick (mitchdz)]

Apologies in advance as I'm not very familiar with these packages.

A change happened starting in Lunar, where util-linux-extra was introduced and hwclock got moved to that package. hwclock is still in the main archive, it's just not seeded anymore as it's bundled in util-linux-extra now.

Am I mistaken in that the utility hwclock from util-linux-extra is required here, and NOT fake-hwclock? I see these two programs referred to almost interchangeably between both bugs and the goal of these packages seem different.

[Pengpeng Sun (pengpengs)]

You are right, Mitchell. The goal has been changed.

The previous goal in https://bugs.launchpad.net/ubuntu-seeds/+bug/2037412 was to add hwclock back to Mantic default installation and cloud image, while Bernard told me that "The install program list is full, this must be maintained to reduce bloat of the operating system.". Then we turned to ask for this approach in this bug, which is adding util-linux-extra as dependency of open-vm-tools, so that hwclock command will be installed when installing open-vm-tools.

To be clear, the hwclock command required by vSphere guest customization is:

# which hwclock
/sbin/hwclock
# /sbin/hwclock --help

Usage:
 hwclock [function] [option...]

Time clocks utility.

Functions:
 -r, --show display the RTC time
     --get display drift corrected RTC time
     --set set the RTC according to --date
 -s, --hctosys set the system time from the RTC
 -w, --systohc set the RTC from the system time
     --systz send timescale configurations to the kernel
 -a, --adjust adjust the RTC to account for systematic drift
     --param-get <param> display the RTC parameter
     --param-set <param>=<value> set the RTC parameter
     --predict predict the drifted RTC time according to --date

Options:
 -u, --utc the RTC timescale is UTC
 -l, --localtime the RTC timescale is Local
 -f, --rtc <file> use an alternate file to /dev/rtc0
     --directisa use the ISA bus instead of /dev/rtc0 access
     --date <time> date/time input for --set and --predict
     --delay <sec> delay used when set new RTC time
     --update-drift update the RTC drift factor
     --noadjfile do not use /etc/adjtime
     --adjfile <file> use an alternate file to /etc/adjtime
     --test dry run; implies --verbose
 -v, --verbose display more details

 -h, --help display this help
 -V, --version display version

Arguments:
 <param> is either a numeric RTC parameter value or one of these aliases:
   - features: supported features (0x0)
   - correction: time correction (0x1)
   - bsm: backup switch mode (0x2)
   See Kernel's include/uapi/linux/rtc.h for parameters and values.

 <param> and <value> accept hexadecimal values if prefixed with 0x, otherwise decimal.

For more details see hwclock(8).

[Lucas Kanashiro (lucaskanashiro)]

I see some confusion here. Let me try to be clear:

To add util-linux-extra as a runtime dependency of open-vm-tools we need to promote the util-linux-extra binary to main (it is in universe right now):

$ rmadison util-linux-extra
 util-linux-extra | 2.38.1-4ubuntu1 | lunar | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
 util-linux-extra | 2.38.1-4ubuntu1.1 | lunar-proposed | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
 util-linux-extra | 2.39.1-4ubuntu2 | mantic/universe | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
 util-linux-extra | 2.39.1-4ubuntu2 | noble/universe | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x

This is a requirement because open-vm-tools is in main, and all its dependencies also need to be in main. In order to promote this package we need to go through the MIR process:

https://github.com/canonical/ubuntu-mir

But taking a step back, I do not think that adding util-linux-extra as a dependency of open-vm-tools is the right way of tackling this issue. open-vm-tools itself makes not direct call to hwclock. open-vm-tools relies on the ability of the system to manage the clock, and that is something that should be available in any Ubuntu installation. hwclock was demoted to main for a reason that I do not know (sorry, did not try to figure this out yet), if we did this right, there should be an alternative option in main for that. If there is no alternative, we should consider promoting it and adding it to the seeds.

[Pengpeng Sun (pengpengs)]

Thanks Lucas, I see your point, open-vm-tools does NOT depend on util-linux-extra directly. By adding this new dependency, it's kind of workaround which hwclock will be available on Ubuntu Mantic and later versions on VMware platform. For ex: Ubuntu cloud image for vSphere platform have open-vm-tools installed by default.

vSphere guest customization has been using hwclock command to set hardware clock since Ubuntu 13.04, if there is an alternative way to do this since Ubuntu Mantic, we can change our code to use it.

[Paride Legovini (paride)]

Hello and thanks for the additional information.

As I understand it (mostly from d/changelog entries), the hwclock tool may have been moved to -extra because modern kernels sync the RTC automatically. Did you check if manual hwclock invocation is actually needed in VMware VMs?

Another question: you mentioned "VMware guest customization". If that's done using some cloud-init or an equivalent tool, can you possibly add the -extra package as a package to install as part of the customization?

[Pengpeng Sun (pengpengs)]

Thanks for the checking.

We have an optional property in Linux customization specification, when doing Linux VM customization, customer can choose True or False for the "hwClockUTC", see it in the API document: https://vdc-download.vmware.com/vmwb-repository/dcr-public/184bb3ba-6fa8-4574-a767-d0c96e2a38f4/ba9422ef-405c-47dd-8553-e11b619185b2/SDK/vsphere-ws/docs/ReferenceGuide/vim.vm.customization.LinuxPrep.html
It's to
"
Specifies whether the hardware clock is in UTC or local time.
True when the hardware clock is in UTC.
False when the hardware clock is in local time.
"

For "VMware guest customization", we have own customization engine or cloud-init to do the real setting in a Linux VM. In this case, only our own customization engine supports customizing hardware clock, cloud-init does NOT.

[Lucas Kanashiro (lucaskanashiro)]

Pengpeng, couldn't you make your "guest customization" depend on the -extra package as Paride suggested? That would be the most straightforward solution I believe.

[Pengpeng Sun (pengpengs)]

Sorry for not stating this clear early.
The "guest customization" does NOT require any other packages installed in VM except open-vm-tools, and "guest customization" isn't a systemd service triggered, this is unlike cloud-init. So we suggested adding the -extra package as dependency of open-vm-tools if possible.

I'm updating "guest customization" to skip hardware clock change when hwclock command is not available, but customer on the released vSphere versions, they will face customization failure on Ubuntu Mantic if they customize hardware clock.

[David Hekimian (dhekimian)]

Can confirm this issue still exists with Noble Numbat 24.04 with open-vm-tools 12.3.5-5build1.

Is there a plan to resolve this before 24.04 is released?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robie Basak (racb)]

The reason hwclock isn't shipped any more is that Ubuntu now uses systemd's functionality to perform the same task instead. To configure RTC translation between UTC and local time, the primary supported method is now "sudo timedatectl set-local-rtc 0" or "sudo timedatectl set-local-rtc 1". See timedatectl(1) for details (including why '1' is unreliable). Use of hwclock should be considered a deprecated method on newer releases of Ubuntu, although I'm not aware of any plans to remove it completely and it looks like the new method is currently entirely compatible with the old one.

This isn't Ubuntu-specific - I imagine that this is the case for all platforms that are exclusively based on systemd.

Given that open-vm-tools itself does not contain the call to hwclock either, I'm not sure that we can consider this to be an actionable bug against Ubuntu itself.

Is it possible for VMware guest customization to use the new recommended method on newer Ubuntu releases instead? Alternatively, I suggest that if you're sending commands to run via open-vm-tools, that you precede that with commands to ensure that the packages those commands depend on are installed first.

[Robie Basak (racb)]

Here's some more background: https://lists.freedesktop.org/archives/systemd-devel/2011-May/002526.html

[John Wolfe (johnwvmw)]

Thanks for the suggestion.

I have passed your suggestions along to the Guest Customization team along with links to this bug report in an internal bug.

[Pengpeng Sun (pengpengs)]

Thanks Robie for the information, this is exactly the VMware guest customization updating in the next vSphere 8.0 update release which is "Using timedatectl command to set hardware clock since Ubuntu 23.10".

However as Ubuntu 24.04 is LTS, it's widely supported on vSphere 8.x line, without hwclock command, customers who are using the current vSphere 8.0 - 8.0U2 releases will encounter this issue.

Understood that timedatctl command shall be used with systemd, still if adding hwclock command back has no conflict, it will be great helpful for customers.

[Robie Basak (racb)]

> still if adding hwclock command back has no conflict, it will be great helpful for customers.

Unfortunately hwclock is in universe so we cannot easily ship it automatically any more. We'd have to promote it back to main first. So it isn't so easy.

If you wish to undertake "Guest Customization" reliably then you must interact correctly with the interfaces provided by the guest OS. These will of course change from release to release of the guest OS, so in general it is always necessary to adapt interactions as required by new guest OS releases. Therefore your customers should not expect any new guest OS release to function correctly with your Guest Customization feature prior to receiving such an update from you.

Unfortunately it is not practical for us to provide backwards compatibility such as this in the general case. We might be able to make an exception but in this particular case this also seems impractical.

It sounds like this problem will go away as soon as you release your fixed vSphere update?

[Pengpeng Sun (pengpengs)]

> If you wish to undertake "Guest Customization" reliably then you must interact correctly with the interfaces provided by the guest OS. These will of course change from release to release of the guest OS, so in general it is always necessary to adapt interactions as required by new guest OS releases. Therefore your customers should not expect any new guest OS release to function correctly with your Guest Customization feature prior to receiving such an update from you.
> It sounds like this problem will go away as soon as you release your fixed vSphere update?

Yes，this is reason we always test "Guest Customization" on new guest os daily/beta build before it's released to catch such failure due to guest os change. If the fix is on vSphere side, customer will have to wait for the next vSphere release which contains the fix and must upgrade to it to get the fix. While if guest side(at least in guest os GA build) can fix it, customer will not encounter the failure at all. Certainly, where the fix goes to is case by case.

[lethargos (lethargos)]

Any news related to this issue? If you offer a temporary solution that skips the use of hwclock without having to change the image beforehand it would be great.

As you probably know, when using vmwaretools you cannot install it beforehand with the cloudimage using cloud-init, because vmware tools needs hwclock immediately and it will error out.

In my case, I think this leads to the network interface not being attached anymore (although the original ova template is set to "Connect at Power On".

[Pengpeng Sun (pengpengs)]

We've updated to use "timedatectl" but "hwclock" to set hardware clock for Ubuntu 24.04 and later releases. The fix is included in the recent releases of vCenter:

    VMware vCenter Server 8.0U3 & 8.0U3a
    VMware vCenter Server 7.0 Update 3q

No image change needed after upgrading vCenter server to the above releases.