change the future OVA(s) to have default VM hardware version 13(ESXi 6.5)

[vmware-gos-Yuhua]

Hi Admin,
Right now the default hardware version for ubuntu 20.10 OVA is VM hardware version 10(ESXi 5.5) and It's too old.

Would you please change the Ubuntu 20.10 OVA to have default VM hardware version 13. If you want to be more conservative, you could pick VM hardware version 11(ESXi 6.0).

when you change to hardware version 13, please add serial port by default due to LP 1895104.

Thanks very much.

Best regards
Yuhua Zou

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1898871/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[John Chittum (jchittum)]

Thank you for opening this ticket. I've added it to the cloud-images project for prioritization.

[John Chittum (jchittum)]

We've confirmed this request for upgrading hardware version for our ova's. We have prioritized this on our backlog, and have it linked to 1895104

[vmware-gos-Yuhua (yhzou)]

Hi jchittum,
"Serial Port 1" is added with ubuntu-21.10-server-cloudimg-amd64.ova
The default configuration about "Serial Port 1" is
   Serial Port 1: Use physical serial port
   Status: Connected at PowerOn

But with the default configuration I get warning message as follows when power on the VM:
     Cannot connect the virtual device serial0 because no corresponding device is available on the host. Do you want to try to connect this virtual device every time you power on the virtual machine?

Workaround:
It can be fixed by Setting "Connected at PowerOn" to false.

Thanks
Yuhua Zou

[John Chittum (jchittum)]

vmware-gos-Yuhua : what is the corresponding ovf template entry for Connected At Power On? We generate all builds in a chroot, with a templated OVF, so we need to identify the exact stanza. What we have now is:

https://git.launchpad.net/livecd-rootfs/tree/live-build/ubuntu-cpc/hooks.d/base/ovf/ubuntu-ova-v1-vmdk.tmpl#n87

      <Item ovf:required="false">
        <rasd:AutomaticAllocation>true</rasd:AutomaticAllocation>
        <rasd:ElementName>serial0</rasd:ElementName>
        <rasd:InstanceID>4</rasd:InstanceID>
        <rasd:ResourceSubType>vmware.serialport.device</rasd:ResourceSubType>
        <rasd:ResourceType>21</rasd:ResourceType>
        <vmw:Config ovf:required="false" vmw:key="yieldOnPoll" vmw:value="false" />
      </Item>

Generally speaking, is there a place to look up all the possible values? I keep going to https://www.dmtf.org/sites/default/files/standards/documents/DSP0243_2.1.1.pdf but that doesn't include a full list of the things that'd full under <vmw: Config> type options, nor what values may be possible (such as vmware.serialport.device). Could you help by pointing to a resource?

[vmware-gos-Yuhua (yhzou)]

Hi jchittum,
   > John Chittum (jchittum) wrote on 2021-11-04: #5

   It can fix the issue by changing "<rasd:AutomaticAllocation>true</rasd:AutomaticAllocation>" to "<rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>"

   Steps:
   1) deploy ubuntu-21.10-server-cloudimg-amd64.ova in ESXi
   2) edit VM setting to uncheck the checkbox "Connected at PowerOn".
   3) export the VM as OVF template
   4) find <rasd:AutomaticAllocation>false</rasd:AutomaticAllocation> when check .ovf file.

   (Note: with the upper same steps, I also try to check/uncheck the checkbox "CD/DVD Drive 1"/"Network adapter 1" and then export as OVF template. I find the only difference of file .ovf is that item "rasd:AutomaticAllocation" is changed to true/false.)

Best regards
Yuhua Zou

[John Chittum (jchittum)]

Thank you for providing more information. Is it possible to find out what sections of the OVF to change without having to boot an image in vsphere? It'd be wonderful to have documentation of generally available configurations and descriptions. I check the primary source document, but it's difficult to use to answer questions such as "how do I make a serial console not available automatically"

https://www.dmtf.org/sites/default/files/standards/documents/DSP0243_2.1.1.pdf

Does VMWare provide guidance on editing OVF files outside of booting an image, making a change, and exporting?

[vmware-gos-Yuhua (yhzou)]

Hi John Chittum,
 > Does VMWare provide guidance on editing OVF files outside of booting an image, making a change, and exporting?

 I try the following steps and it works.

 1) extract ubuntu-impish-21.10-cloudimg.ova, we will see three files ubuntu-impish-21.10-cloudimg.ovf, ubuntu-impish-21.10-cloudimg.mf and ubuntu-impish-21.10-cloudimg.vmdk

 2) edit file ubuntu-impish-21.10-cloudimg.ovf and change "<rasd:AutomaticAllocation>true</rasd:AutomaticAllocation>" to "<rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>" for the element serial0 as described in #5

 3) produce the SHA256 for modified file ubuntu-impish-21.10-cloudimg.ovf with the command
    shasum -a 256 ubuntu-impish-21.10-cloudimg.ovf

 4) edit file ubuntu-impish-21.10-cloudimg.mf to update the SHA256 value of file ubuntu-impish-21.10-cloudimg.ovf

 5) repack ubuntu-impish-21.10-cloudimg.ova with the command:
    tar cf ubuntu-impish-21.10-cloudimg.ova ubuntu-impish-21.10-cloudimg.ovf ubuntu-impish-21.10-cloudimg.mf ubuntu-impish-21.10-cloudimg.vmdk

 6) (optional) deploy the repacked ubuntu-impish-21.10-cloudimg.ova in ESXi and I don't hit previous warning message when VM is powered on.

Thanks
Yuhua Zou

[vmware-gos-Yuhua (yhzou)]

Hi John Chittum (jchittum)
Do you have any plan to upgrade the default VM hardware ?

Best regards
Yuhua Zou

[John Chittum (jchittum)]

Sorry, I didn't receive a reply to the original question:

Does VMWare provide guidance on editing OVF files outside of booting an image, making a change, and exporting?

What was provided was specific instructions for a single XML stanza. What we are worried about is making a change that isn't supported, while only having limited internal testing available. Having documentation available letting us know best practices and expected values for editing an OVMF _without first booting into VSphere_ is important.

our builds happen in a chroot environment, and are never booted. That's why understanding best practice and having documentation about what's supported and required is very helpful.

for VMHardware, for instance, is there a way to easily look up what is supported for that stanza? then we can make sure that when a version of ESXI is retired, we can properly up-rev the stanza.

As for prioritization, we will try and add this to the queue before EOY for the next release. We'll need community testing of those daily images before being able to backport a change through livecd-rootfs. The testing is important to ensure a wide variety of environments operate properly.

[John Savanyo (jsavanyo)]

Hi John,

Apologies for the delay in replying to your earlier questions.

VMware's current documentation [1] on OVF files are limited to providing guidance to end-users that describes workflows like exporting VMs to OVF format and importing back to VMs. We currently do not have documentation focused on ISVs who are programmatically generating OVF files. In a conversation with internal technical leaders yesterday, we agreed that there is a need to provide better documentation for ISVs. It would be a significant amount of effort to produce a human authored document that describes all of the many virtual devices and settings supported by VMware VMs and it would difficult to maintain the document in sync with the software. As a result, we plan to investigate adapting some of our internal software that can read and write OVF files to also automatically produce more detailed documentation on the supported virtual devices. Using this approach, the documentation and software will be in sync. In the mean time, the best way to figure out how to represent something in OVF format is to create a VM on ESXi using that configuration, export the VM to OVF/OVA format and look at the contents of the OVF file...

We agree that using chroot environment to build OVA is the best approach and the most secure. To help ISVs with this process we previously published the open-vmdk project [2] that can help automate this process using open-source code that can be included in third-party build systems. This code can convert flat file system to vmdk format. @jchittum, Is Canonical using open-vmdk code to create ovf and vmdk files or some other approach like qemu?

Regarding virtual devices supported by different generations for VMware VMs, the best source of information is vSphere's documentation on virtual hardware versions [3]. A project is underway to improve these docs and you should expect to see enhancements by the end of the year published.

Regarding testing of OVA images, VMware can help. Please reach out to Yuhua Zou who created this bug and Haiying Ding (<email address hidden>). For automation of testing, please check out the open-source project [4] that provides an ansible environment to test operating systems on vSphere/ESXi. Canonical should have access to not-for-resale (NFR) licenses for VMware products that can be used for internal testing through VMware TAP program [5]. To access these licenses contact Dorian Naveh inside Canonical.

Please let us know if you have any other questions.

Thanks,
John

[1]: https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vm-administration/GUID-AFEDC48B-C96F-4088-9C1F-4F0A30E965DE.html?hWord=N4IghgNiBcIPIDUBiIC+Q
[2]: https://github.com/vmware/open-vmdk
[3]: https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vm-administration/GUID-789C3913-1053-4850-A0F0-E29C3D32B6DA.html
[4]: https://github.com/vmware/ansible-vsphere-gos-validation
[5]: https://www.vmware.com/partners/work-with-vmware/tap.html

[vmware-gos-Yuhua (yhzou)]

> Regarding testing of OVA images, VMware can help. Please reach out to Yuhua Zou who created this bug

Please let me know when the new OVA image is ready. Thanks.

Best regards
Yuhua Zou

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Arunesh Pandey (aruneshpa)]

Hi John,

Just wanted to check if there has been any update on this? I verified that the latest daily Mantic Minotaur build is still using VM hardware version 10(ESXi 5.5) which is out of support. This will impact a significant number of customers who want to use the stock OVA with features offered by newer hardware.

Regards,
Arunesh

[John Chittum (jchittum)]

At this point, we need to have a meeting. We have a number of hurdles that commenting on this ticket will not resolve. My email is in my profile, please have your management reach out, and include Dorian.

For Mantic, it's already post-Freeze. Due to our lack of testing environment, making the change before 23.10 goes out would be problematic.

###
please see the previous comments -- I still do not have any real guidance on

I'm still awaiting the ISV Documentation. Without it, we're shooting in the dark, and tossing images over the wall for testing.

1. VSphere support and OVF information: [0]
   b. backwards compatibility -- if i use HW version 15, is it happily supported in ESXi 8? What are the tradoffs? will older HW Versions not boot on newer VSphere at all? What is the minimum to make it work "everywhere?"
   c. Features -- what do we need to do here? the docs list tons of features, when they're added, levels of support, etc. Is there something we need to be doing?
2. what are "defaults" vs. "environment" setups. How do we ensure our OVF will boot in many environments, but is fully configurable as needed?
3. best practices for producing a "base" OVA, knowing that every environment is different (so what values need set in which stanzas to be able to ensure users can override as needed).

> Is Canonical using open-vmdk code to create ovf and vmdk files or some other approach like qemu?

first, all our code here is open source, so please feel free to read everything. I've linked this all previous:

1. https://git.launchpad.net/livecd-rootfs/tree/live-build/functions#n319
   a. function that is always used.
   b. utilizes https://packages.ubuntu.com/kinetic/python3-vmdkstream
2. vmdk base generation: https://git.launchpad.net/livecd-rootfs/tree/live-build/ubuntu-cpc/hooks.d/base/vmdk-image.binary
   a. gist: takes our rootfs ext4 and creates the vmdk
3. OVA: https://git.launchpad.net/livecd-rootfs/tree/live-build/ubuntu-cpc/hooks.d/base/vmdk-ova-image.binary
   a. gist: utilizes an OVF template, and shoves in pieces, and packages with the vmdk
4. the template: https://git.launchpad.net/livecd-rootfs/tree/live-build/ubuntu-cpc/hooks.d/base/ovf/ubuntu-ova-v1-vmdk.tmpl

As this is all open source, you can also issue MPs directly! Launchpad follows a forking workflow. You can also attach diffs directly to tickets

as for `open-vmdk` specifically, I don't see it packaged for Ubuntu. I do not see it in Debian either. Would you be open to having this packaged for distros to use? Otherwise, we cannot use it for building images. All tools must either be

a. a part of the distro or;
b. developed internally by Canonical

> For automation of testing, please check out the open-source project

The issue here is licenses and testing. I'll reach out to Dorian again, but this trail went cold a few years ago. We're still going to have issues keeping up with changes in VSphere.

[0] https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vm-administration/GUID-789C3913-1053-4850-A0F0-E29C3D32B6DA.html