network customization with cloud-init does not work on Ubuntu18.04 Beta2 Server

[Pengpeng Sun]

This issue does NOT exist on Ubuntu18.04 Beta2 Desktop, it only reproduces on Ubuntu18.04 Beta2 Server

Ubuntu18.04 Beta2 Server is installed on a VMware VM.

VMware customization Config File was copied to /var/run/vmware-imc/cust.cfg correctly.
Check cloud-init.log, it parsed the cust.cfg, but VM network is not customized correctly according to cust.cfg.

[Pengpeng Sun (pengpengs)]

attached cloud-init.log

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Pengpeng Sun (pengpengs)]

I compared Desktop with Server after customization.

Ubuntu 18.04 Desktop:
>ls /etc/network
if-post-down.d if-pre-up.d if-up.d interfaces interfaces.before_vmware_customization interfaces.d
>cat /etc/network/interfaces
# DO NOT EDIT THIS FILE BY HAND -- AUTOMATICALLY GENERATED BY cloud-init
source /etc/network/interfaces.d/*.cfg
>ls /etc/network/interfaces.d
50-cloud-init.yaml

Ubuntu 18.04 Server:
>ls /etc/network
if-post-down.d if-pre-up.d if-up.d interfaces interfaces.before_vmware_customization
>cat /etc/network/interfaces
# DO NOT EDIT THIS FILE BY HAND -- AUTOMATICALLY GENERATED BY cloud-init
source /etc/network/interfaces.d/*.cfg
>ls /etc/netplan
01-netcfg.yaml 50-cloud-init.yaml

·On Desktop, it has ’50-cloud-init.cfg’ under /etc/network/interfaces.d which is correct network configuration according to customization spec
·On Server, there is no /etc/network/interfaces.d directory, the ’50-cloud-init.cfg’ is under /etc/netplan and it does not change according to customization spec.

[Chad Smith (chad.smith)]

We don't quite have enough information here. It looks like there are multiple boots listed in this log (and ifconfig shouldn't be in stock bionic images too). Our expectation is that netplan renderer should always be used on Bionic with stock ubuntu images, not eni (etc/network/interfaces), so I'm not certain why the 'desktop' images that are being used are actually going down the route of writing ENI files.

You could try a "cloud-init collect-logs" on a fresh install and attach the cloud-init.tar.gz to this bug, but I fear this particular bug may not have enough information to be able to reproduce/understand what's going on.

[Chad Smith (chad.smith)]

Marked incomplete until we get more info here.

[Chad Smith (chad.smith)]

Pengpeng Sun,

Thank you for filing this bug against cloud-init, if you are able to provide the additional information per cloud-init collect-logs (or steps to reproduce) then please do set the bug status back to 'New' and we'll relook at this issue.

[Pengpeng Sun (pengpengs)]

Hi Chad,

Thanks for your investigation. You are right, I tried many times on that Beta2 build(also did apt install nettools to use ifconfig), for Desktop, I observes networking configuration is not handled by netplan, while eni(/etc/network/interfaces) takes effect.

Today I did the following steps and collected cloud-init logs on the GA version of Ubuntu18.04 Server.
1. Installed Ubuntu18.04 Server GA build on a VMware VM
2. Ran 'apt install cloud-init' to install cloud-init
3. Did VMware guest customization by cloud-init on this VM
4. Ran 'cloud-init collect-logs', please find the generated .gz file in attachment

I set the bug status back to "New", please relook at this issue. Let me know if any other info needed here.

Thanks,
Pengpeng

[Pengpeng Sun (pengpengs)]

the cloud-init log

[Pengpeng Sun (pengpengs)]

ping: please help to relook this bug.

[Chad Smith (chad.smith)]

Looks like network config parsing on this system is falling over and OVF datasource can't get detected as a result. There error is coming out of cloudinit/sources/helpers/vmware/imc/config_nic.py", line 185, in gen_ipv4_route as no gateway is present?

  https://pastebin.ubuntu.com/p/qVJxDJWZRV/

[Pengpeng Sun (pengpengs)]

Thanks a lot Chad, we will look at this and back to you later

[Pengpeng Sun (pengpengs)]

We have found the root cause, need update code at https://git.launchpad.net/cloud-init/tree/cloudinit/sources/helpers/vmware/imc/config_nic.py, I will submit a fix on this.

[John Savanyo (jsavanyo)]

This is related to VMware Bug #2125119

[Server Team CI bot (server-team-bot)]

This bug is fixed with commit 9c35f976 to cloud-init on branch master.
To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=9c35f976

[Scott Moser (smoser)]

This bug is believed to be fixed in cloud-init in version 18.4. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[Scott Moser (smoser)]

This bug is fixed in cosmic.

This bug was fixed in the package cloud-init - 18.4-0ubuntu1
---------------

cloud-init (18.4-0ubuntu1) cosmic; urgency=medium

  * New upstream release.
    - release 18.4 (LP: #1795741)
    - tests: allow skipping an entire cloud_test without running.
    - tests: disable lxd tests on cosmic
    - cii-tests: use unittest2.SkipTest in ntp_chrony due to new deps
    - lxd: adjust to snap installed lxd.
    - docs: surface experimental doc in instance-data.json
    - tests: fix ec2 integration tests. process meta_data instead of meta-data
    - Add support for Infiniband network interfaces (IPoIB). [Mark Goddard]
    - cli: add cloud-init query subcommand to query instance metadata
    - tools/tox-venv: update for new features.
    - pylint: ignore warning assignment-from-no-return for _write_network
    - stages: Fix bug causing datasource to have incorrect sys_cfg.
      (LP: #1787459)
    - Remove dead-code _write_network distro implementations.
    - net_util: ensure static configs have netmask in translate_network result
      [Thomas Berger] (LP: #1792454)
    - Fall back to root:root on syslog permissions if other options fail.
      [Robert Schweikert]
    - tests: Add mock for util.get_hostname. [Robert Schweikert] (LP: #1792799)

 -- Chad Smith <email address hidden> Tue, 02 Oct 2018 19:55:23 -0600

[vmware-gos-Yuhua (yhzou)]

This issue is fixed when check ubuntu-18.10-live-server-amd64.iso with cloud-init 18.4.7

[Scott Moser (smoser)]

I'm marking this fix-released for bionic and xenial.
The fix is available in bionic-updates and xenial-updates
and was SRU'd under bug 1795953.

If you are experiencing this problem still, re-open the bug.

Thank you.

[Matti Haack (matttih)]

We still encounter problems with cloud init network configuration in bionic

https://kb.vmware.com/s/article/59557 gives a workaround.

and open /etc/cloud/cloud.cfg file with root privileges and add the entry:
disable_vmware_customization: false
fixes the issue.

It is possible to add this config line as a default for bionic?

[Dan Watkins (oddbloke)]

Hi Matti,

That sounds like a separate bug from this one. Could you file a new bug (with the output of `cloud-init collect-logs` from a failing instance attached), please?

Thanks!

Dan

[Pengpeng Sun (pengpengs)]

Hi Matti,

With "disable_vmware_customization: false" entry added, its enabling Guest OS Customization with Cloud-Init. Was the bionic installed on a VMware VM and did Guest OS Customization?

Thanks,
Pengpeng

[Qi Zhang (keiraqiz)]

Can I be removed from the recipient list about GOSC server updates? I don’t think I have any chance to use GOSC testing servers for now.

Thanks
--
Best Regards,
Qi (Keira)

From: "<email address hidden>" <email address hidden>
Date: Thursday, October 17, 2019 at 13:48
To: GOSV-QE <email address hidden>
Subject: [GOSC Testbed Update] ESXi/VC Servers 20191017 : w2-hs2-e0214 for main Cycle59

Testbed:
Build
Product
Branch
Build Type
Changeset
Build Requestor
Build Start Time
Build End Time
Build Status
Host
Nimbus Deploy Status
ob-14845525
server
main
beta
7496542
cat-recommended
2019-10-11 09:17:51
2019-10-11 10:50:00
succeeded
10.197.4.34
success
ob-14845527
vcenter
main
beta
7496542
cat-recommended
2019-10-11 09:17:51
2019-10-11 10:41:39
succeeded
10.160.79.232
PASS

Account:
ESXi: root/ca$hc0w
VC: <email address hidden>/Admin!23

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3162