cloud-init sets wrong netmask causing broken network config on Oracle Cloud
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
cloud-init (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Bionic |
Fix Released
|
Undecided
|
Unassigned | ||
Focal |
Fix Released
|
Undecided
|
Unassigned | ||
Jammy |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
[ Impact ]
* On Oracle cloud with 22.04 and 22.10 images may not contain `/run/net-
The rendered network configuration is incomplete and lacks proper default routes and/or DNS configuration
resulting in improper network egress routes and rules and absent DNS settings resulting in hostname lookup errors.
Because IMDS data is incomplete to fully configure DNS, cloud-init 22.3.4 configures DHCP on the primary NIC based on the MAC address set in Oracle IMDS network confing and only defines static network config to setup secondary NICs and secondary routes.
[ Test Plan ]
Kinetic only (where 22.3.3 was released)
* download daily image for kinetic
* put daily ubuntu cloudimage to oracle object store via oci cmdline
* import dailyimage from via oracle cli from storage bucket
* launch imported customimage as a Flex instance type
* Validate that WARNING IMDS is shown implying writing net config from IMDS content instead of /run/net-ens3.conf
* validate failure on 22.3.3 no default route set and invalid netplan config
* setup network dhcp on primary interface manually
Bionic, Focal, Jammy and Kinetic
* upgrade cloud-init 22.3.4
* rm /etc/netplan/
* sudo cloud-init clean --logs --reboot
* Validate network config, default routes and nslookup canonical.com
[ Where problems could occur ]
* This behavior was a regression introduced only in kinetic and -proposed series 22.3.3 and returs to previous published behavior of cloud-init 22.2 which was dhcp on primary ethernet device so there should be no regression here beyond fixing 22.3.3 in kinetic and the -proposed streams.
[ Other Info ]
[ Original Description ]
Testing U22.10 Cloud image:
https:/
on Oracle Cloud, with with cloud-init 22.3-13-
cloud-init is able to find correct network information (note: the use if /21 as netmask):
ephemeral.
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '100.103.
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
url_helper.
ephemeral.
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '100.103.
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
url_helper.
url_helper.
url_helper.
url_helper.
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'del', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'del', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'down'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'del', '100.103.
However, at later stage it forgets about the netmask:
stages.py[DEBUG]: applying net config names for {'config': [{'name': 'ens3', 'type': 'physical', 'mac_address': '02:00:
creates:
$ cat /run/systemd/
[Match]
MACAddress=
[Link]
Name=ens3
WakeOnLan=off
MTUBytes=9000
[Match]
MACAddress=
Name=ens3
[Link]
MTUBytes=9000
[Network]
LinkLocalAddres
Address=
Note /24 here.
If one is unlucky and get an IP in upper octets (in the subnet),
ip command will refuse to set default gateway in routing table as IP of gw is outside subnet.
Hence, we end up with no working network configuration:
root@v:# ip a
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,
link/ether 02:00:17:06:ae:e9 brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 100.103.27.215/24 brd 100.103.27.255 scope global ens3
valid_lft forever preferred_lft forever
inet6 fe80::17ff:
valid_lft forever preferred_lft forever
root@v:# ip r
100.103.27.0/24 dev ens3 proto kernel scope link src 100.103.27.215
root@v:#
Any ideas what's causing this or how to debug more deeply?
Thanks in advance.
Changed in cloud-init (Ubuntu): | |
status: | Triaged → Fix Committed |
description: | updated |
summary: |
- cloud-init sets wrong netmask causing borken network config on Oracle + cloud-init sets wrong netmask causing broken network config on Oracle Cloud |
description: | updated |
FYI: reverting to cloud-init 22.1-14- g2e17a0d6- 0ubuntu1~ 22.04.5 (even on U22.10) seems to resolve the issue.