| 2022-09-15 08:45:12 |
Terje Røsten |
bug |
|
|
added bug |
| 2022-09-16 11:17:49 |
Terje Røsten |
attachment added |
|
cloud-init.tar.gz https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1989686/+attachment/5616432/+files/cloud-init.tar.gz |
|
| 2022-09-16 14:32:19 |
James Falcon |
cloud-init (Ubuntu): status |
New |
Triaged |
|
| 2022-09-16 14:32:24 |
James Falcon |
cloud-init (Ubuntu): importance |
Undecided |
High |
|
| 2022-09-16 20:48:36 |
James Falcon |
cloud-init (Ubuntu): status |
Triaged |
Fix Committed |
|
| 2022-09-21 05:55:09 |
Launchpad Janitor |
cloud-init (Ubuntu): status |
Fix Committed |
Fix Released |
|
| 2022-10-03 23:41:31 |
Chad Smith |
description |
Testing U22.10 Cloud image:
https://cloud-images.ubuntu.com/kinetic/20220914/kinetic-server-cloudimg-amd64.img
on Oracle Cloud, with with cloud-init 22.3-13-g70ce6442-0ubuntu1~22.10.1.
cloud-init is able to find correct network information (note: the use if /21 as netmask):
ephemeral.py[DEBUG]: Attempting setup of ephemeral network on ens3 with 100.103.27.215/21 brd 100.103.31.255
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '100.103.27.215/21', 'broadcast', '100.103.31.255', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
url_helper.py[DEBUG]: [0/3] open 'http://169.254.169.254/opc/v2/instance/' with {'url': 'http://169.254.169.254/opc/v2/instance/', 'stream': False
ephemeral.py[DEBUG]: Attempting setup of ephemeral network on ens3 with 100.103.27.215/21 brd 100.103.31.255
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '100.103.27.215/21', 'broadcast', '100.103.31.255', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
url_helper.py[DEBUG]: [0/3] open 'http://169.254.169.254/opc/v2/instance/' with {'url': 'http://169.254.169.254/opc/v2/instance/', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'headers': {'User-Agent': 'Cloud-Init/22.3-13-g70ce6442-0ubuntu1~22.10.1', 'Authorization': 'Bearer Oracle'}} configuration
url_helper.py[DEBUG]: Read from http://169.254.169.254/opc/v2/instance/ (200, 4154b) after 1 attempts
url_helper.py[DEBUG]: [0/3] open 'http://169.254.169.254/opc/v2/vnics/' with {'url': 'http://169.254.169.254/opc/v2/vnics/', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'headers': {'User-Agent': 'Cloud-Init/22.3-13-g70ce6442-0ubuntu1~22.10.1', 'Authorization': 'Bearer Oracle'}} configuration
url_helper.py[DEBUG]: Read from http://169.254.169.254/opc/v2/vnics/ (200, 280b) after 1 attempts
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'del', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'del', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'down'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'del', '100.103.27.215/21', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
However, at later stage it forgets about the netmask:
stages.py[DEBUG]: applying net config names for {'config': [{'name': 'ens3', 'type': 'physical', 'mac_address': '02:00:17:06:ae:e9', 'mtu': 9000, 'subnets': [{'type': 'static', 'address': '100.103.27.215'}]}], 'version': 1}
creates:
$ cat /run/systemd/network/10-netplan-ens3.*
[Match]
MACAddress=02:00:17:06:ae:e9
[Link]
Name=ens3
WakeOnLan=off
MTUBytes=9000
[Match]
MACAddress=02:00:17:06:ae:e9
Name=ens3
[Link]
MTUBytes=9000
[Network]
LinkLocalAddressing=ipv6
Address=100.103.27.215/24
Note /24 here.
If one is unlucky and get an IP in upper octets (in the subnet),
ip command will refuse to set default gateway in routing table as IP of gw is outside subnet.
Hence, we end up with no working network configuration:
root@v:# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
link/ether 02:00:17:06:ae:e9 brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 100.103.27.215/24 brd 100.103.27.255 scope global ens3
valid_lft forever preferred_lft forever
inet6 fe80::17ff:fe06:aee9/64 scope link
valid_lft forever preferred_lft forever
root@v:# ip r
100.103.27.0/24 dev ens3 proto kernel scope link src 100.103.27.215
root@v:#
Any ideas what's causing this or how to debug more deeply?
Thanks in advance. |
[ Impact ]
* On Oracle cloud with 22.04 and 22.10 images may not contain `/run/net-ens3.conf` network definitions when launching imported custom cloud-images from Oracle object store. cloud-init 22.3.3 attempts to render static network configuration in these cases and rely solely on Oracle IMDS for network configuration as published at http://169.254.169.254/opc/v2/vnics.
The rendered network configuration is incomplete and lacks proper default routes and/or DNS configuration
resulting in improper network egress routes and rules and absent DNS settings resulting in hostname lookup errors.
Because IMDS data is incomplete to fully configure DNS, cloud-init 22.3.4 configures DHCP on the primary NIC based on the MAC address set in Oracle IMDS network confing and only defines static network config to setup secondary NICs and secondary routes.
[ Test Plan ]
* Launch daily oracle image
* remove kblic default network config file /run/net-ens3.config to foce cloud-init to render network from Oracle IMDS
* upgrade cloud-init 22.3.4
* sudo cloud-init clean --logs
* snapshot this custom image and launch it on oci cmdline
* Validate network config, routes and nslookup canonical.com
[ Where problems could occur ]
* This behavior was a regression introduced only in kinetic and -proposed series 22.3.3 and returs to previous published behavior of cloud-init 22.2 which was dhcp on primary ethernet device so there should be no regression here beyond fixing 22.3.3 in kinetic and the -proposed streams.
[ Other Info ]
[ Original Description ]
Testing U22.10 Cloud image:
https://cloud-images.ubuntu.com/kinetic/20220914/kinetic-server-cloudimg-amd64.img
on Oracle Cloud, with with cloud-init 22.3-13-g70ce6442-0ubuntu1~22.10.1.
cloud-init is able to find correct network information (note: the use if /21 as netmask):
ephemeral.py[DEBUG]: Attempting setup of ephemeral network on ens3 with 100.103.27.215/21 brd 100.103.31.255
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '100.103.27.215/21', 'broadcast', '100.103.31.255', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
url_helper.py[DEBUG]: [0/3] open 'http://169.254.169.254/opc/v2/instance/' with {'url': 'http://169.254.169.254/opc/v2/instance/', 'stream': False
ephemeral.py[DEBUG]: Attempting setup of ephemeral network on ens3 with 100.103.27.215/21 brd 100.103.31.255
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '100.103.27.215/21', 'broadcast', '100.103.31.255', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
url_helper.py[DEBUG]: [0/3] open 'http://169.254.169.254/opc/v2/instance/' with {'url': 'http://169.254.169.254/opc/v2/instance/', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'headers': {'User-Agent': 'Cloud-Init/22.3-13-g70ce6442-0ubuntu1~22.10.1', 'Authorization': 'Bearer Oracle'}} configuration
url_helper.py[DEBUG]: Read from http://169.254.169.254/opc/v2/instance/ (200, 4154b) after 1 attempts
url_helper.py[DEBUG]: [0/3] open 'http://169.254.169.254/opc/v2/vnics/' with {'url': 'http://169.254.169.254/opc/v2/vnics/', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'headers': {'User-Agent': 'Cloud-Init/22.3-13-g70ce6442-0ubuntu1~22.10.1', 'Authorization': 'Bearer Oracle'}} configuration
url_helper.py[DEBUG]: Read from http://169.254.169.254/opc/v2/vnics/ (200, 280b) after 1 attempts
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'del', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'del', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'down'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'del', '100.103.27.215/21', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
However, at later stage it forgets about the netmask:
stages.py[DEBUG]: applying net config names for {'config': [{'name': 'ens3', 'type': 'physical', 'mac_address': '02:00:17:06:ae:e9', 'mtu': 9000, 'subnets': [{'type': 'static', 'address': '100.103.27.215'}]}], 'version': 1}
creates:
$ cat /run/systemd/network/10-netplan-ens3.*
[Match]
MACAddress=02:00:17:06:ae:e9
[Link]
Name=ens3
WakeOnLan=off
MTUBytes=9000
[Match]
MACAddress=02:00:17:06:ae:e9
Name=ens3
[Link]
MTUBytes=9000
[Network]
LinkLocalAddressing=ipv6
Address=100.103.27.215/24
Note /24 here.
If one is unlucky and get an IP in upper octets (in the subnet),
ip command will refuse to set default gateway in routing table as IP of gw is outside subnet.
Hence, we end up with no working network configuration:
root@v:# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
link/ether 02:00:17:06:ae:e9 brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 100.103.27.215/24 brd 100.103.27.255 scope global ens3
valid_lft forever preferred_lft forever
inet6 fe80::17ff:fe06:aee9/64 scope link
valid_lft forever preferred_lft forever
root@v:# ip r
100.103.27.0/24 dev ens3 proto kernel scope link src 100.103.27.215
root@v:#
Any ideas what's causing this or how to debug more deeply?
Thanks in advance. |
|
| 2022-10-03 23:48:27 |
Chris Halse Rogers |
cloud-init (Ubuntu Bionic): status |
New |
Fix Committed |
|
| 2022-10-03 23:48:29 |
Chris Halse Rogers |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2022-10-03 23:48:31 |
Chris Halse Rogers |
bug |
|
|
added subscriber SRU Verification |
| 2022-10-03 23:48:36 |
Chris Halse Rogers |
tags |
|
verification-needed verification-needed-bionic |
|
| 2022-10-03 23:55:19 |
Chris Halse Rogers |
cloud-init (Ubuntu Focal): status |
New |
Fix Committed |
|
| 2022-10-03 23:55:25 |
Chris Halse Rogers |
tags |
verification-needed verification-needed-bionic |
verification-needed verification-needed-bionic verification-needed-focal |
|
| 2022-10-03 23:57:18 |
Chris Halse Rogers |
cloud-init (Ubuntu Jammy): status |
New |
Fix Committed |
|
| 2022-10-03 23:57:25 |
Chris Halse Rogers |
tags |
verification-needed verification-needed-bionic verification-needed-focal |
verification-needed verification-needed-bionic verification-needed-focal verification-needed-jammy |
|
| 2022-10-04 14:50:59 |
Chad Smith |
summary |
cloud-init sets wrong netmask causing borken network config on Oracle Cloud |
cloud-init sets wrong netmask causing broken network config on Oracle Cloud |
|
| 2022-10-07 02:52:36 |
Chad Smith |
description |
[ Impact ]
* On Oracle cloud with 22.04 and 22.10 images may not contain `/run/net-ens3.conf` network definitions when launching imported custom cloud-images from Oracle object store. cloud-init 22.3.3 attempts to render static network configuration in these cases and rely solely on Oracle IMDS for network configuration as published at http://169.254.169.254/opc/v2/vnics.
The rendered network configuration is incomplete and lacks proper default routes and/or DNS configuration
resulting in improper network egress routes and rules and absent DNS settings resulting in hostname lookup errors.
Because IMDS data is incomplete to fully configure DNS, cloud-init 22.3.4 configures DHCP on the primary NIC based on the MAC address set in Oracle IMDS network confing and only defines static network config to setup secondary NICs and secondary routes.
[ Test Plan ]
* Launch daily oracle image
* remove kblic default network config file /run/net-ens3.config to foce cloud-init to render network from Oracle IMDS
* upgrade cloud-init 22.3.4
* sudo cloud-init clean --logs
* snapshot this custom image and launch it on oci cmdline
* Validate network config, routes and nslookup canonical.com
[ Where problems could occur ]
* This behavior was a regression introduced only in kinetic and -proposed series 22.3.3 and returs to previous published behavior of cloud-init 22.2 which was dhcp on primary ethernet device so there should be no regression here beyond fixing 22.3.3 in kinetic and the -proposed streams.
[ Other Info ]
[ Original Description ]
Testing U22.10 Cloud image:
https://cloud-images.ubuntu.com/kinetic/20220914/kinetic-server-cloudimg-amd64.img
on Oracle Cloud, with with cloud-init 22.3-13-g70ce6442-0ubuntu1~22.10.1.
cloud-init is able to find correct network information (note: the use if /21 as netmask):
ephemeral.py[DEBUG]: Attempting setup of ephemeral network on ens3 with 100.103.27.215/21 brd 100.103.31.255
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '100.103.27.215/21', 'broadcast', '100.103.31.255', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
url_helper.py[DEBUG]: [0/3] open 'http://169.254.169.254/opc/v2/instance/' with {'url': 'http://169.254.169.254/opc/v2/instance/', 'stream': False
ephemeral.py[DEBUG]: Attempting setup of ephemeral network on ens3 with 100.103.27.215/21 brd 100.103.31.255
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '100.103.27.215/21', 'broadcast', '100.103.31.255', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
url_helper.py[DEBUG]: [0/3] open 'http://169.254.169.254/opc/v2/instance/' with {'url': 'http://169.254.169.254/opc/v2/instance/', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'headers': {'User-Agent': 'Cloud-Init/22.3-13-g70ce6442-0ubuntu1~22.10.1', 'Authorization': 'Bearer Oracle'}} configuration
url_helper.py[DEBUG]: Read from http://169.254.169.254/opc/v2/instance/ (200, 4154b) after 1 attempts
url_helper.py[DEBUG]: [0/3] open 'http://169.254.169.254/opc/v2/vnics/' with {'url': 'http://169.254.169.254/opc/v2/vnics/', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'headers': {'User-Agent': 'Cloud-Init/22.3-13-g70ce6442-0ubuntu1~22.10.1', 'Authorization': 'Bearer Oracle'}} configuration
url_helper.py[DEBUG]: Read from http://169.254.169.254/opc/v2/vnics/ (200, 280b) after 1 attempts
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'del', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'del', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'down'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'del', '100.103.27.215/21', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
However, at later stage it forgets about the netmask:
stages.py[DEBUG]: applying net config names for {'config': [{'name': 'ens3', 'type': 'physical', 'mac_address': '02:00:17:06:ae:e9', 'mtu': 9000, 'subnets': [{'type': 'static', 'address': '100.103.27.215'}]}], 'version': 1}
creates:
$ cat /run/systemd/network/10-netplan-ens3.*
[Match]
MACAddress=02:00:17:06:ae:e9
[Link]
Name=ens3
WakeOnLan=off
MTUBytes=9000
[Match]
MACAddress=02:00:17:06:ae:e9
Name=ens3
[Link]
MTUBytes=9000
[Network]
LinkLocalAddressing=ipv6
Address=100.103.27.215/24
Note /24 here.
If one is unlucky and get an IP in upper octets (in the subnet),
ip command will refuse to set default gateway in routing table as IP of gw is outside subnet.
Hence, we end up with no working network configuration:
root@v:# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
link/ether 02:00:17:06:ae:e9 brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 100.103.27.215/24 brd 100.103.27.255 scope global ens3
valid_lft forever preferred_lft forever
inet6 fe80::17ff:fe06:aee9/64 scope link
valid_lft forever preferred_lft forever
root@v:# ip r
100.103.27.0/24 dev ens3 proto kernel scope link src 100.103.27.215
root@v:#
Any ideas what's causing this or how to debug more deeply?
Thanks in advance. |
[ Impact ]
* On Oracle cloud with 22.04 and 22.10 images may not contain `/run/net-ens3.conf` network definitions when launching imported custom cloud-images from Oracle object store. cloud-init 22.3.3 attempts to render static network configuration in these cases and rely solely on Oracle IMDS for network configuration as published at http://169.254.169.254/opc/v2/vnics.
The rendered network configuration is incomplete and lacks proper default routes and/or DNS configuration
resulting in improper network egress routes and rules and absent DNS settings resulting in hostname lookup errors.
Because IMDS data is incomplete to fully configure DNS, cloud-init 22.3.4 configures DHCP on the primary NIC based on the MAC address set in Oracle IMDS network confing and only defines static network config to setup secondary NICs and secondary routes.
[ Test Plan ]
Kinetic only (where 22.3.3 was released)
* download daily image for kinetic
* put daily ubuntu cloudimage to oracle object store via oci cmdline
* import dailyimage from via oracle cli from storage bucket
* launch imported customimage as a Flex instance type
* Validate that WARNING IMDS is shown implying writing net config from IMDS content instead of /run/net-ens3.conf
* validate failure on 22.3.3 no default route set and invalid netplan config
* setup network dhcp on primary interface manually
Bionic, Focal, Jammy and Kinetic
* upgrade cloud-init 22.3.4
* rm /etc/netplan/50-cloud-init.yaml
* sudo cloud-init clean --logs --reboot
* Validate network config, default routes and nslookup canonical.com
[ Where problems could occur ]
* This behavior was a regression introduced only in kinetic and -proposed series 22.3.3 and returs to previous published behavior of cloud-init 22.2 which was dhcp on primary ethernet device so there should be no regression here beyond fixing 22.3.3 in kinetic and the -proposed streams.
[ Other Info ]
[ Original Description ]
Testing U22.10 Cloud image:
https://cloud-images.ubuntu.com/kinetic/20220914/kinetic-server-cloudimg-amd64.img
on Oracle Cloud, with with cloud-init 22.3-13-g70ce6442-0ubuntu1~22.10.1.
cloud-init is able to find correct network information (note: the use if /21 as netmask):
ephemeral.py[DEBUG]: Attempting setup of ephemeral network on ens3 with 100.103.27.215/21 brd 100.103.31.255
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '100.103.27.215/21', 'broadcast', '100.103.31.255', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
url_helper.py[DEBUG]: [0/3] open 'http://169.254.169.254/opc/v2/instance/' with {'url': 'http://169.254.169.254/opc/v2/instance/', 'stream': False
ephemeral.py[DEBUG]: Attempting setup of ephemeral network on ens3 with 100.103.27.215/21 brd 100.103.31.255
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'add', '100.103.27.215/21', 'broadcast', '100.103.31.255', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'append', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
url_helper.py[DEBUG]: [0/3] open 'http://169.254.169.254/opc/v2/instance/' with {'url': 'http://169.254.169.254/opc/v2/instance/', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'headers': {'User-Agent': 'Cloud-Init/22.3-13-g70ce6442-0ubuntu1~22.10.1', 'Authorization': 'Bearer Oracle'}} configuration
url_helper.py[DEBUG]: Read from http://169.254.169.254/opc/v2/instance/ (200, 4154b) after 1 attempts
url_helper.py[DEBUG]: [0/3] open 'http://169.254.169.254/opc/v2/vnics/' with {'url': 'http://169.254.169.254/opc/v2/vnics/', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'headers': {'User-Agent': 'Cloud-Init/22.3-13-g70ce6442-0ubuntu1~22.10.1', 'Authorization': 'Bearer Oracle'}} configuration
url_helper.py[DEBUG]: Read from http://169.254.169.254/opc/v2/vnics/ (200, 280b) after 1 attempts
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'del', '169.254.0.0/16', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-4', 'route', 'del', '0.0.0.0/0', 'via', '100.103.24.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'link', 'set', 'dev', 'ens3', 'down'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['ip', '-family', 'inet', 'addr', 'del', '100.103.27.215/21', 'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
However, at later stage it forgets about the netmask:
stages.py[DEBUG]: applying net config names for {'config': [{'name': 'ens3', 'type': 'physical', 'mac_address': '02:00:17:06:ae:e9', 'mtu': 9000, 'subnets': [{'type': 'static', 'address': '100.103.27.215'}]}], 'version': 1}
creates:
$ cat /run/systemd/network/10-netplan-ens3.*
[Match]
MACAddress=02:00:17:06:ae:e9
[Link]
Name=ens3
WakeOnLan=off
MTUBytes=9000
[Match]
MACAddress=02:00:17:06:ae:e9
Name=ens3
[Link]
MTUBytes=9000
[Network]
LinkLocalAddressing=ipv6
Address=100.103.27.215/24
Note /24 here.
If one is unlucky and get an IP in upper octets (in the subnet),
ip command will refuse to set default gateway in routing table as IP of gw is outside subnet.
Hence, we end up with no working network configuration:
root@v:# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
link/ether 02:00:17:06:ae:e9 brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 100.103.27.215/24 brd 100.103.27.255 scope global ens3
valid_lft forever preferred_lft forever
inet6 fe80::17ff:fe06:aee9/64 scope link
valid_lft forever preferred_lft forever
root@v:# ip r
100.103.27.0/24 dev ens3 proto kernel scope link src 100.103.27.215
root@v:#
Any ideas what's causing this or how to debug more deeply?
Thanks in advance. |
|
| 2022-10-07 04:12:00 |
Chad Smith |
tags |
verification-needed verification-needed-bionic verification-needed-focal verification-needed-jammy |
verification-done verification-done-bionic verification-done-focal verification-done-jammy |
|
| 2022-10-14 00:08:39 |
Launchpad Janitor |
cloud-init (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2022-10-14 00:08:39 |
Launchpad Janitor |
bug watch added |
|
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1014584 |
|
| 2022-10-14 00:08:50 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2022-10-14 00:09:52 |
Launchpad Janitor |
cloud-init (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2022-10-14 00:12:23 |
Launchpad Janitor |
cloud-init (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
