netplan configuration not re-generated

Bug #1847583 reported by Alexander Weber on 2019-10-10
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
cloud-init
Undecided
Unassigned

Bug Description

via https://bugs.launchpad.net/cloud-init/+bug/1846535/comments/40

steps to reproduce: (ec2/gcp/azure all have the same issue)

* start a ubuntu 18.04 instance
* update cloud-init to `19.2-36-g059d049c-0ubuntu2~18.04.1`

cloud-init-output.log

```
Cloud-init v. 19.2-36-g059d049c-0ubuntu1~18.04.1 running 'init-local' at Thu, 10 Oct 2019 08:33:00 +0000. Up 11.53 seconds.
Cloud-init v. 19.2-36-g059d049c-0ubuntu1~18.04.1 running 'init' at Thu, 10 Oct 2019 08:33:03 +0000. Up 14.77 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+----------------------------+---------------+--------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+----------------------------+---------------+--------+-------------------+
ci-info: | ens5 | True | 172.31.18.122 | 255.255.240.0 | global | 02:a8:af:c9:34:90 |
ci-info: | ens5 | True | fe80::a8:afff:fec9:3490/64 | . | link | 02:a8:af:c9:34:90 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +--------+------+----------------------------+---------------+--------+-------------------+
ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
ci-info: +-------+-------------+-------------+-----------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-------------+-------------+-----------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 172.31.16.1 | 0.0.0.0 | ens5 | UG |
ci-info: | 1 | 172.31.16.0 | 0.0.0.0 | 255.255.240.0 | ens5 | U |
ci-info: | 2 | 172.31.16.1 | 0.0.0.0 | 255.255.255.255 | ens5 | UH |
ci-info: +-------+-------------+-------------+-----------------+-----------+-------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | 1 | fe80::/64 | :: | ens5 | U |
ci-info: | 3 | local | :: | ens5 | U |
ci-info: | 4 | ff00::/8 | :: | ens5 | U |
ci-info: +-------+-------------+---------+-----------+-------+
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.

root@ip-172-31-18-122:~# networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 ens5 ether routable configured

2 links listed.

root@ip-172-31-18-122:~# cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        ens5:
            dhcp4: true
            dhcp6: true
            match:
                macaddress: 02:a8:af:c9:34:90
            set-name: ens5

```
* update packages and install kernel 5.3.5
* create a image of the now running instance
* start a instance from the image
* unable to connect
* stop the instance
* mount the disk to another running instance

mounting the disk to another instance, one can see that the netplan configuration was not updated

```
Cloud-init v. 19.2-36-g059d049c-0ubuntu1~18.04.1 running 'modules:config' at Thu, 10 Oct 2019 08:53:35 +0000. Up 24.40 seconds.
Cloud-init v. 19.2-36-g059d049c-0ubuntu1~18.04.1 running 'modules:final' at Thu, 10 Oct 2019 08:53:36 +0000. Up 25.32 seconds.
Cloud-init v. 19.2-36-g059d049c-0ubuntu1~18.04.1 finished at Thu, 10 Oct 2019 08:53:36 +0000. Datasource DataSourceEc2Local. Up 25.43 seconds
Cloud-init v. 19.2-36-g059d049c-0ubuntu2~18.04.1 running 'init-local' at Thu, 10 Oct 2019 09:15:51 +0000. Up 16.36 seconds.
Cloud-init v. 19.2-36-g059d049c-0ubuntu2~18.04.1 running 'init' at Thu, 10 Oct 2019 09:15:52 +0000. Up 17.47 seconds.
ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
ci-info: +--------+-------+-----------+-----------+-------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+-------+-----------+-----------+-------+-------------------+
ci-info: | ens5 | False | . | . | . | 06:c1:88:5c:97:58 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +--------+-------+-----------+-----------+-------+-------------------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: +-------+-------------+---------+-----------+-------+

root@ip-172-31-40-235:/mnt# cat etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        ens5:
            dhcp4: true
            dhcp6: true
            match:
                macaddress: 02:a8:af:c9:34:90
            set-name: ens5

```

Dan Watkins (daniel-thewatkins) wrote :

Hi Alexander,

Thanks for the bug! Can you collect /var/log/cloud-init.log and /var/log/cloud-init-output.log from an affected instance, attach them to the bug and set it back to New, please?

Thanks!

Dan

Changed in cloud-init:
status: New → Incomplete

This is the same problem we're having that I referenced in a different bug.

We had the same problem in both GCP and AWS. One of my early attempts to work-around the issue was apt hold the cloud-init package to an older version. When that failed I thought our problem was not cloud-init related, but turns out that it doesn't matter if I hold back the version cloud init was still using the latest release when handling the networking.

I've attached our cloud-init.log.

We ended up implementing the work around from a different bug report, disabling cloud init's networking configuration and writing our own netplan configuration. This allowed us to continue with new image builds.

Our image build pipeline is driven by CI, and generally does this:
- packer creates image based on a specific ubuntu-bionic image
- packer uses a combination of exec + salt provisioners
- packer copies the image to all regions and accounts where we deploy
- terraform is run updating launch templates to use the new AMI + apply user data
- other tooling cycles nodes through the autoscale group causing them to upgrade

I don't know if the fact that we use packer, build the image, then build again from that image is relevant, but it seems like it might be related to why the packer images using these AMI's with the default cloud-init work, but the images based off of these do not.

Changed in cloud-init:
status: Incomplete → New
Alexander Weber (deshke) wrote :

as wished one cloud-init-output.log

Alexander Weber (deshke) wrote :

and one cloud-init.log

Changed in cloud-init:
status: New → Triaged
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers