Bug #1869181 “[Focal] cloud-init service never get nework active...” : Bugs : OEM Priority Project

Alex Tu (alextu) on 2020-03-26

Changed in oem-priority:
importance:	Undecided → Critical

Revision history for this message

Paride Legovini (paride) wrote on 2020-03-27:

#1

Hello Alex, is it possible for you to collect the full cloud-init logs (using the `cloud-init collect-logs` command) and the full NetworkManager logs, and attach them to this report? Thanks!

Revision history for this message

Paul Larson (pwlars) wrote on 2020-03-30:

#2

I had this problem with the stock image for both eoan and focal as well. I don't think it's really the "recommended" solution, but the only way to fix it that I've found is to remove "Before=sysinit.target" from /lib/systemd/system/cloud-init.service in the target install. I haven't seen any negative effects from this so far.

Revision history for this message

Dan Watkins (oddbloke) wrote on 2020-03-31:

#3

Marking this as Incomplete as we're waiting on the logs requested in comment #1.

Changed in cloud-init:
status:	New → Incomplete

Revision history for this message

Alex Tu (alextu) wrote on 2020-04-08:

#4

cloud-init.tar.gz Edit (119.1 KiB, application/x-tar)

Revision history for this message

Alex Tu (alextu) wrote on 2020-04-08:

#5

journalctl.log Edit (865.6 KiB, text/plain)

the NetworkManager logs are covered by journalctl log.

Changed in cloud-init:
status:	Incomplete → New

Revision history for this message

Alex Tu (alextu) wrote on 2020-04-08:

#6

Sorry fo reply late due to busy days after vocation.
I provided the logs and changed the state to New for notice.

Changed in oem-priority:
assignee:	nobody → Alex Tu (alextu)

Revision history for this message

Ryan Harper (raharper) wrote on 2020-04-08:

#7

Download full text (3.8 KiB)

Something related to NetworkManager is preventing systemd-networkd from starting.

networkd-dispatcher[1163]: WARNING: systemd-networkd is not running, output will be incomplete.

Normally after network-pre.target, systemd-networkd-wait-online.service runs
and blocks until systemd-networkd has brough up configured interfaces.
systemd-networkd did not start so when cloud-init.service runs network is
*not* up like it expects.

systemd[1]: Finished Initial cloud-init job (pre-networking).
systemd[1]: Reached target Network (Pre).
systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
cloud-init[931]: Cloud-init v. 20.1-10-g71af48df-0ubuntu2 running 'init' at Wed, 08 Apr 2020 05:45:26 +0000. Up 31.04 seconds.
cloud-init[931]: ci-info: +++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++
cloud-init[931]: ci-info: +-----------+-------+-----------+-----------+-------+-------------------+
cloud-init[931]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
cloud-init[931]: ci-info: +-----------+-------+-----------+-----------+-------+-------------------+
cloud-init[931]: ci-info: | enp2s0 | False | . | . | . | 6c:2b:59:59:9d:ca |

Notice later, NetworkManager takes over the ethernet device:

kernel: r8169 0000:02:00.0 enp2s0: Link is Up - 1Gbps/Full - flow control rx/tx
kernel: IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0: link becomes ready
NetworkManager[1035]: <info> [1586324867.0369] device (enp2s0): carrier: link connected
NetworkManager[1035]: <info> [1586324867.0380] device (enp2s0): state change: unavailable -> disconnected (reason 'carrier-changed', sys-iface-state: 'managed')
NetworkManager[1035]: <info> [1586324867.0403] policy: auto-activating connection 'netplan-enp2s0' (7ea6f90b-3495-3533-948a-ef0035687c34)
NetworkManager[1035]: <info> [1586324867.0425] device (enp2s0): Activation: starting connection 'netplan-enp2s0' (7ea6f90b-3495-3533-948a-ef0035687c34)
NetworkManager[1035]: <info> [1586324867.0430] device (enp2s0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
NetworkManager[1035]: <info> [1586324867.0445] manager: NetworkManager state is now CONNECTING
NetworkManager[1035]: <info> [1586324867.0454] device (enp2s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
NetworkManager[1035]: <info> [1586324867.0470] device (enp2s0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
avahi-daemon[1011]: Joining mDNS multicast group on interface enp2s0.IPv4 with address 192.168.101.86.

The network config maas sends is for Ubuntu Server, which defaults ethernet
devices to be controlled by systemd-networkd; I suspect Desktop has a
different policy in place which puts all network devices in NetworkManager
control, so this config that MAAS sends (also, this MAAS looks old, it's
sending network config v1 (not v2, netplan)).

network:
    config:
    - id: enp2s0
        mac_address: 6c:2b:59:59:9d:ca
        mtu: 1500
        name: enp2s0
        subnets:
        - address: 192.168.101.86/24
            dns_nameservers:
            - 192.168.101.1
      ...

Something related to NetworkManager is preventing systemd-networkd from starting.

networkd-dispatcher[1163]: WARNING: systemd-networkd is not running, output will be incomplete.

Normally after network-pre.target, systemd-networkd-wait-online.service runs
and blocks until systemd-networkd has brough up configured interfaces.
systemd-networkd did not start so when cloud-init.service runs network is
*not* up like it expects.

systemd[1]: Finished Initial cloud-init job (pre-networking).
systemd[1]: Reached target Network (Pre).
systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
cloud-init[931]: Cloud-init v. 20.1-10-g71af48df-0ubuntu2 running 'init' at Wed, 08 Apr 2020 05:45:26 +0000. Up 31.04 seconds.
cloud-init[931]: ci-info: +++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++
cloud-init[931]: ci-info: +-----------+-------+-----------+-----------+-------+-------------------+
cloud-init[931]: ci-info: |   Device  |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |
cloud-init[931]: ci-info: +-----------+-------+-----------+-----------+-------+-------------------+
cloud-init[931]: ci-info: |   enp2s0  | False |     .     |     .     |   .   | 6c:2b:59:59:9d:ca |

Notice later, NetworkManager takes over the ethernet device:

kernel: r8169 0000:02:00.0 enp2s0: Link is Up - 1Gbps/Full - flow control rx/tx
kernel: IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0: link becomes ready
NetworkManager[1035]: <info>  [1586324867.0369] device (enp2s0): carrier: link connected
NetworkManager[1035]: <info>  [1586324867.0380] device (enp2s0): state change: unavailable -> disconnected (reason 'carrier-changed', sys-iface-state: 'managed')
NetworkManager[1035]: <info>  [1586324867.0403] policy: auto-activating connection 'netplan-enp2s0' (7ea6f90b-3495-3533-948a-ef0035687c34)
NetworkManager[1035]: <info>  [1586324867.0425] device (enp2s0): Activation: starting connection 'netplan-enp2s0' (7ea6f90b-3495-3533-948a-ef0035687c34)
NetworkManager[1035]: <info>  [1586324867.0430] device (enp2s0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
NetworkManager[1035]: <info>  [1586324867.0445] manager: NetworkManager state is now CONNECTING
NetworkManager[1035]: <info>  [1586324867.0454] device (enp2s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
NetworkManager[1035]: <info>  [1586324867.0470] device (enp2s0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
avahi-daemon[1011]: Joining mDNS multicast group on interface enp2s0.IPv4 with address 192.168.101.86.

The network config maas sends is for Ubuntu Server, which defaults ethernet
devices to be controlled by systemd-networkd;  I suspect Desktop has a
different policy in place which puts all network devices in NetworkManager
control, so this config that MAAS sends (also, this MAAS looks old, it's
sending network config v1 (not v2, netplan)).

network:
    config:
    -   id: enp2s0
        mac_address: 6c:2b:59:59:9d:ca
        mtu: 1500
        name: enp2s0
        subnets:
        -   address: 192.168.101.86/24
            dns_nameservers:
            - 192.168.101.1
            dns_search:
            - maas
            gateway: 192.168.101.1
            routes:
            -   destination: 10.101.46.0/24
                gateway: 192.168.101.1
                metric: 200
            type: static
        type: physical
    -   address:
        - 192.168.101.1
        search:
        - maas
        type: nameserver
    version: 1

On Focal desktops, this file is now present:

% cat /etc/netplan/01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager

You will need to use a script/preseed to modify this value to
be networkd like this:

network:
  version: 2
  renderer: networkd

Revision history for this message

Ryan Harper (raharper) wrote on 2020-04-08:

#8

Please confirm if such a file is present in the desktop image.

Changed in cloud-init:
status:	New → Incomplete

Revision history for this message

Alex Tu (alextu) wrote on 2020-04-10:

#9

Thanks a lot! it works with the change of #7.

I can have preseed/script for that change, but also wondering if it is possible to make could-init also natively compatible with the environment that installed NetworkManager.

Revision history for this message

Ryan Harper (raharper) wrote on 2020-04-10:

#10

Possibly; what's the origin of the image? Is there an "official" Ubuntu Desktop image? And if so, does it *only* have network-manager? or both? Can we get the tarball of the image you're deploying?

I suspect there is a netplan bug here for network-manager renderer where it does not add the network-manager-wait-online.service into the correct targets (they should run before network-online.target) so that NM can bring up networking before the target).

Revision history for this message

Alex Tu (alextu) wrote on 2020-04-14:

#11

I would like to revise #9, the steps of original test I did is:
1. trigger the maas deploy and wait until 1st login after provision done.
2. MaaS still show deploying when I manually login to target machine.
3. manually apply the change of #7 then reboot.
4. this reboot responded MaaS and the status be changed to deployed on MaaS.

But when I just inject the change of #7 into MaaS image to let the change is there before 1st login, the network will never up. So, I can not just use the change of #7 so far.

The origin of the image is OEM image which is almost the same as stock Ubuntu Focal but added dell-recovery. And we just let the deployed machine to 1st boot into recovery mode to install cloud-init related packages then target machine can response to MaaS server in next boot.(which way learned from certification team SRU process)

For the question: "And if so, does it *only* have network-manager? or both?"
Just have a check that both networkd-dispatcher and network-manager be installed (which is same as stock Ubuntu desktop)
- dpkg -l : https://paste.ubuntu.com/p/5m5wsq9y5g/

I can share the tarball but it is 3GB. I'm not sure if I can attach so big file on launchpad. I will try. BTW, the nature of dell-recovery will block out machines not produced by Dell. I am afraid you can not test it if you don't have a Dell laptop.

For MaaS image converted from official stock Ubuntu desktop, pwlars might have more experience to me because the certification team used to need it for SRU as well.

Revision history for this message

Ryan Harper (raharper) wrote on 2020-04-14:

#12

Download full text (3.7 KiB)

Thanks for the additional details.

From the logs and some local testing, the best choice for
getting things to work with the desktop image via MAAS is

to remove the 01-network-manager-all.yaml file before rebooting into the
target. This will allow networking to be fully managed by networkd.

We will likely need to file some bugs to get NetworkManager and cloud-init
and netplan to work together.

1) NetworkManager.service and NetworkManager-wait-online.service run too late
to work with cloud-init, This can be fixed by

a) Adding DefaultDependencies=no to both NM service files
b) cloud-init.service also adds a After=NetworkManager-wait-online.service

Note, both of these changes can be done via systemd-drop in files but
ideally they would go into the package itself.

2) The rendered NM configuration is not complete enough to auto enable
the configuration.

cloud-init writes a netplan config file and calls netplan generate which
with NetworkManager renderer populates /run/NetworkManager with config

# find /run/NetworkManager/ -type f
/run/NetworkManager/resolv.conf
/run/NetworkManager/no-stub-resolv.conf
/run/NetworkManager/devices/30
/run/NetworkManager/conf.d/10-globally-managed-devices.conf
/run/NetworkManager/system-connections/netplan-eth0.nmconnection

NetworkManager.service comes up and it can "see" an 'eth0' connection
and a 'netplan-eth0' connection. The 'eth0' connection is enabled.

# nmcli conn
NAME UUID TYPE DEVICE
eth0 e8ffebad-dba7-4035-a9a8-29d9e004e6c1 ethernet eth0
netplan-eth0 626dd384-8b3d-3690-9511-192b2c79b3fd ethernet --

eth0 has no IP (it's configured for dhcp)

After running netplan apply, we can see this change
# netplan --debug apply
** (generate:476): DEBUG: 15:39:02.332: Processing input file /etc/netplan/01-network-manager-all.yaml..
** (generate:476): DEBUG: 15:39:02.332: starting new processing pass
** (generate:476): DEBUG: 15:39:02.332: Processing input file /etc/netplan/50-cloud-init.yaml..
** (generate:476): DEBUG: 15:39:02.332: starting new processing pass
** (generate:476): DEBUG: 15:39:02.332: eth0: setting default backend to 2
** (generate:476): DEBUG: 15:39:02.332: Configuration is valid
** (generate:476): DEBUG: 15:39:02.332: Generating output files..
** (generate:476): DEBUG: 15:39:02.332: networkd: definition eth0 is not for us (backend 2)
(generate:476): GLib-DEBUG: 15:39:02.332: posix_spawn avoided (fd close requested)
DEBUG:netplan generated networkd configuration changed, restarting networkd
DEBUG:netplan generated NM configuration changed, restarting NM
DEBUG:eth0 not found in {}
DEBUG:Merged config:
network:
  bonds: {}
  bridges: {}
  ethernets:
    eth0:
      dhcp4: true
      match:
        macaddress: 00:16:3e:0d:4c:a7
      set-name: eth0
  vlans: {}
  wifis: {}

DEBUG:Skipping non-physical interface: lo
DEBUG:device eth0 operstate is up, not changing
DEBUG:{}
DEBUG:netplan triggering .link rules for lo
DEBUG:netplan triggering .link rules for eth0

# nmcli conn
NAME UUID TYPE DEVICE
netplan-eth0 626dd384-8b3d-3690-9511-192b2c79b3fd ethernet eth0

Looking into NM...

Thanks for the additional details.

From the logs and some local testing, the best choice for
getting things to work with the desktop image via MAAS is

to remove the 01-network-manager-all.yaml file before rebooting into the
target.  This will allow networking to be fully managed by networkd.

We will likely need to file some bugs to get NetworkManager and cloud-init
and netplan to work together.

1) NetworkManager.service and NetworkManager-wait-online.service run too late
to work with cloud-init,  This can be fixed by

a) Adding DefaultDependencies=no to both NM service files
   b) cloud-init.service also adds a After=NetworkManager-wait-online.service

Note, both of these changes can be done via systemd-drop in files but
   ideally they would go into the package itself.

2) The rendered NM configuration is not complete enough to auto enable
the configuration.

cloud-init writes a netplan config file and calls netplan generate which
with NetworkManager renderer populates /run/NetworkManager with config

# find /run/NetworkManager/ -type f
/run/NetworkManager/resolv.conf
/run/NetworkManager/no-stub-resolv.conf
/run/NetworkManager/devices/30
/run/NetworkManager/conf.d/10-globally-managed-devices.conf
/run/NetworkManager/system-connections/netplan-eth0.nmconnection

NetworkManager.service comes up and it can "see" an 'eth0' connection
and a 'netplan-eth0' connection.  The 'eth0' connection is enabled.

# nmcli conn
NAME          UUID                                  TYPE      DEVICE
eth0          e8ffebad-dba7-4035-a9a8-29d9e004e6c1  ethernet  eth0
netplan-eth0  626dd384-8b3d-3690-9511-192b2c79b3fd  ethernet  --

eth0 has no IP (it's configured for dhcp)

After running netplan apply, we can see this change
# netplan --debug apply
** (generate:476): DEBUG: 15:39:02.332: Processing input file /etc/netplan/01-network-manager-all.yaml..
** (generate:476): DEBUG: 15:39:02.332: starting new processing pass
** (generate:476): DEBUG: 15:39:02.332: Processing input file /etc/netplan/50-cloud-init.yaml..
** (generate:476): DEBUG: 15:39:02.332: starting new processing pass
** (generate:476): DEBUG: 15:39:02.332: eth0: setting default backend to 2
** (generate:476): DEBUG: 15:39:02.332: Configuration is valid
** (generate:476): DEBUG: 15:39:02.332: Generating output files..
** (generate:476): DEBUG: 15:39:02.332: networkd: definition eth0 is not for us (backend 2)
(generate:476): GLib-DEBUG: 15:39:02.332: posix_spawn avoided (fd close requested)
DEBUG:netplan generated networkd configuration changed, restarting networkd
DEBUG:netplan generated NM configuration changed, restarting NM
DEBUG:eth0 not found in {}
DEBUG:Merged config:
network:
  bonds: {}
  bridges: {}
  ethernets:
    eth0:
      dhcp4: true
      match:
        macaddress: 00:16:3e:0d:4c:a7
      set-name: eth0
  vlans: {}
  wifis: {}

DEBUG:Skipping non-physical interface: lo
DEBUG:device eth0 operstate is up, not changing
DEBUG:{}
DEBUG:netplan triggering .link rules for lo
DEBUG:netplan triggering .link rules for eth0

# nmcli conn
NAME          UUID                                  TYPE      DEVICE
netplan-eth0  626dd384-8b3d-3690-9511-192b2c79b3fd  ethernet  eth0

Looking into NM logs, there is a auto-enable policy that netplan apply
triggers that does not seem to happen with just restarting NM.

# journalctl -o short-precise -b -u NetworkManager.service  | grep policy
Apr 14 15:39:02.871859 f6 NetworkManager[520]: <info>  [1586878742.8718] policy: auto-activating connection 'netplan-eth0' (626dd384-8b3d-3690-9511-192b2c79b3fd)
Apr 14 15:39:02.887910 f6 NetworkManager[520]: <info>  [1586878742.8878] policy: set 'netplan-eth0' (eth0) as default for IPv4 routing and DNS

I don't think cloud-init itself should have to run 'netplan apply'; we don't
do this for networkd backend.

Rex Tsai (chihchun) on 2020-08-15

tags:

added: oem-priority

Yuan-Chen Cheng (ycheng-twn) on 2021-08-14

Changed in oem-priority:
importance:	Critical → High

Revision history for this message

James Falcon (falcojr) wrote on 2023-05-12:

#13

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3635

Changed in cloud-init:
status:	Incomplete → Expired

OEM Priority Project

[Focal] cloud-init service never get nework actived during MaaS deploy.

Bug Description

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Affects		Status	Importance	Assigned to	Milestone
	OEM Priority Project	New	High	Alex Tu
	cloud-init	Expired	Undecided	Unassigned