NetworkManager service does not start during bootup

[jean-christophe manciot]

Ubuntu 16.04 4.4.0-22
systemd 229-4
network-manager 1.1.93
network-manager-gnome 1.2.0

Multiple services and targets are deleted by systemd to "break ordering cycle" (cf. boot.log), but NetworkManager is the only one of them (network.target,rpcbind,sysinit.target,open-iscsi.service,remote-fs-pre.target,iscsid.service,zfs-fuse.service - cf. their respective logs) which is not started after bootup:

sudo systemctl status NetworkManager.service
● NetworkManager.service - Network Manager
   Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

May 18 05:00:10 samsung-ubuntu systemd[1]: NetworkManager.service: Job NetworkManager.service/verify-active deleted to break ordering cycle starting with network.target/start

Attachments:
-----------
- journalctl -l -b0 --system _COMM=systemd > boot.log
- systemctl -l status rpcbind > rpcbind.log
- systemctl -l status network.target > network.target.txt
- systemctl -l status remote-fs-pre.target > remote-fs-pre.target.log
- systemctl -l status sysinit.target > sysinit.target.log
- systemctl -l status zfs-fuse.service > zfs-fuse.service.log
- systemctl -l status open-iscsi.service > open-iscsi.service.log
- systemctl -l status iscsid.service > iscsid.service.log

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[rot13 (pp)]

Seems related to bug #1487679, the only difference being iscsi requiring network instead of nbd-client.

[Marzanna (marzanna)]

This happens only sometimes on my system. In most of cases NetworkManager starts normally but dnscrypt-proxy fails. It happens on two installations.
$ sudo journalctl | grep network
июл 26 11:53:24 sunsetshimmer systemd[1]: network.target: Found ordering cycle on network.target/start
июл 26 11:53:24 sunsetshimmer systemd[1]: network.target: Found dependency on NetworkManager.service/start
июл 26 11:53:24 sunsetshimmer systemd[1]: network.target: Found dependency on dbus.service/start
июл 26 11:53:24 sunsetshimmer systemd[1]: network.target: Found dependency on basic.target/start
июл 26 11:53:24 sunsetshimmer systemd[1]: network.target: Found dependency on sockets.target/start
июл 26 11:53:24 sunsetshimmer systemd[1]: network.target: Found dependency on dnscrypt-proxy.socket/start
июл 26 11:53:24 sunsetshimmer systemd[1]: network.target: Found dependency on network.target/start
июл 26 11:53:24 sunsetshimmer systemd[1]: network.target: Breaking ordering cycle by deleting job NetworkManager.service/start
июл 26 11:53:24 sunsetshimmer systemd[1]: NetworkManager.service: Job NetworkManager.service/start deleted to break ordering cycle starting with network.target/start
июл 26 11:53:26 sunsetshimmer systemd[1]: Started Trigger resolvconf update for networkd DNS.
июл 26 11:53:26 sunsetshimmer systemd[1]: Starting Raise network interfaces...
июл 26 11:53:26 sunsetshimmer systemd[1]: Started Raise network interfaces.
июл 26 11:53:26 sunsetshimmer systemd[1]: Starting Anonymizing overlay network for TCP (multi-instance-master)...
июл 26 11:53:26 sunsetshimmer systemd[1]: Starting Anonymizing overlay network for TCP...
июл 26 11:53:26 sunsetshimmer systemd[1]: Starting load-balanced unspoofable packet switching network...
июл 26 11:53:26 sunsetshimmer systemd[1]: Started Anonymizing overlay network for TCP (multi-instance-master).
июл 26 11:53:27 sunsetshimmer systemd[1]: Started load-balanced unspoofable packet switching network.
июл 26 11:53:27 sunsetshimmer systemd[1]: Started Anonymizing overlay network for TCP.

[Mel B. (bighype)]

About 1 in 5 reboots result in this bug not starting my NetworkManager. I get something along these lines:

Sep 28 16:18:13 melb systemd[1]: Set hostname to <melb>.
Sep 28 16:18:13 melb systemd[1]: network.target: Found ordering cycle on network.target/start
Sep 28 16:18:13 melb systemd[1]: network.target: Found dependency on NetworkManager.service/start
Sep 28 16:18:13 melb systemd[1]: network.target: Found dependency on dbus.service/start
Sep 28 16:18:13 melb systemd[1]: network.target: Found dependency on basic.target/start
Sep 28 16:18:13 melb systemd[1]: network.target: Found dependency on sockets.target/start
Sep 28 16:18:13 melb systemd[1]: network.target: Found dependency on dnscrypt-proxy.socket/start
Sep 28 16:18:13 melb systemd[1]: network.target: Found dependency on network.target/start
Sep 28 16:18:13 melb systemd[1]: network.target: Breaking ordering cycle by deleting job NetworkManager.service/start
Sep 28 16:18:13 melb systemd[1]: NetworkManager.service: Job NetworkManager.service/start deleted to break ordering cycle starting with network.target/start

[anairamzap (anairamzap)]

I can confirm I'm getting this error too. Althou it seems to be a bit random I can confirm I didn't get it in Ubuntu 12.04.
I wish I could be more precise but I can't tell if this is something caused by other process... so I can't tell if started happening on 16.04 or after installing some package.

Ubuntu 16.04
3.13.0-92-generic
NetworkManager --version: 1.2.2

Here I paste some interesting lines from dmesg:

-----
[ 19.350616] systemd[1]: network-online.target: Found ordering cycle on network-online.target/start
[ 19.350623] systemd[1]: network-online.target: Found dependency on NetworkManager-wait-online.service/start
[ 19.350628] systemd[1]: network-online.target: Found dependency on NetworkManager.service/start
[ 19.350631] systemd[1]: network-online.target: Found dependency on basic.target/start
[ 19.350634] systemd[1]: network-online.target: Found dependency on sockets.target/start
[ 19.350638] systemd[1]: network-online.target: Found dependency on uuidd.socket/start
[ 19.350641] systemd[1]: network-online.target: Found dependency on sysinit.target/start
[ 19.350645] systemd[1]: network-online.target: Found dependency on firestarter.service/start
[ 19.350648] systemd[1]: network-online.target: Found dependency on network-online.target/start
[ 19.350654] systemd[1]: network-online.target: Breaking ordering cycle by deleting job NetworkManager-wait-online.service/start
[ 19.350659] systemd[1]: NetworkManager-wait-online.service: Job NetworkManager-wait-online.service/start deleted to break ordering cycle starting with network-online.target/start
[ 19.350750] systemd[1]: network.target: Found ordering cycle on network.target/start
[ 19.350754] systemd[1]: network.target: Found dependency on NetworkManager.service/start
[ 19.350758] systemd[1]: network.target: Found dependency on basic.target/start
[ 19.350762] systemd[1]: network.target: Found dependency on sockets.target/start
[ 19.350765] systemd[1]: network.target: Found dependency on uuidd.socket/start
[ 19.350768] systemd[1]: network.target: Found dependency on sysinit.target/start
[ 19.350771] systemd[1]: network.target: Found dependency on firestarter.service/start
[ 19.350775] systemd[1]: network.target: Found dependency on network-online.target/start
[ 19.350778] systemd[1]: network.target: Found dependency on network.target/start
[ 19.350781] systemd[1]: network.target: Breaking ordering cycle by deleting job NetworkManager.service/start
[ 19.350785] systemd[1]: NetworkManager.service: Job NetworkManager.service/start deleted to break ordering cycle starting with network.target/start
-----

I'm not sure if the problem is in NetworkManager service itself or if there is any other service messing around and causing these "ordering cycles" which seems to end killing the NetworkManager process.

[Martin (mcsk)]

[ 6.678485] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
[ 6.678598] systemd[1]: Detected architecture x86-64.
[ 6.678822] systemd[1]: Set hostname to <nb>.
[ 6.772575] systemd[1]: sockets.target: Found ordering cycle on sockets.target/start
[ 6.772578] systemd[1]: sockets.target: Found dependency on uuidd.socket/start
[ 6.772580] systemd[1]: sockets.target: Found dependency on sysinit.target/start
[ 6.772581] systemd[1]: sockets.target: Found dependency on firestarter.service/start
[ 6.772583] systemd[1]: sockets.target: Found dependency on network-online.target/start
[ 6.772584] systemd[1]: sockets.target: Found dependency on NetworkManager-wait-online.service/start
[ 6.772585] systemd[1]: sockets.target: Found dependency on basic.target/start
[ 6.772586] systemd[1]: sockets.target: Found dependency on sockets.target/start
[ 6.772588] systemd[1]: sockets.target: Breaking ordering cycle by deleting job uuidd.socket/start
[ 6.772590] systemd[1]: uuidd.socket: Job uuidd.socket/start deleted to break ordering cycle starting with sockets.target/start
[ 6.772624] systemd[1]: sockets.target: Found ordering cycle on sockets.target/start
[ 6.772625] systemd[1]: sockets.target: Found dependency on avahi-daemon.socket/start
[ 6.772627] systemd[1]: sockets.target: Found dependency on sysinit.target/start
[ 6.772628] systemd[1]: sockets.target: Found dependency on firestarter.service/start
[ 6.772629] systemd[1]: sockets.target: Found dependency on network-online.target/start
[ 6.772631] systemd[1]: sockets.target: Found dependency on NetworkManager-wait-online.service/start
[ 6.772632] systemd[1]: sockets.target: Found dependency on basic.target/start
[ 6.772633] systemd[1]: sockets.target: Found dependency on sockets.target/start
[ 6.772634] systemd[1]: sockets.target: Breaking ordering cycle by deleting job avahi-daemon.socket/start
[ 6.772636] systemd[1]: avahi-daemon.socket: Job avahi-daemon.socket/start deleted to break ordering cycle starting with sockets.target/start
[ 6.772653] systemd[1]: network-online.target: Found ordering cycle on network-online.target/start
[ 6.772654] systemd[1]: network-online.target: Found dependency on network.target/start
[ 6.772656] systemd[1]: network-online.target: Found dependency on NetworkManager.service/start
[ 6.772657] systemd[1]: network-online.target: Found dependency on dbus.service/start
[ 6.772658] systemd[1]: network-online.target: Found dependency on dbus.socket/start
[ 6.772660] systemd[1]: network-online.target: Found dependency on sysinit.target/start
[ 6.772661] systemd[1]: network-online.target: Found dependency on firestarter.service/start
[ 6.772662] systemd[1]: network-online.target: Found dependency on network-online.target/start
[ 6.772663] systemd[1]: network-online.target: Breaking ordering cycle by deleting job network.target/start
[ 6.772665] systemd[1]: network.target: Job network.target/start deleted to break ordering cycle starting with network...

[Norbert (nrbrtx)]

I get this issue on my system updated from 12.04 to 16.04.

Here is syslog:
May 1 23:50:11 xxx systemd[1]: sysinit.target: Found dependency on NetworkManager.service/verify-active
May 1 23:50:11 xxx systemd[1]: sysinit.target: Found dependency on NetworkManager.service/verify-active
May 1 23:50:11 xxx systemd[1]: network-online.target: Found dependency on NetworkManager.service/verify-active
May 1 23:50:11 xxx systemd[1]: network-online.target: Found dependency on NetworkManager.service/verify-active
May 1 23:50:11 xxx systemd[1]: network-online.target: Breaking ordering cycle by deleting job NetworkManager.service/verify-active
May 1 23:50:11 xxx systemd[1]: NetworkManager.service: Job NetworkManager.service/verify-active deleted to break ordering cycle starting with network-online.target/start
May 1 23:50:11 xxx systemd[1]: sysinit.target: Found dependency on NetworkManager.service/verify-active
May 1 23:50:11 xxx systemd[1]: network.target: Found dependency on NetworkManager.service/verify-active
May 1 23:50:11 xxx systemd[1]: network.target: Breaking ordering cycle by deleting job NetworkManager.service/verify-active
May 1 23:50:11 xxx systemd[1]: NetworkManager.service: Job NetworkManager.service/verify-active deleted to break ordering cycle starting with network.target/start

I did not touch any service files. debsums reports no changes.

How to fix this bug?

[Norbert (nrbrtx)]

I removed other "cycle breaker" - "Breaking ordering cycle by deleting job decnet.service/start"
with "sudo apt-get purge dnet-common".
Hope it will help.

[Norbert (nrbrtx)]

Got this bug again on other xenial system.

Here is a log:
$ cat /var/log/syslog | grep break
Aug 23 13:55:13 hostname systemd[1]: nbd-client.service: Job nbd-client.service/start deleted to break ordering cycle starting with sysinit.target/start
Aug 24 01:17:38 hostname systemd[1]: cups.socket: Job cups.socket/start deleted to break ordering cycle starting with sockets.target/start
Aug 24 01:17:38 hostname systemd[1]: nbd-client.service: Job nbd-client.service/start deleted to break ordering cycle starting with sockets.target/start
Aug 24 01:17:38 hostname systemd[1]: network-online.target: Job network-online.target/start deleted to break ordering cycle starting with sockets.target/start
Aug 24 01:17:38 hostname systemd[1]: network.target: Job network.target/start deleted to break ordering cycle starting with sockets.target/start
Aug 24 01:17:38 hostname systemd[1]: NetworkManager.service: Job NetworkManager.service/start deleted to break ordering cycle starting with sockets.target/start

This results offline system. I need manually start networkmanager service.

[Norbert (nrbrtx)]

This bug should have CRITICAL Importance as other bug 1594715.

[Norbert (nrbrtx)]

Today I got:

Oct 19 15:25:56 server systemd[1]: network-online.target: Found ordering cycle on network-online.target/stop
Oct 19 15:25:56 server systemd[1]: network-online.target: Found dependency on network.target/stop
Oct 19 15:25:56 server systemd[1]: network-online.target: Found dependency on NetworkManager.service/stop
Oct 19 15:25:56 server systemd[1]: network-online.target: Found dependency on basic.target/stop
Oct 19 15:25:56 server systemd[1]: network-online.target: Found dependency on paths.target/stop
Oct 19 15:25:56 server systemd[1]: network-online.target: Found dependency on cups.path/stop
Oct 19 15:25:56 server systemd[1]: network-online.target: Found dependency on sysinit.target/stop
Oct 19 15:25:56 server systemd[1]: network-online.target: Found dependency on nbd-client.service/stop
Oct 19 15:25:56 server systemd[1]: network-online.target: Found dependency on network-online.target/stop
Oct 19 15:25:56 server systemd[1]: network-online.target: Breaking ordering cycle by deleting job network.target/stop
Oct 19 15:25:56 server systemd[1]: network.target: Job network.target/stop deleted to break ordering cycle starting with network-online.target/stop

Are you kidding? Server OS boots offline and you think it's normal?

[Norbert (nrbrtx)]

Got this bug again other Ubuntu 16.04.3 LTS system:

$ journalctl | grep -i network
ноя 06 22:49:57 norbert-vaio kernel: FUJITSU Extended Socket Network Device Driver - version 1.0 - Copyright (c) 2015 FUJITSU LIMITED
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Found ordering cycle on network.target/start
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Found dependency on NetworkManager.service/start
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Found dependency on basic.target/start
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Found dependency on sockets.target/start
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Found dependency on uuidd.socket/start
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Found dependency on sysinit.target/start
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Found dependency on console-cyrillic.service/start
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Found dependency on remote-fs.target/start
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Found dependency on remote-fs-pre.target/start
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Found dependency on nfs-server.service/start
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Found dependency on network.target/start
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Breaking ordering cycle by deleting job NetworkManager.service/start
ноя 06 22:49:57 norbert-vaio systemd[1]: NetworkManager.service: Job NetworkManager.service/start deleted to break ordering cycle starting with network.target/start

$ journalctl | grep -i break
ноя 06 22:49:57 norbert-vaio systemd[1]: network.target: Breaking ordering cycle by deleting job NetworkManager.service/start
ноя 06 22:49:57 norbert-vaio systemd[1]: NetworkManager.service: Job NetworkManager.service/start deleted to break ordering cycle starting with network.target/start
ноя 06 22:49:57 norbert-vaio systemd[1]: nfs-server.service: Breaking ordering cycle by deleting job rpcbind.socket/start
ноя 06 22:49:57 norbert-vaio systemd[1]: rpcbind.socket: Job rpcbind.socket/start deleted to break ordering cycle starting with nfs-server.service/start

I'm trying to remove nfs-kernel-server. But it is still critical bug.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.