"Waiting for network configuration" on every boot

[Jani Uusitalo]

Upgrade from 0.7~alpha5.1ubuntu6 to 0.7~beta2ubuntu1 brought this on: on every boot the "Waiting for network configuration" and "Waiting up to 60 seconds more for network" messages appear and boot is thus about 2 minutes slower than before. It also says it's booting without network, but the network is in fact there after I log in.

Downgrading back to 0.7~alpha5.1ubuntu6 makes the wait go away again.

Reproduced this on three different computers on two different networks, both wired and wireless.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: ifupdown 0.7~beta2ubuntu1
ProcVersionSignature: Ubuntu 3.2.0-8.15-generic 3.2.0
Uname: Linux 3.2.0-8-generic i686
ApportVersion: 1.90-0ubuntu2
Architecture: i386
Date: Sun Jan 15 21:10:34 2012
EcryptfsInUse: Yes
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release i386 (20111012)
SourcePackage: ifupdown
UpgradeStatus: Upgraded to precise on 2011-12-09 (37 days ago)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Stéphane Graber (stgraber)]

Can you please attach the following information from a boot with the beta2 version?
 - /etc/network/interfaces
 - /var/log/boot.log
 - output of 'cat /proc/mounts'
 - output of 'ls -l /etc/network/'
 - output of 'ls -l /var/'
 - output of 'ls -l /run'

Thanks

[Jani Uusitalo (uusijani)]

Attaching /etc/network/interfaces.

[Jani Uusitalo (uusijani)]

Attaching /var/log/boot.log.

[Jani Uusitalo (uusijani)]

jani@amilo:~$ cat /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=1015948k,nr_inodes=213349,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,relatime,size=410548k,mode=755 0 0
/dev/disk/by-uuid/f5b168d7-b4d0-40d8-b76a-8bda4fb90be5 / ext4 rw,relatime,errors=remount-ro,user_xattr,acl,barrier=1,data=ordered 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
/home/jani/.Private /home/jani/Yksityinen ecryptfs rw,relatime,ecryptfs_fnek_sig=6c6125b6685cf891,ecryptfs_sig=0540f7be64aaba29,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs 0 0
gvfs-fuse-daemon /home/jani/.gvfs fuse.gvfs-fuse-daemon rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0

[Jani Uusitalo (uusijani)]

jani@amilo:~$ ls -l /etc/network/
yhteensä 24
drwxr-xr-x 2 root root 4096 tammi 13 13:26 if-down.d
drwxr-xr-x 2 root root 4096 joulu 23 15:17 if-post-down.d
drwxr-xr-x 2 root root 4096 joulu 30 16:18 if-pre-up.d
drwxr-xr-x 2 root root 4096 tammi 13 13:26 if-up.d
-rw-r--r-- 1 root root 32 marra 19 11:16 interfaces
-rw-r--r-- 1 root root 64 marra 19 11:03 interfaces~
lrwxrwxrwx 1 root root 12 marra 3 20:03 run -> /run/network

[Jani Uusitalo (uusijani)]

jani@amilo:~$ ls -l /var/
yhteensä 44
drwxr-xr-x 2 root root 4096 tammi 15 07:35 backups
drwxr-xr-x 17 root root 4096 marra 26 11:59 cache
drwxrwxrwt 2 root root 4096 joulu 23 15:05 crash
drwxr-xr-x 3 root root 4096 marra 20 12:13 games
drwxr-xr-x 67 root root 4096 tammi 2 22:17 lib
drwxrwsr-x 2 root staff 4096 loka 9 10:29 local
lrwxrwxrwx 1 root root 9 tammi 15 21:04 lock -> /run/lock
drwxr-xr-x 16 root root 4096 tammi 15 21:07 log
drwxrwsr-x 2 root mail 4096 loka 12 17:27 mail
drwxr-xr-x 2 root root 4096 loka 12 17:27 opt
lrwxrwxrwx 1 root root 4 tammi 15 21:04 run -> /run
drwxr-xr-x 7 root root 4096 marra 20 16:33 spool
drwxrwxrwt 2 root root 4096 tammi 15 21:10 tmp

[Jani Uusitalo (uusijani)]

jani@amilo:~$ ls -l /run/
yhteensä 64
-rw-r--r-- 1 root root 5 tammi 15 21:07 acpid.pid
srw-rw-rw- 1 root root 0 tammi 15 21:07 acpid.socket
-rw-r--r-- 1 root root 5 tammi 15 21:07 atd.pid
drwxr-xr-x 2 avahi avahi 80 tammi 15 21:05 avahi-daemon
drwxr-xr-x 2 root root 60 tammi 15 21:07 console
drwxr-xr-x 2 root root 60 tammi 15 21:07 ConsoleKit
-rw-r--r-- 1 root root 5 tammi 15 21:07 console-kit-daemon.pid
-rw-r--r-- 1 root root 5 tammi 15 21:07 crond.pid
---------- 1 root root 0 tammi 15 21:07 crond.reboot
drwxr-xr-x 3 root lp 120 tammi 15 21:05 cups
drwxr-xr-x 2 messagebus messagebus 80 tammi 15 21:05 dbus
-rw-r--r-- 1 root root 4 tammi 15 22:01 dhclient-eth1.pid
drwxr-xr-x 2 root root 40 tammi 15 21:07 initramfs
drwx--x--x 3 root root 60 tammi 15 21:07 lightdm
-rw-r--r-- 1 root root 5 tammi 15 21:07 lightdm.pid
drwxrwxrwt 3 root root 60 tammi 15 21:08 lock
-rw-r--r-- 1 root root 126 tammi 15 21:05 motd
drwxr-xr-x 2 root root 60 tammi 15 21:05 mount
drwxr-xr-x 2 root root 80 tammi 15 21:05 network
-rw-r--r-- 1 root root 0 tammi 15 21:05 network-interface-security
-rw-r--r-- 1 root root 3 tammi 15 21:05 NetworkManager.pid
-rw-r--r-- 1 root root 2189 tammi 15 21:05 nm-dhclient-eth1.conf
-rw------- 1 root root 40 tammi 15 21:05 nm-dns-dnsmasq.conf
-rw-r--r-- 1 root root 4 tammi 15 21:05 nm-dns-dnsmasq.pid
drwxr-xr-x 4 root root 80 tammi 15 21:07 pm-utils
drwxr-xr-x 2 root root 40 tammi 15 21:07 pppconfig
-rw-r--r-- 1 root root 4 tammi 15 21:05 rsyslogd.pid
drwxr-xr-x 3 root root 160 tammi 15 21:07 samba
drwxrwxr-x 2 root utmp 40 tammi 15 21:05 screen
srw-rw-rw- 1 root root 0 tammi 15 21:07 sdp
drwxr-xr-x 2 root root 40 tammi 15 21:05 sendsigs.omit.d
drwxrwxrwt 2 root root 140 tammi 15 21:08 shm
drwxr-xr-x 2 root root 40 tammi 15 21:05 sshd
-rw-r--r-- 1 root root 4 tammi 15 21:05 sshd.pid
drwxr-xr-x 7 root root 180 tammi 15 21:08 udev
drwxr-xr-x 2 root root 60 tammi 15 21:05 udev-configure-printer
drwx------ 2 root root 40 tammi 15 21:08 udisks
-rw-r--r-- 1 root root 4 tammi 15 21:05 upstart-socket-bridge.pid
-rw-r--r-- 1 root root 4 tammi 15 21:05 upstart-udev-bridge.pid
-rw-rw-r-- 1 root utmp 3840 tammi 15 21:08 utmp
drwxr-x--- 2 root root 60 tammi 15 21:05 wpa_supplicant

[Stéphane Graber (stgraber)]

Thanks for the above, I didn't find anything obviously wrong in these.

Can you also attach:
 - ls -l "/run/"
 - "ifconfig -a"
 - "dpkg -l"
 - "initctl status"

One case where the above could be possible is if you only have a loopback interface on your system and no other interfaces or NetworkManager doesn't emit net-device-up IFACE=ethX for your other interfaces.

Hopefully the output from these few commands should help figure out exactly what's happening or at least help me reproduce the issue on a system I have around here.

[Stéphane Graber (stgraber)]

Sorry, I didn't see your "ls -l /run" before posting my comment, no need to paste it again :)
The output also points towards NetworkManager running on this machine, giving me some idea of what might be broken (though pretty weird that didn't show up in the tests ...)

[Stéphane Graber (stgraber)]

Just thought of a bunch more that might be useful:
 - /var/log/syslog
 - /var/log/kern.log

And if you don't mind a reboot and know of to do that:
 - reboot with "--log" appended to the kernel boot command line
 - make a tarball out of /var/log/upstart/* and attach it to the bug

Please note that these upstart logging instructions may cause a kernel panic at boot time because of an upstart bug (that's why it's not the current default). If it does, just reboot without the extra "--log". We expect the next upstart upload to fix that bug and then turn logging on by default (which should make debugging this kind of bug much much easier).

I had planned to do a review of ifupdown bugs tomorrow morning and try to fix or reproduce as many as possible so I probably won't spend much more time on this bug today but hopefully with these few files I should have enough to figure out what happened.

Thanks

[Jani Uusitalo (uusijani)]

Thanks for taking a look Stéphane. Here's /var/log/syslog, the rest will follow below. And do ask if you need more, it's no problem for me to reboot or otherwise test things on this laptop, as I also have my main desktop to work with.

[Jani Uusitalo (uusijani)]

jani@amilo:~$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:0a:e4:2b:ba:12
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
          Interrupt:10

eth1 Link encap:Ethernet HWaddr 00:04:23:75:9e:35
          inet addr:88.115.166.144 Bcast:88.115.175.255 Mask:255.255.240.0
          inet6 addr: fe80::204:23ff:fe75:9e35/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3013 errors:2 dropped:2 overruns:0 frame:0
          TX packets:2344 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2388786 (2.3 MB) TX bytes:991777 (991.7 KB)
          Interrupt:10 Base address:0xe000 Memory:e0203000-e0203fff

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:90 errors:0 dropped:0 overruns:0 frame:0
          TX packets:90 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:7504 (7.5 KB) TX bytes:7504 (7.5 KB)

[Jani Uusitalo (uusijani)]

The initctl command apparently needs some additional parameter:

jani@amilo:~$ initctl status
initctl: missing job name
Try `initctl --help' for more information.
jani@amilo:~$ sudo initctl status
initctl: missing job name
Try `initctl --help' for more information.

I'll attach the list of jobs it currently knows about.

[Stéphane Graber (stgraber)]

My bad, I indeed meant "initctl list".

Thanks for all the information, I'll take a long at it tomorrow morning.

[Stéphane Graber (stgraber)]

In your log files, I see:
run-parts: /etc/network/if-up.d/announce-ip exited with return code 1
Failed to bring up lo.

Can you attach that script?

[Jani Uusitalo (uusijani)]

Sure. This script sources another local file for configuration, but that one's just a one-line variable declaration for the $URL. (I'm omitting that one because it has my password for the script's server side.)

[Stéphane Graber (stgraber)]

Thanks, I think I should have enough to reproduce the issue here now (finishing to upgrade my test VM).

If you have a minute, can you try booting your system without your announce-ip script?
Having it "exit 1" seems to disturb ifupdown trying to bring up your loopback device, I'm not sure if something changed in ifupdown's handling of a script failure, but it's probably worth trying without the script and see if that solves it.

[Jani Uusitalo (uusijani)]

You're right: it's announce-ip and another local script from /etc/network/if-up.d/ (I'm attaching it here) that both seem to trigger this if either is present. This other one's sort of like the opposite of my announce-ip, there to update /etc/hosts with data from hosts elsewhere on the net my server knows about (that announce themselves with announce-ip). This combination used to work back with ifupdown alpha but now with beta causes the wait.

To clarify: with both scripts removed from if-up.d the 60+ second wait doesn't appear during boot, but with either script in place the wait is there.

To be sure, I installed the latest ifupdown that's just appeared in the repos (0.7~beta2ubuntu2) also, and there's no change compared to 0.7~beta2ubuntu1 (wait is there when the scripts are there).

The question now becomes whether I've initially designed the scripts wrong with regards to how things in if-up.d should work, and it just happened to work with the alpha, or did something in ifupdown's handling of if-up.d change between alpha and beta so that things that should work no longer do.

With my knowledge of things I'd bet on the former, but you can probably enlighten me on this. My assumption's been that the scripts in if-up.d are run after network interfaces have been brought up.

[Stéphane Graber (stgraber)]

I think it makes sense for ifupdown to stop processing an interface if one of the scripts returns a non-zero return code.

I'll go check in the code to see exactly what's the intended behaviour and confirm that it indeed was an upstream "bugfix" between alpha1 and beta2 that's causing that change for you.

[Stéphane Graber (stgraber)]

Ok, I just confirmed it by looking at the changelog actually:
  * Stop on run-parts failure (Closes: #547587).

So if any of the scripts fails, ifupdown will stop processing the interface.

I'd suggest you change your scripts to either not fire if it's not the right interface or simply change these exit 1 to an exit 0 + logging so it doesn't block the interface from being initialised but you'll still now it failed.

I'm marking this bug as Invalid as it was actually an upstream bugfix causing that change in behaviour.

Thanks for all your help in tracking this one down.

[Jani Uusitalo (uusijani)]

No problem, thank you for your rapid response to this and for the learning I gained! I've modified the scripts accordingly and it all works smoothly (and as intended) now.

[Pritam Baral (pritambaral)]

I have been facing a similar problem, though with the scripts from another package "firestarter". But, until I came across this page, I had no idea why I was facing this problem. I figure there might be other scripts too that exit with a non-zero code. Couldn't the logging part be implemented in ifupdown itself, and it be allowed to move on, without having to stop. In my situation, I thought I was stuck, and when I rebooted into recovery, I had no internet. Only after manually enabling, setting up my Ethernet could I resume boot. But it might not be as simple to other users.

[Stéphane Graber (stgraber)]

Well, the fact that ifupdown would ignore the return code in the past was a bug of the 0.7 branch (possibly some 0.6 versions too), so no I doubt upstream will re-introduce the bug to "fix" some scripts.

The logging will be enable by the time we release 12.04 so I don't see any need to spend days implementing something similar in ifupdown.

[Benjamin Kerensa (bkerensa)]

I'm having the same network config wait issue :P its adding 60-70 seconds to my boot time :P

[Lito (lito-eordes)]

I have the same problem, but the waiting message doesn't appear when I have the ethernet wire pluged.

With the wire the computer boots fine, but without the wire (having enabled or disabled the wireless) I need to wait the first and second message.

[emptythevoid (emptythevoid)]

Just upgraded my netbook from 11.10 to 12.04 beta 1 (32 bit) and I'm seeing the "waiting for network configuration" appearing at boot whether an ethernet LAN cable is attached or not. It doesn't take 60 seconds, but it is an annoying delay. After boot, networking behaves normally.

[Mitch Claborn (mitch-news)]

This still happens to me sporaidcally in 12.04. Should this be fixed?

[Lilleman (lilleman)]

Still got this issue in 12.04 fully patched here to.

[Lilleman (lilleman)]

After quite some digging (I'm not very good at it ;). I found /var/log/syslog and found out that my DHCP-client was hanging around waiting for answer on my wired eth0, even though no cable is connected.

Maybe its my computer that is broken, but I disabled "iface eth0 inet dhcp" and "iface eth1 inet dhcp" in /etc/networking/interfaces, and now it boots fast without the "waiting for network". And once inside, all interfaces works well.

Hope this helps someone, and it would be great if someone more skilled looked into this. I'd gladly give any help I can. :)

[bancaldo (bancaldo-gmail)]

A similar problem for me with Ubuntu 12.04.

If I have the default situation in /etc/network/interfaces:

auto lo
iface lo inet loopback

there are no problems.

If I enable wlan0 for wireless connection (i.e.):

auto wlan0
iface wlan0 inet static
....

the wi-fi connection has no problem, but if I reboot I Have the 'Waiting for...' message.
After 2 minutes and with wired cable unplugged I can login but I have to "clean" /etc/network/interfaces
to the default situation.
Even if I connect the wired cable, I have no connection.
The network manager icon has disappeared and I get an app crash error message.
I have to reboot again and then everything works fine, no 'waiting' message, connection ready and network manager
working...

bye
ps. Sorry for my poor english...

[Daniil Ivanov (daniil-ivanov)]

This bug seems to be a duplicate of bug 940485.

[Hassaan (kh-iftikhar)]

I had the same problem on

Linux localhost 3.2.0-36-generic #57-Ubuntu SMP Tue Jan 8 21:44:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Distributor ID: Ubuntu
Description: Ubuntu 12.04.1 LTS
Release: 12.04
Codename: precise

Lenovo ThinkPad T430s
and by removing these lines from /etc/networ/interfaces, it's all fixed. It boots pretty fast and wireless and ethernet works fine.

#remove these lines
auto eth0
iface eth0 inet dhcp

[jbowen7 (jbowen7)]

# Removing the lines
auto eth0
iface eth0 inet dhcp

does work, however the bug remains. An example of the problem would be using ifup and ifdown to configure a bridge.

auto eth0
iface eth0 inet manual

auto br0
    bridge_ports eth0
    foo
    foo
    bar

[meiradarocha (joseantoniorocha)]

I have a firewire netowork cable whit fixed IP in /etc/network/interfaces, and have the same problem.
I do a workaround with this tip (comment the sleeps lines):

http://askubuntu.com/questions/185515/disable-network-configuration-services-during-boot-time

A better ideea than disabling the service is to remove the sleep time. To do this, open /etc/init/failsafe.conf with your favourite editor, and around line 25 you should see the following lines of code:

# Plymouth errors should not stop the script because we *must* reach
# the end of this script to avoid letting the system spin forever
# waiting on it to start.
$PLYMOUTH message –text=”Waiting for network configuration…” || :
sleep 40
$PLYMOUTH message –text=”Waiting up to 60 more seconds for network configuration…” || :
sleep 59
$PLYMOUTH message –text=”Booting system without full network configuration…” || :

To solve your problem, just comment the sleep times (add '#' in front of the text). It should look like this:

# Plymouth errors should not stop the script because we *must* reach
# the end of this script to avoid letting the system spin forever
# waiting on it to start.
$PLYMOUTH message –text=”Waiting for network configuration…” || :
#sleep 40
$PLYMOUTH message –text=”Waiting up to 60 more seconds for network configuration…” || :
#sleep 59
$PLYMOUTH message –text=”Booting system without full network configuration…” || :

[carloslp (carloslp)]

Just for the record.

I had the same problem and I fixed it by changing in /etc/network/interfaces the line

auto etho

to

allow-hotplug eth0

[Markus Klimt (markus-klimt)]

Confirmed on Xubuntu 14.04 with Kernel 3.13.0-24-generic, install done via minimal CD and Xubuntu Desktop installed afterwards.

The "allow-hotplug" change suggested by Carloslp FIXED the problem.

I installed the OS via ethernet and wanted to use wifi afterwards (That might be relevant).