wpasupplicant doesn't start when the network start

[Hassan El Jacifi]

Binary package hint: wpasupplicant

I am using an ipw2200 wireless card with wpasupplicant. My wireless network uses WPA-PSK with AES and a hiden ssid. This combination worked great using a previous version of Dapper.

Actually the wpa_supplicant doesn't start with a network. I will do a restart for my network manually.

14:09 root@Mars /home/waver# pkill wpa
zsh: 5081 exit 1 pkill wpa
14:09 root@Mars /home/waver# /etc/init.d/networking stop && /etc/init.d/networki
ng start
 * Deconfiguring network interfaces... [ ok ]
 * Configuring network interfaces... [ ok ]
14:09 root@Mars /home/waver# ps faux | grep wpa
root 5377 0.0 0.0 3532 880 ? Ss 14:11 0:00 /sbin/wpa_supplicant -B -P /var/run/wpa_supplicant.eth1.pid -i eth1 -c /etc/wpa_supplicant/wpa_supplicant.conf -D wext

My /etc/network/Interface:

auto lo
iface lo inet loopback

auto eth1
iface eth1 inet static
        address 192.168.1.3
        netmask 255.255.255.0
        network 192.168.1.0
        broadcast 192.168.1.255
        gateway 192.168.1.1
        dns-nameservers 192.168.1.1
        wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

My /etc/wpa_supplicant/wpa_supplicant.conf :

ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0

eapol_version=1
ap_scan=1
fast_reauth=1

network={
        ssid="Myssid"
        bssid=00:77:77:77:8d:05
        scan_ssid=1
        auth_alg=OPEN
        key_mgmt=WPA-PSK
        pairwise=CCMP TKIP
        group=CCMP TKIP
        psk=6ac6a1b4f104c1f964f36406c84245c758fc672972db00a3c6f608f674e34b0a
        priority=2
}

Sometimes i have in my debug.log:

May 11 13:34:50 localhost kernel: [4294686.126000] ieee80211_crypt: registered algorithm 'CCMP'
May 11 13:37:05 localhost kernel: [4294827.842000] CCMP: decrypt failed: STA=00:77:77:77:8d:05

OR

May 11 14:08:07 localhost kernel: [4294686.288000] ieee80211_crypt: registered algorithm 'NULL'
May 11 14:08:07 localhost kernel: [4294688.558000] ieee80211_crypt: registered algorithm 'CCMP'

My kernel: 2.6.15-22-686
Version wpasupplicant 0.4.8-3ubuntu1

[Andre Bouillet (andre-bouillet)]

Same problem here. My /etc/network/interfaces looks like this:

iface lo inet loopback

iface ath0 inet static
# mtu 1492
address 192.168.1.29
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1
wpa-driver madwifi
wpa-ssid dd-wrt
wpa-bssid 00:14:BF:C9:2B:70
wpa-key-mgmt WPA-PSK
wpa-psk (hidden)
auto lo

auto ath0

The network is not initially starting. Just after "sudo /etc/init.d/networking restart" it works flawlessly.

My kernel: 2.6.15-23-amd64-generic
wpasupplicant: 0.4.8-3ubuntu1

[Doug Winter (doug-isotoma)]

I have the same problem, which I believe is also the one reported here:

http://ubuntuforums.org/showthread.php?p=1307027

Moving the wpa config out into a separate configuration file, as detailed there, resolved the problem for me.

[n8han (nathan-technically)]

adding this line to my ath0 block made it work on boot:
        pre-up sleep 10

(everything is still in /etc/network/interfaces)

[Christian Haeberli (christian-haeberli)]

I see the same problem and both workarounds work for me.

But there is another (related?) problem remaining: once connected (either without workaround and having networking restarted or with one of the workarounds) it is not possible to restart networking again. The connection cannot be reestablished.

[Reinhard Tartler (siretart)]

since wpasupplicant starts fine when called manually, I hardly think this is a bug with the package itself but some obscure race condition with the interaction with ifupdown. I'm therfore reassigning this bug.

[Alessio Gaeta (meden)]

I have the same problem too: wpasupplicant doesn't start automatically at boot and to get a working interface I have to:

$ ifdown eth1 [<--]
$ ifup eth1

So it seems that the network is actually started and configured, only wpasupplicant refuses to start.

I have only one suspect: maybe there is some odd interaction with udev rule ifupdown.rules? Udev is started before than Networking in the init process and (as far as I understood) tries to configure the interfaces it found. So, when Networking starts, interfaces are already brought up (and in fact I can't do ifup directly to make eth1 working).
Am I saying nonsense?

[Reinhard Tartler (siretart)]

Alessio Gaeta <email address hidden> writes:
> maybe there is some odd interaction with udev rule ifupdown.rules?
> Udev is started before than Networking in the init process and (as far
> as I understood) tries to configure the interfaces it found. So, when
> Networking starts, interfaces are already brought up (and in fact I
> can't do ifup directly to make eth1 working). Am I saying nonsense?

I think that is very likly. In debian, there is no udev rule at all, and
things work as intended. In udev, ifupdown is called via udev. It seems
to me that for some reason wpasupplicant either

a) fails to start
b) gets killed by udev

--
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4

[Alessio Gaeta (meden)]

I can confirm the udev "critical race": removing /etc/udev/rules.d/85-ifupdown.rules and setting the wireless interface as auto everything works as intended (interface correctly brought up at boot time by /etc/init.d/networking).
I can't explane why udev cannot work correctly even if it uses ifup (as networking init script does); again a suspect: are the 802.11 stack related modules loaded properly before udev starts? I have a modprobe rule (maybe redundant) that loads ieee80211, ieee80211_crypt, ieee80211_crypt_tkip modules together with the ipw2200 one; does udev do the same thing? Or it uses insmod, bypassing modprobe rules?
And, anyway, where is the usefulness of a udev rule that does the same of netbase scripts? I think is better mantaining things as simple as possible... (Debian does it... :) )

[Reinhard Tartler (siretart)]

#45842 seems to be related to this.

[Roman Yepishev (rye)]

Dear all,

Today I managed to make it run on boot while maintaining udev support (i.e. the interface goes down with the card removal and is brought up when my PCMCIA card is inserted).

First of all (for those who come from earlier releases), looks like udev is confused a bit with DRIVERS=="?*" test (/etc/udev/rules.d/85-ifupdown.rules). This does not match on device removal so the ifupdown state of the interface
becomes out of sync with the real world. By deleting this condition the ACTION=="remove" starts being called. This is fixed in the last ifupdown source.

The next thing is that we can not start the network before the /etc/init.d/networking is run. It may be anything, missing modules, keys, passwords, users, so we should modify the rules to check that the network has already been initialized, possibly by flag in /var/run/network.

The file is touched when `networking start` is issued, and the file is removed on `networking stop`. The restart part probably will have only create file part.

Then we add the file TEST condition to 85-ifupdown.rules: SUBSYSTEM=="net", TEST=="/var/run/network/initialized", GOTO="net_start" (instead of "SUBSYSTEM!="net", GOTO="net_end"" or SUBSYSTEM=="net", DRIVERS="?*", GOTO="net_start"")

This will match when file is present so we make sure we run after /etc/init.d/networking.

I have attached the patch for a live system, /etc/init.d/networking and /etc/udev/rules.d/85-ifupdown.rules. It may not apply cleanly if default 85-ifupdown.rules is not the latest one. In this case, see the passage about TEST condition.

[gelin (dnelubin)]

To: Roman Yepishev

Thanks a lot. Your patch works.

[Chef vom Dienst (herr-ernst)]

Confirmed. Bug also exists with me, patch works. Patch should be included in Ubuntu 8.04.

[Gabriele Monti (psicus78)]

Some questions and some considerations.

I was wondering what happens if 85-ifupdown.rules is removed. I believe /etc/init.d/networking would still do its job and networking would work fine.
If a card is removed (or added) udev wouldn't care (dinamically) about the removal (or addiction), am I wrong? But, if I understood correctly how udev work, a reboot would allow udev to get the change in the interfaces. So if this is the case I believe it should not be a problem to remove 85-ifupdown.rules, I think the number of users that remove or add cards is lower than the number of those who need a static IP for wireless and however a reboot should make everything working.

However, NM 0.7 should solve the problem by allowing system wide settings.

[Gareth Fitzworthington (mapping-gp-deactivatedaccount)]

This is a Network Manager bug.
Fix released for Hardy.

Marking as duplicate.
Any backport requests should be first examined against:
https://help.ubuntu.com/community/UbuntuBackports
Then filed against the main bug stream. Unless this duplicate message is viewed as errant. In which case please un-duplicate.

[Gareth Fitzworthington (mapping-gp-deactivatedaccount)]

Sorry. holding off on duplicate. Anyone trying to use Hidden AP then see Bug #50214
Also, anything other than "Roaming Mode" pre-Hardy see this comment:
https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/110187/comments/7
Thanks.

[Saivann Carignan (oxmosys)]

Setting back to confirmed.

Gareth Fitzworthington : This bug exist in latest Hardy beta with all updates. This is not a duplicate. This bug happens when the wireless network is configured manually (ex. with a static IP) with WPA/WPA2. Hardy won't connect to the network at boot, but it will always work when you do /etc/init.d/networking restart. This issue persist and is still confirmed in Hardy.

[Heinrich Kruger (heindsight-deactivatedaccount)]

Gareth Fitzworthington wrote on 2008-04-05: (permalink)
>This is a Network Manager bug.
>Fix released for Hardy.

This is definitely not a Network Manager bug. I (and others who have commented on this bug and its duplicates) do not use Network Manager at all. It is an ifupdown bug that occurs because the ifupdown udev rule tries to bring up interfaces listed as auto in /etc/network/interfaces (and thus ignored by Network Manager) too early at boot.

[Gareth Fitzworthington (mapping-gp-deactivatedaccount)]

Yes, apologies guys. This bug got mixed up in a list of other bugs I was working on. Launchpad appears to have set it to "invalid". Not sure why. I didn't direct it to do that. I'll be more careful in future.
Again, sorry for causing a bother.

[adair (al-lairds)]

I would like to confirm this bug. I have only started to have problems with automatic wifi connection since running the Hardy beta. Doing sudo /etc/init.d/networking restart seems the most convenient way of overcoming it until there is a proper fix.

[jerico (jerico-dev)]

https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.15/+bug/40763 is a (partial) duplicate and confirms the problem for a current Hardy Heron install.

[Reinhard Tartler (siretart)]

captain haddock <email address hidden> writes:

> https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.15/+bug/40763
> is a (partial) duplicate and confirms the problem for a current Hardy
> Heron install.

It does not. bug #40763 is about an issue in ipw2200. Later in the
bugtrail some user seems to suffer from this bug, but that does not make
the two bugs the same.

--
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4

[Reinhard Tartler (siretart)]

https://bugs.edge.launchpad.net/ubuntu/+source/wpasupplicant/+bug/44194/comments/10 has the right idea.

[Reinhard Tartler (siretart)]

https://bugs.edge.launchpad.net/ubuntu/+source/wpasupplicant/+bug/44194/comments/10 has the righjt idea. I'm going to work on integrating this patch

[Launchpad Janitor (janitor)]

This bug was fixed in the package ifupdown - 0.6.8ubuntu9

---------------
ifupdown (0.6.8ubuntu9) intrepid; urgency=low

  * debian/ifupdown.rules: (the udev rule) ensure that the init script
    /etc/network/networking has been started to ensure that the udev
    script does not start too early in the boot process. This way we
    ensure that the root file system is already writable and ifupdown has
    a chance to actually work. LP: #44194
  * Add versioned dependency on a version of netbase that actually does
    indicate that it has been configured.

 -- Reinhard Tartler <email address hidden> Sat, 21 Jun 2008 23:32:56 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package netbase - 4.30ubuntu2

---------------
netbase (4.30ubuntu2) intrepid; urgency=low

  * introduce flag file for indicating the network has been
    configured. This is needed so that ifupdown udev rules does not 'ifup'
    the interface too early. Related to LP: #44194

 -- Reinhard Tartler <email address hidden> Sat, 21 Jun 2008 23:22:40 +0200

[Reinhard Tartler (siretart)]

nothing to do in this bug for wpasupplicant

[Reinhard Tartler (siretart)]

okay, I've uploaded a very similar patch to what Roman has proposed to intrepid. I tested the packages in a kvm instance, obviously without wpasupplicant.

My current guess what's going wrong here:

The udev rules gets activated too early in the boot sequence, most probably with S10udev, where udev gets started. This is before any fscks have been done and also before the root filesystem has been remounted rw in S20checkroot.sh. This is not really the best time to start things like wpasupplicant.

Syncronizing the udev rule with S40networking is probably still not exactly the right solution. However I still think it is correct for this particular problem, because at that time, we are expect that the network is able to come up. In fact it should be safe to do already right after S20checkroot.sh. Waiting until S40networking isn't too bad as well, however.

[Roman Yepishev (rye)]

Confirming proper udev behavior after installing updated netbase & ifupdown over Ubuntu 8.04 system.
Thanks for the fix!

[Ole Jon Bjørkum (olejonbj)]

Bug confirmed with ipw2200. Installed Wicd (http://wicd.sourceforge.net) and the Ubuntu-version works like a charm. Wicd uninstalls network-manager, and do not use /etc/network/interfaces.

[valeshino (alan-khalis)]

Ditto Ole Jon. On a fresh install of 8.04 using ipw2200. Using pre-up sleep 10 before any wpa information in /etc/network/interfaces resolved issue, but what a sloppy workaround.

[Stephen Parry (sgparry)]

I can also confirm Reinhard's fix works on 8.04 if you manually copy in just the ifupdown.rules and networking files from the patch. Many thanks Reinhard!

Can we please have this officially backported as an update to 8.04?

[Mathias Gug (mathiaz)]

Reopening this bug in wpa_supplicant. It seems that the issue is with the ifupdown script from wpa_supplicant. If the script cannot be run early in the boot process it should fail gracefully and leave the interface unconfigured until the networking init script is run. Could the wpa_supplicant ifupdown script be run in debugging mode (set -x) to figure out why it fails?

I've also reverted the udev rules change to wait for the networking init script to have run. It breaks the boot process as we're trying to configure network interfaces as soon as they're discovered.

[Reinhard Tartler (siretart)]

Mathias Gug <email address hidden> writes:

> Reopening this bug in wpa_supplicant.

Reporters claim this bug to be fixed. What symptoms did recently arise?

> It seems that the issue is with the ifupdown script from
> wpa_supplicant.

What is the issue?

> If the script cannot be run early in the boot process it should fail
> gracefully and leave the interface unconfigured until the networking
> init script is run.

That might be or might not be feasible, depending on the reason why it
fails. The change I've introduced delays the configuration of networking
until the netbase init script is run. I fail to see what the problem is
with that approach.

> Could the wpa_supplicant ifupdown script be run in debugging mode (set
> -x) to figure out why it fails?

sure, just put set -x at bootup!

> I've also reverted the udev rules change to wait for the networking init
> script to have run. It breaks the boot process as we're trying to
> configure network interfaces as soon as they're discovered.

Instead of unilaterarily reverting the change I committed ages ago, it
would have been more appropriate to first discuss the symptoms in a bug
instead of just removing a fix that has been sufficiently tested. I
consider your revert as a violation of the CoC, points 2 and 4.

The more correct fix IMO would be to to change the udev rule to flag the
interfaces that need "delayed" starting via the init script and
"late-start" them in /etc/init.d/networking.

Still you haven't explained what exatly fails. Is it hot-plugging
cardbus cards? Or is it starting networking for NICs connected via PCI?

Please elaborate on this issue.

--
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4

[Stephen Parry (sgparry)]

Just to reiterate my request from earlier: once you've all decided what exactly this fix should be, can we please have it back-ported to Hardy? It is supposed to be the LTS release and Wifi is next to unusable without it.

[Mathias Gug (mathiaz)]

Hi Reinhard,

On Mon, Dec 01, 2008 at 10:49:52PM +0100, Reinhard Tartler wrote:
> Reporters claim this bug to be fixed. What symptoms did recently arise?
>

On the wpa_supplicant front, none. However it raises issue with iscsi
devices as outlined below.

> > It seems that the issue is with the ifupdown script from
> > wpa_supplicant.
>
> What is the issue?

I don't know. That's why I suggested to run the ifupdown script in
debugging mode in order to gather more information about the failure. I
don't have access to an environment to do such testing, thus the
suggestion I made in this bug.

> > If the script cannot be run early in the boot process it should fail
> > gracefully and leave the interface unconfigured until the networking
> > init script is run.
>
> That might be or might not be feasible, depending on the reason why it
> fails. The change I've introduced delays the configuration of networking
> until the netbase init script is run. I fail to see what the problem is
> with that approach.

Waiting until S40networking is run brings up an issue with iscsi
devices. The current way of handling iscsi devices is to treat them as
local devices by making sure that all iscsi devices are created before
local filesystems are mounted (S35mountall.sh and S30checkfs.sh). That
requires ifupdown scripts to be run as soon as network interfaces are
discovered instead of S40networking.

> > I've also reverted the udev rules change to wait for the networking init
> > script to have run. It breaks the boot process as we're trying to
> > configure network interfaces as soon as they're discovered.
>
> Instead of unilaterarily reverting the change I committed ages ago, it
> would have been more appropriate to first discuss the symptoms in a bug
> instead of just removing a fix that has been sufficiently tested. I
> consider your revert as a violation of the CoC, points 2 and 4.
>
> The more correct fix IMO would be to to change the udev rule to flag the
> interfaces that need "delayed" starting via the init script and
> "late-start" them in /etc/init.d/networking.
>

soren and Keybuk discussed this issue [1] (16:40 - 16:49). Their
proposal is to fix the ifupdown scripts so they work early in the
process. In the case of open-iscsi, the iscsi utilities had to be moved
from /usr/{bin/sbin} to /bin,/sbin.

If this is not possible ifupdown scripts should exit gracefully so that
the network interface is left unconfigured and will be configured later
by S40networking.

[1]: http://irclogs.ubuntu.com/2008/11/20/%23ubuntu-devel.txt

> Still you haven't explained what exatly fails. Is it hot-plugging
> cardbus cards? Or is it starting networking for NICs connected via PCI?
>
> Please elaborate on this issue.
>

I've discussed this issue with dustin and soren on irc [2] (16:16):

[2] http://irclogs.ubuntu.com/2008/11/20/%23ubuntu-server.txt

soren's point is that delaying network interface configuration to
S40networking is too late in the boot process and against what Ubuntu
has been trying to achieve with the boot process.

Although the udev rule changes fixes the issue encounter with
wpa_supplicant it breaks the iscsi use case (and other network devices
such ...

[Reinhard Tartler (siretart)]

Mathias Gug <email address hidden> writes:

> Waiting until S40networking is run brings up an issue with iscsi
> devices. The current way of handling iscsi devices is to treat them as
> local devices by making sure that all iscsi devices are created before
> local filesystems are mounted (S35mountall.sh and S30checkfs.sh). That
> requires ifupdown scripts to be run as soon as network interfaces are
> discovered instead of S40networking.

ah, I see. This means that you've decided that booting from iscsi
devices (feature A) is "more important" than booting with having wpa
statically configured (feature B) in /etc/network/interfaces.

This particular bug is about B. I don't agree that A warrants breaking
B. What you have done is to deliberatly break B in order to allow A to
work.

How about this: ifupdown needs to somehow know that a statically
configured wpa device cannot be upped early in the boot process, but
allow other devices to be configured. So the delayed starting of the
interface is made conditionally on that fact. The question is who
detects this. I see two possibilites

 - ifupdown
 - the wpasupplicant ifupdown hook.

I'd suggest looking in the 2nd first.

--
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4

[kelmo (kelrin)]

On Tuesday 02 December 2008 16:52:21 Reinhard Tartler wrote:
> Mathias Gug <email address hidden> writes:
>
> > Waiting until S40networking is run brings up an issue with iscsi
> > devices. The current way of handling iscsi devices is to treat them as
> > local devices by making sure that all iscsi devices are created before
> > local filesystems are mounted (S35mountall.sh and S30checkfs.sh). That
> > requires ifupdown scripts to be run as soon as network interfaces are
> > discovered instead of S40networking.
>
> ah, I see. This means that you've decided that booting from iscsi
> devices (feature A) is "more important" than booting with having wpa
> statically configured (feature B) in /etc/network/interfaces.
>
> This particular bug is about B. I don't agree that A warrants breaking
> B. What you have done is to deliberatly break B in order to allow A to
> work.
>
> How about this: ifupdown needs to somehow know that a statically
> configured wpa device cannot be upped early in the boot process, but
> allow other devices to be configured. So the delayed starting of the
> interface is made conditionally on that fact. The question is who
> detects this. I see two possibilites
>
> - ifupdown
> - the wpasupplicant ifupdown hook.
>
> I'd suggest looking in the 2nd first.

I disagree to an extent, the problem doesn't sound specific to wpa_supplicant's
shell glue at all, but rather, the inflexibility of the statically maintained
ifupdown software on an ever evolving Debian/Ubuntu software ecosystem.

A "cheap" fix could be included in an individual ifupdown hook script, but it
strikes me that the central network dispatcher/configurator framework should be
striving to handle situations such as that described, as future hooks may also
fall prey to this pitfall.

Thanks, Kel.

[Soren Hansen (soren)]

>> Waiting until S40networking is run brings up an issue with iscsi
>> devices. The current way of handling iscsi devices is to treat them
>> as local devices by making sure that all iscsi devices are created
>> before local filesystems are mounted (S35mountall.sh and
>> S30checkfs.sh). That requires ifupdown scripts to be run as soon as
>> network interfaces are discovered instead of S40networking.
> ah, I see. This means that you've decided that booting from iscsi
> devices (feature A) is "more important" than booting with having wpa
> statically configured (feature B) in /etc/network/interfaces.

Sheesh, calm down, dude. This is not an either/or situation. There's no
doubt that your "fix" breaks several completely reasonable use cases.
It's completely reasonable to attempt to configure network interfaces as
they're discovered by udev. wpasupplicant for some reason deals very
badly with this. Glancing at its pre-up script, I'm guessing your
problem is this:

# quit if executables are not installed
if [ ! -x "$WPA_SUP_BIN" ] || [ ! -x "$WPA_CLI_BIN" ]; then
        exit 0
fi

If wpasupplicant isn't around (/usr isn't mounted, for instance), you
exit 0. This tells ifupdown that everything is fine and that you don't
want to do anything else to configure this device, and ifup happily
marks the interface as configured.

If wpasupplicant can't do its thing before /var/run/network/initialized
exists, why don't you just check for that _in_ _wpasupplicant's_ _if-*_
scripts instead of ruining for everyone else by putting it in udev?

> This particular bug is about B. I don't agree that A warrants breaking
> B. What you have done is to deliberatly break B in order to allow A to
> work.

Dude... You're the one who broke it to begin with. Mathias fixed the
breakage you introduced and reopened this bug to let you know that this
problem is now back (because your solution wasn't acceptable). This
seems perfectly reasonable to me. In fact, I'm the one who suggested it
to begin with. Mathias was just nice enough to do it for me.

> - ifupdown
> - the wpasupplicant ifupdown hook.

> I'd suggest looking in the 2nd first.

Yes, someone with the appropriate testing environment (perhaps even
someone who's being bitten by this bug) or with enough interest in
wpasupplicant should look into it. Sure, it would be nice if Mathias
did this, but I really don't believe it's his responsibility.

Imagine this: If I had a problem with a module in the kernel, and I knew
it only happened with SMP kernels, it would fix my problem 100% if I
disabled SMP in our kernels. If someone on the kernel team found that to
be inappropriate and reverted that change thus breaking my module again,
it doesn't make it their job to fix my module. Sure, it'd be nice if
they did, but it's not their responsiblity. Putting that check in udev
is a very inappropriately big hammer to solve this problem.

[Soren Hansen (soren)]

> I disagree to an extent, the problem doesn't sound specific to
> wpa_supplicant's shell glue at all,

If wpasupplicant tells ifupdown that it's done configuring the
interface, and that everything went fine, but it that's not actually the
case... That's wrong. No matter how you look at it.

> but rather, the inflexibility of the statically maintained ifupdown
> software on an ever evolving Debian/Ubuntu software ecosystem.

What exactly do you mean by "statically maintained"?

> A "cheap" fix could be included in an individual ifupdown hook script,
> but it strikes me that the central network dispatcher/configurator
> framework should be striving to handle situations such as that
> described, as future hooks may also fall prey to this pitfall.

"situations such as that described"? You mean a particular application
having specific needs? It's not ifupdown's responsibility to know about
all these things. If wpasupplicant doesn't know when it can expect to be
succesful, how on earth should ifupdown know?

Leaving this decision to each of the modular network configuration
things is the right thing to do. I don't see any reason at all why you'd
want to put something like that into ifupdown.

[Reinhard Tartler (siretart)]

Soren Hansen <email address hidden> writes:

>> but rather, the inflexibility of the statically maintained ifupdown
>> software on an ever evolving Debian/Ubuntu software ecosystem.
>
> What exactly do you mean by "statically maintained"?

WPA keys written to /etc/network/interfaces as outlined in
/usr/share/doc/wpasupplicant/README.Debian.gz, Section 2. Mode #1:
Managed Mode.

That mode is pretty popular among debian users, properly documented,
widely advocated and obviously also used by a lot of ubuntu users,
judging from the numbers of duplicates of this bug.

--
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4

[Reinhard Tartler (siretart)]

Soren Hansen <email address hidden> writes:

> Glancing at its pre-up script, I'm guessing your problem is this:
>
> # quit if executables are not installed
> if [ ! -x "$WPA_SUP_BIN" ] || [ ! -x "$WPA_CLI_BIN" ]; then
> exit 0
> fi

This is a perfectly valid use case when the wpasupplicant package is
removed but not purged.

The wpasupplicant-ifupdown script is installed as a conffile, because we
expected that local users might want to do some changes to
it. /etc/network/if-*.d is installed in /etc/ as conffile for a reason
after all.

What we could do however is to move the ifupdown conffile to
/var/lib/wpasupplicant or something and exit with exitcode 1 to indicate
to ifupdown that starting the interface has failed. After skimming the
ifupdown code, it seems that /etc/init.d/networking (which calls 'ifup
-a') *should* then retry to 'up' the interface.

Kel, how do you think about this idea?

--
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4

[kelmo (kelrin)]

On Friday 05 December 2008 00:25:02 Reinhard Tartler wrote:
> Soren Hansen <email address hidden> writes:
>
> > Glancing at its pre-up script, I'm guessing your problem is this:
> >
> > # quit if executables are not installed
> > if [ ! -x "$WPA_SUP_BIN" ] || [ ! -x "$WPA_CLI_BIN" ]; then
> > exit 0
> > fi
>
> This is a perfectly valid use case when the wpasupplicant package is
> removed but not purged.
>
> The wpasupplicant-ifupdown script is installed as a conffile, because we
> expected that local users might want to do some changes to
> it. /etc/network/if-*.d is installed in /etc/ as conffile for a reason
> after all.
>
> What we could do however is to move the ifupdown conffile to
> /var/lib/wpasupplicant or something and exit with exitcode 1 to indicate
> to ifupdown that starting the interface has failed. After skimming the
> ifupdown code, it seems that /etc/init.d/networking (which calls 'ifup
> -a') *should* then retry to 'up' the interface.
>
> Kel, how do you think about this idea?

After Soren jumped all over my last casual remark, i don't dare comment until
such a time i can review the entire bug, which aint gonna be for a few weeks.

I haven't seen any sign of this bug in the Debian BTS and have not read the
full LP 44194.

Kel.

[Soren Hansen (soren)]

>>> but rather, the inflexibility of the statically maintained ifupdown
>>> software on an ever evolving Debian/Ubuntu software ecosystem.
>> What exactly do you mean by "statically maintained"?
> WPA keys written to /etc/network/interfaces as outlined in
> /usr/share/doc/wpasupplicant/README.Debian.gz, Section 2. Mode #1:
> Managed Mode.

Ah. I would have perhaps called that "statically configured", not
"statically maintained software". Ok.

> That mode is pretty popular among debian users, properly documented,
> widely advocated and obviously also used by a lot of ubuntu users,
> judging from the numbers of duplicates of this bug.

I've never questioned the validity of the bug, only the fix.

[Soren Hansen (soren)]

> Glancing at its pre-up script, I'm guessing your problem is this:
>
>> # quit if executables are not installed
>> if [ ! -x "$WPA_SUP_BIN" ] || [ ! -x "$WPA_CLI_BIN" ]; then
>> exit 0
>> fi
> This is a perfectly valid use case when the wpasupplicant package is
> removed but not purged.

I know why the check is there. That doesn't mean that the problem here
doesn't stem from that very check. If you have a separate /usr, the
wpasupplicant binary is likely to not be available when the interface is
discovered and the first configuration attempt is made. That's the case
you need to handle somehow. One of the ways to do so is to add a check
for the /var/run/network/initialized flag *before* the above check, and
*fail* if it's not there. That way, ifupdown will fail to configure the
interface the first time around, and will retry it when we reach S40networking.

> The wpasupplicant-ifupdown script is installed as a conffile, because
> we expected that local users might want to do some changes to it.
> /etc/network/if-*.d is installed in /etc/ as conffile for a reason
> after all.

I'm familiar with the concept of conffiles.

> What we could do however is to move the ifupdown conffile to
> /var/lib/wpasupplicant

What purpose would that serve? Whether a file is a conffile or not does
not depend on whether it's in /etc or not. Besides, there's not
guarantee that /var will be mounted at that point, and then you lose
again.

[Soren Hansen (soren)]

kelmo, you wouldn't read about this bug on Debian's BTS, because it's an Ubuntu specific bug. Last I checked Debian hasn't changed its boot process to do all this stuff dynamically, i.e. attempt configuration of everything you possibly can when it's discovered.

[Soren Hansen (soren)]

By the way, kelmo: Sorry if you felt "I jumped all over your casual remark". I was simply genuinely surprised why you'd suggest what you did. ifupdown is a framework (as you say yourself) for doing network configuration. It leaves everything but the very simple things to a sort of "modules" in the form of {pre,post}-{up,down} scripts so that we can put as little logic in ifupdown as possible and leave as many of the variables and as much as the handling as possible to these modules, and I don't understand how or why you'd change that.

[kelmo (kelrin)]

On Thursday 04 December 2008 20:29:26 Soren Hansen wrote:
> > I disagree to an extent, the problem doesn't sound specific to
> > wpa_supplicant's shell glue at all,
>
> If wpasupplicant tells ifupdown that it's done configuring the
> interface, and that everything went fine, but it that's not actually the
> case... That's wrong. No matter how you look at it.

It is, in my opinion, a pitfall of the design of ifupdown, as explained later
in this email.

>
> > but rather, the inflexibility of the statically maintained ifupdown
> > software on an ever evolving Debian/Ubuntu software ecosystem.
>
> What exactly do you mean by "statically maintained"?

Sorry for poor choice of words. It means that ifupdown software, in my opinion,
is in deep maintenance mode and is no longer actively developed. Meanwhile,
the Debian and Ubuntu systems and the demands of users of the named systems
evolve at a very rapid pace, especially when compared to ifupdown's progress
in the same time period.

>
> > A "cheap" fix could be included in an individual ifupdown hook script,
> > but it strikes me that the central network dispatcher/configurator
> > framework should be striving to handle situations such as that
> > described, as future hooks may also fall prey to this pitfall.
>
> "situations such as that described"? You mean a particular application
> having specific needs? It's not ifupdown's responsibility to know about
> all these things. If wpasupplicant doesn't know when it can expect to be
> succesful, how on earth should ifupdown know?

So the problems as I see them are:

* wpa_supplicant to the best of my knowledge requires stuff from /usr at runtime,
but is unable to signal/communicate with ifupdown to let it know that.
* The wpasupplicant package installs its binaries to /sbin, which makes it seem
like the binaries can be used in operations which occur at early boot when /usr
is unmounted, but in fact they cannot.
* wpa_supplicant is executed via shell scripts which are installed as conffiles
and must also adhere to the Debian policy that applies to conffiles. The
conffile is unable to know whether a required binary has been removed, or if it
is simply not available yet because the system is in an early stage of system
initialization and /usr is on another filesystem which is still to be mounted.

Firstly, the case of requiring /usr is not wpa_supplicant specific. For example
iw will require libnl from /usr in future if libnl doesn't change [1].

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=499537#22

Secondly, wpa_supplicant and wpa_cli live in /sbin and not /usr/sbin. The
speculation that i've read in this report which hightlights the following
code is the cause of the problem is likely invalid:
> # quit if executables are not installed
> if [ ! -x "$WPA_SUP_BIN" ] || [ ! -x "$WPA_CLI_BIN" ]; then
> exit 0
> fi

because:
> WPA_SUP_BIN="/sbin/wpa_supplicant"
> WPA_CLI_BIN="/sbin/wpa_cli"

The most likely reason that wpa_supplicant fails to do anything at early boot
sequence with an unmounted /usr is related to fact it links with stuff there.
Someone could redirect stdout and stderr via exec trick to a logfile to try and
capture any err...

[kelmo (kelrin)]

On Friday 05 December 2008 05:13:10 Soren Hansen wrote:
> kelmo, you wouldn't read about this bug on Debian's BTS, because it's an
> Ubuntu specific bug. Last I checked Debian hasn't changed its boot
> process to do all this stuff dynamically, i.e. attempt configuration of
> everything you possibly can when it's discovered.
>

Yeah, the allow-hotplug ifupdown class provided by /lib/udev/net.agent does
this, but it delays interface ifup action until lo is configured. This usually
means the ifup action is performed at roughtly the time /etc/init.d/networking
has done its thing.

ifplugd's hotplug agent (recent versions) does similarly.

Thanks, Kel.

[Michael Evans (mjevans1983)]

I think this can probably be decomposed in to a series of problems that result in this bug:

1) Networking in specific, but all devices in general, should have a way of specifying if it's an 'initrd', 'early', 'late' or 'later/optional' type of event. initrd items would be rolled in to the initrd script with their dependencies and run before the root file system was mounted. early items would run after root was mounted RO but before anything else. late would run after root is mounted RO (very likely with all other filesystems being mounted first at this point). later/optional events would run in the background automatically after a terminal/X login was started if the hardware existed, or would trigger after that stage when hardware is hot-plugged.

2) early WPA might work if the correct directories are mounted as RW using a ram based filesystem and any files that required write access were there. (/dev/shm comes to mind)

3) Sane defaults for these things, and 'Are you REALLY sure you want to do this? It looks like you're going to brick your system if you start your crypto late.' with full mollyguard style 'enter the system name and module name in question' confirmation boxes for deviations on items marked as critical.

3 would allow for the current (everything but /boot) LUKS + LVM setups to properly integrate (instead of with a warning and occasional failures) in to the initrd but later upgrade if users wanted and it was supported, to unlocking things like /home based on the users current login information, or some other UI improvement.

[Michael Evans (mjevans1983)]

Drat, I thought I'd corrected it, but probably undid once too far later in editing. it should read: "Late would run after root is mounted RW (very likely with all other filesystems being mounted first at this point)." not RO.

[Colin Watson (cjwatson)]

Having reviewed the whole bug, I have to say that this all seems like an order of magnitude more work than is really required. The suggested approaches involve lots of complex interface definition, strict ordering constraints, and the need to warn users when they violate constraints. Why not make it all simpler instead? Simpler systems with fewer constraints tend to be more robust.

There is precedent in Ubuntu for moving libraries and programs to the root filesystem when it is necessary to deal properly with hardware activation, whether that be hard ordering constraints (such as fuse and ntfs-3g being needed early on in some environments) or cases such as this where jobs could in principle be reordered to work in most cases but it's really much simpler to decouple things. I think the proper fix for this bug is to move the various libraries needed by wpasupplicant to the root filesystem, and to ensure that wpasupplicant can operate with only /var/lock/ and /var/run/ writable, as is the case in the early stages of the Ubuntu boot process. /var/lock/ and /var/run/ are mounted as temporary filesystems precisely so that programs can run early, without having to arrange to have themselves deferred until a later time. It is obviously simpler not to need to defer execution.

While I'm sympathetic to the goal of a small root filesystem, it should take second place to a correct and reliable boot process. This proposed change would increase the size of the root filesystem by perhaps 2MB, which I think we can live with. And, of course, as has already been observed it's a bug for programs in the root filesystem to link to libraries in /usr, which wpasupplicant already does; it's often reasonable to fix this by changing the libraries.

The only extra complexity I have found here so far is that wpasupplicant expects to be able to write to log files in /var/log/. This is perhaps not absolutely critical to fixing this bug but it would obviously be desirable to fix it. Any ideas? One simple approach would be to move it to syslog with an appropriate facility, priority, and prefix; I don't know whether the wpasupplicant maintainers would find this acceptable.

[kelmo (kelrin)]

On Friday 27 February 2009 20:51:41 Colin Watson wrote:
> Having reviewed the whole bug, I have to say that this all seems like an
> order of magnitude more work than is really required. The suggested
> approaches involve lots of complex interface definition, strict ordering
> constraints, and the need to warn users when they violate constraints.
> Why not make it all simpler instead? Simpler systems with fewer
> constraints tend to be more robust.
>
> There is precedent in Ubuntu for moving libraries and programs to the
> root filesystem when it is necessary to deal properly with hardware
> activation, whether that be hard ordering constraints (such as fuse and
> ntfs-3g being needed early on in some environments) or cases such as
> this where jobs could in principle be reordered to work in most cases
> but it's really much simpler to decouple things. I think the proper fix
> for this bug is to move the various libraries needed by wpasupplicant to
> the root filesystem, and to ensure that wpasupplicant can operate with
> only /var/lock/ and /var/run/ writable, as is the case in the early
> stages of the Ubuntu boot process. /var/lock/ and /var/run/ are mounted
> as temporary filesystems precisely so that programs can run early,
> without having to arrange to have themselves deferred until a later
> time. It is obviously simpler not to need to defer execution.
>

Yeah, makes sense.

> While I'm sympathetic to the goal of a small root filesystem, it should
> take second place to a correct and reliable boot process. This proposed
> change would increase the size of the root filesystem by perhaps 2MB,
> which I think we can live with. And, of course, as has already been
> observed it's a bug for programs in the root filesystem to link to
> libraries in /usr, which wpasupplicant already does; it's often
> reasonable to fix this by changing the libraries.
>
> The only extra complexity I have found here so far is that wpasupplicant
> expects to be able to write to log files in /var/log/. This is perhaps
> not absolutely critical to fixing this bug but it would obviously be
> desirable to fix it. Any ideas? One simple approach would be to move it
> to syslog with an appropriate facility, priority, and prefix; I don't
> know whether the wpasupplicant maintainers would find this acceptable.
>

syslog debug output was just committed upstream development branch [0] fwiw.

Thanks, Kel.

[0] http://w1.fi/gitweb/gitweb.cgi?p=hostap.git;a=commitdiff;h=daa70d49de46278cf5a92b279d1d262af573cb63

[Colin Watson (cjwatson)]

zlib (1:1.2.3.3.dfsg-12ubuntu2) jaunty; urgency=low

  * Move zlib.so.* to /lib (LP: #44194).

 -- Colin Watson <email address hidden> Wed, 18 Mar 2009 16:28:00 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package pcsc-lite - 1.4.102-1ubuntu2

---------------
pcsc-lite (1.4.102-1ubuntu2) jaunty; urgency=low

  * Move runtime libraries to /lib, for the benefit of wpasupplicant
    (LP: #44194).

 -- Colin Watson <email address hidden> Wed, 18 Mar 2009 16:34:28 +0000

[Stephen Parry (sgparry)]

Is this fix too involved to be backported to Hardy as well? Hardy is the current LTS release...

[Ludovic Rousseau (ludovic-rousseau-gmail)]

Regarding the patch for pcsc-lite (moving libpcsclite.so.1 from /usr/lib to /usr) it is only a half fix. The smart card support will not be available since the daemon is still in /usr/sbin/pcscd and will only be started later (when /usr is available)

You should also:
- move pcscd in /sbin
- move all the smart card reader drivers from/usr/lib/pcsc/drivers to /lib/pcsc/drivers
- start /etc/init.d/pcsclite before wpa_supplicant.

Another solution is to NOT link wpa_suplicant with libpcsclite.so and do a dynamic loading of libpcsclite.so using dlopen(3) only when needed.
It may be impossible to use wpa_supplicant to mount /usr from a network server _and_ use the smart card for authentication in wpa_supplicant.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssl - 0.9.8g-15ubuntu2

---------------
openssl (0.9.8g-15ubuntu2) jaunty; urgency=low

  * Move runtime libraries to /lib, for the benefit of wpasupplicant
    (LP: #44194). Leave symlinks behind in /usr/lib (except on the Hurd)
    since we used to set an rpath there.

 -- Colin Watson <email address hidden> Fri, 06 Mar 2009 12:48:52 +0000

[Colin Watson (cjwatson)]

Stephen: Yes, I think so. I'd be pretty worried about stability of the LTS release if we did this.

Ludovic: Thanks for the note. What would you recommend? My personal inclination would be to say that smartcard auth for mounting /usr is sufficiently niche that we should just not support that, but you may have more experience than I here ...

[Ludovic Rousseau (ludovic-rousseau-gmail)]

Colin: I agree. A smart card is generally used to identify a user. So it is a good idea to use it to mount a home directory. But mounting /usr or / would using a smart card is a very strange configuration and very uncommon.

[Colin Watson (cjwatson)]

Could people affected by this bug who are running up-to-date Jaunty systems please follow the directions at:

  https://launchpad.net/~cjwatson/+archive/ppa

... to install the wpasupplicant package from there, and report whether it fixes this bug? This test is only valid if you're already running up-to-date Jaunty.

[Colin Watson (cjwatson)]

With Steve Langasek's agreement as release manager, I'm going to go ahead and upload my changes to Jaunty even though we haven't yet confirmed that they fix the original bug. Lars Wirzenius reported that my package at least doesn't break his use of WPA, so I don't think I've gone too badly wrong. I'll keep on monitoring this bug and reopen it if it seems to be necessary.

[Launchpad Janitor (janitor)]

This bug was fixed in the package wpasupplicant - 0.6.6-2ubuntu1

---------------
wpasupplicant (0.6.6-2ubuntu1) jaunty; urgency=low

  * Operate reasonably when /var is not yet mounted read-write (LP: #44194):
    - 07_syslog.patch: Backport syslog logging support from upstream.
    - 08_syslog_supplement.patch: Add a few more bits missing from the
      upstream patch, based on
      http://cvs.fedoraproject.org/viewvc/rpms/wpa_supplicant/OLPC-2/wpa_supplicant-0.5.7-use-syslog.patch.
      Compile with -DCONFIG_DEBUG_SYSLOG if CONFIG_DEBUG_SYSLOG is set in
      the configuration file.
    - Enable CONFIG_DEBUG_SYSLOG.
    - debian/ifupdown/functions.sh: Set WPA_SUP_LOGFILE to "syslog" by
      default, and use the -s option in this case.
    - debian/ifupdown/functions.sh: Silence wpa_log_* if /var/log is not yet
      writable; there is little we can do in this case (logger is in /usr,
      so may well also be unusable), and the user can always get more
      information by reconnecting later.

 -- Colin Watson <email address hidden> Thu, 09 Apr 2009 02:31:17 +0100

[Saivann Carignan (oxmosys)]

I confirm that the bug is fixed in jaunty with wpasupplicant 0.6.6-2ubuntu1 . Extreme thanks to Colin Watson, Steve Langasek and all other people that helped this bug to get fixed.

[kelmo (kelrin)]

On Thursday 09 April 2009 02:37:37 Colin Watson wrote:
> Could people affected by this bug who are running up-to-date Jaunty
> systems please follow the directions at:
>
> https://launchpad.net/~cjwatson/+archive/ppa
>
> ... to install the wpasupplicant package from there, and report whether
> it fixes this bug? This test is only valid if you're already running up-
> to-date Jaunty.
>

Any chance I could see a diff so that divergence between Debian and Ubuntu can
be avoided in future wpasupplicant package revisions?

Thanks, Kel.

[Reinhard Tartler (siretart)]

kelmo <email address hidden> writes:

>> https://launchpad.net/~cjwatson/+archive/ppa
>>
>> ... to install the wpasupplicant package from there, and report whether
>> it fixes this bug? This test is only valid if you're already running up-
>> to-date Jaunty.
>>
>
> Any chance I could see a diff so that divergence between Debian and Ubuntu can
> be avoided in future wpasupplicant package revisions?

The diff is published here:

http://patches.ubuntu.com/w/wpasupplicant/

both as full debdiff and with the dpatches extracted.

--
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4

[kelmo (kelrin)]

On Friday 10 April 2009 22:50:44 Reinhard Tartler wrote:
> kelmo <email address hidden> writes:
>
> >> https://launchpad.net/~cjwatson/+archive/ppa
> >>
> >> ... to install the wpasupplicant package from there, and report whether
> >> it fixes this bug? This test is only valid if you're already running up-
> >> to-date Jaunty.

Colin, I suggest one more change: change/remove
debian/patches/03_dbus_service_activation_logfile.patch and patch the D-Bus
service activation file to add the -s (syslog) command line option.

> >>
> >
> > Any chance I could see a diff so that divergence between Debian and Ubuntu can
> > be avoided in future wpasupplicant package revisions?
>
> The diff is published here:
>
> http://patches.ubuntu.com/w/wpasupplicant/
>
> both as full debdiff and with the dpatches extracted.

Thanks, I merged the difference into what will become 0.6.9-2 in Debian.

Kel.

[Colin Watson (cjwatson)]

Thanks, Kel and Reinhard. Sorry I didn't reply or send the patch earlier; the Easter holidays slowed me down somewhat (I was in a tearing hurry on Thursday to get everything done). Thanks a lot for the merge - I appreciate the work.

Good catch regarding the D-Bus service. I've uploaded the attached patch.

[Paul Boddie (paul-boddie)]

Thanks to everyone who figured this bug out. I installed backported versions of ifupdown and netbase from Intrepid and this specific problem seems to be solved for me.

However, after having to hunt down the apparent cause of the problem (useless wlan0 interface being brought up at boot which does actually work properly after an ifdown/ifup combination but not before) in order to be able to formulate it in a way that might yield decent search results, then having to sift through the usual "hit everything with a hammer" antics on Ubuntu Forums ("sleep 20", inserting "ifdown/ifup" in random init.d files) before finally striking gold in the form of this and numerous duplicate bug reports, it is somewhat disappointing that two years or so pass before anything resembling a fix is pushed out, and by then no-one is interested in applying it to the "long term support" release. (It would also have helped had I not needed to strip away layer upon layer of tools - KNetworkManager and the intrusive/disruptive avahi-daemon/avahi-autoipd - all to accomplish the probably extremely common use-case of wanting to automatically connect to one's home network.)

I know that people are reluctant to update packages in released distributions in order to avoid breaking functionality, but when the definition of a "fix" excludes the act of making functionality work that was broken upon a distribution's release, one has to wonder which caveats also apply to the definitions of "support", "long term support" and "update" being used, especially since I started out with the Kubuntu 8.04.2 "update" which was surely released after the fixes issued above. And suggesting that people just upgrade to the next distribution version isn't exactly acceptable, even without such things as the KDE 3-to-4 transition which would be involved if I were to chase the fixed packages, not least because I might well experience breakage elsewhere. In the end, I'd be running some beta or other, which would hardly be consistent with the original objective of running a stable, working distribution.

[patpat (masottaus)]

I agree 100% with Paul;
I preffer a slower rate on new releases than this mess of unsolved bugs on LTS editions...

how can we serioulsy consider an LTS edition if bugs are only solved on new releases forcing us to upgrade when what we really want is stability???