Message-Id: <email address hidden>
Date: Wed, 21 Sep 2005 03:38:06 +0100
From: Scott James Remnant <email address hidden>
To: <email address hidden>
Cc: <email address hidden>, Marco d'Itri <email address hidden>
Subject: The case of udev and the missing /dev/input/mice
Background: in the upcoming Ubuntu 5.10 we've been having some problems
with /dev/input/mice not being created on startup despite the "mousedev"
module being hard-loaded early in the boot sequence.
(http://bugzilla.ubuntu.com/show_bug.cgi?id=3D12915 for those interested).
Debian has had similar problems too (http://bugs.debian.org/317333) and
found that starting udevd earlier manually seemed to fix it.
After much debugging, I've finally figured out what's going on ... it's
a bit of a story, but here goes...
Your system boots up and gets to the S:S20modules-init-tools stage,
that's where we read /etc/modules and modprobe the modules in order.
Now modprobe is basically just a kernel request, and these days tends to
return pretty quicky to userspace without blocking for everything to
happen.
Deep Black Magic happens inside the kernel, and once it's done it
generates a series of hotplug events which it passes back to userspace
through two means; by running the program specified
in /proc/sys/kernel/hotplug with interesting environment; and also
through a netlink socket.
/proc/sys/kernel/hotplug is "udevsend", a tool that gathers up this
environment and sends it over a local socket to the "udevd" process that
marshals all of these events. If there's no daemon listening it tries
to start one up, and will retry sending the event for a while until it
gets to the other end.
Now we have a whole bunch of udevsend processes all run at pretty much
the same time, all of these try to start up udevd and all of the udevd
processes try to bind to the local socket to receive events on. One of
them wins, the rest die and go away. A little time passes by which time
all of the running udevsend will have dispatched their event to this
udevd that will marshal them.
This udevd _also_ begins listening on the netlink socket, as it's a
better way to get events from the kernel than having it execute
something which mucks around with IPC to get it to us.
Meanwhile the kernel is happily generating both /proc/sys/kernel/hotplug
and netlink events for what's happening on the box, in fact it's been
doing this all the time udevd has been getting its clothes on.
If the module sequence loaded is something like "psmouse, mousedev, ...,
lp" (exactly as it is in breezy machines that have been upgraded from
warty/hoary[0]) you may find that the first netlink event you receive is
actually for the printer port.
But that's ok, we had udevsend events for the rest...
Well, that's the theory; sadly here's the practice.
On receiving the netlink event for the printer port, udevd disables
receipt of any "sequence numbered" events from udevsend (ie. those that
will almost certainly be duplicated over the netlink socket).
Unfortunately this means all the udevsend events we're about to receive
from the processes that backed off a second or so while fighting over
who got to start udevd[1].
These udevsend processes deliver their events to udevd, which cheerfully
ignores them because it thinks it's going to get another copy over the
netlink socket any second now. Unfortunately the netlink event has
already been and gone, and we just ignored an event we weren't supposed
to.
The two problems as I see them are:
1) The fact that receiving a netlink event disables sequence numbered
udevsend events, when there's already code to deal with de-duping
events anyway. Is there actually any need for this additional check,
can't we just queue both events and have them ignored by
msg_queue_insert() ?
2) That this ignoring of events is done at receipt, rather than in queue
order. This means that the "later" parport_pc netlink event is able
to disable queueing of udevsend events with a lower sequence number.
I can envisage that #1 is necessary in case the time between receiving
the udevsend and netlink event is so long that we've already processed
and removed one of the events by the time the second is queued. In
which case the problem becomes fixing #2, however unless the kernel
promises strict ordering of events over the netlink socket (which I
doubt, otherwise it wouldn't need sequence numbers), we can't assume
that we've received all of the pre-netlink events we are going to.
I suspect the right solution is actually to implement history of what
events we've already processed, and de-dupe them that way; rather than
ignoring messages on receipt.
Scott
[0] A common "fix" has been to simply install breezy fresh; this happens
to change the /etc/modules order slightly and thus hide the bug.
[1] And if we deliberately start udevd before we begin any of this
module loading, it sees the netlink event, and thus again hides the=20
bug.
--=20
Scott James Remnant
<email address hidden>
--=-iu5esJA+UWFxRG9iv+ol
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part
Message-Id: <email address hidden>
Date: Wed, 21 Sep 2005 03:38:06 +0100
From: Scott James Remnant <email address hidden>
To: <email address hidden>
Cc: <email address hidden>, Marco d'Itri <email address hidden>
Subject: The case of udev and the missing /dev/input/mice
--=-iu5esJA+ UWFxRG9iv+ ol Transfer- Encoding: quoted-printable
Content-Type: text/plain
Content-
Background: in the upcoming Ubuntu 5.10 we've been having some problems bugzilla. ubuntu. com/show_ bug.cgi? id=3D12915 for those interested).
with /dev/input/mice not being created on startup despite the "mousedev"
module being hard-loaded early in the boot sequence.
(http://
Debian has had similar problems too (http:// bugs.debian. org/317333) and
found that starting udevd earlier manually seemed to fix it.
After much debugging, I've finally figured out what's going on ... it's
a bit of a story, but here goes...
Your system boots up and gets to the S:S20modules- init-tools stage,
that's where we read /etc/modules and modprobe the modules in order.
Now modprobe is basically just a kernel request, and these days tends to
return pretty quicky to userspace without blocking for everything to
happen.
Deep Black Magic happens inside the kernel, and once it's done it kernel/ hotplug with interesting environment; and also
generates a series of hotplug events which it passes back to userspace
through two means; by running the program specified
in /proc/sys/
through a netlink socket.
/proc/sys/ kernel/ hotplug is "udevsend", a tool that gathers up this
environment and sends it over a local socket to the "udevd" process that
marshals all of these events. If there's no daemon listening it tries
to start one up, and will retry sending the event for a while until it
gets to the other end.
Now we have a whole bunch of udevsend processes all run at pretty much
the same time, all of these try to start up udevd and all of the udevd
processes try to bind to the local socket to receive events on. One of
them wins, the rest die and go away. A little time passes by which time
all of the running udevsend will have dispatched their event to this
udevd that will marshal them.
This udevd _also_ begins listening on the netlink socket, as it's a
better way to get events from the kernel than having it execute
something which mucks around with IPC to get it to us.
Meanwhile the kernel is happily generating both /proc/sys/ kernel/ hotplug
and netlink events for what's happening on the box, in fact it's been
doing this all the time udevd has been getting its clothes on.
If the module sequence loaded is something like "psmouse, mousedev, ...,
lp" (exactly as it is in breezy machines that have been upgraded from
warty/hoary[0]) you may find that the first netlink event you receive is
actually for the printer port.
But that's ok, we had udevsend events for the rest...
Well, that's the theory; sadly here's the practice.
On receiving the netlink event for the printer port, udevd disables
receipt of any "sequence numbered" events from udevsend (ie. those that
will almost certainly be duplicated over the netlink socket).
Unfortunately this means all the udevsend events we're about to receive
from the processes that backed off a second or so while fighting over
who got to start udevd[1].
These udevsend processes deliver their events to udevd, which cheerfully
ignores them because it thinks it's going to get another copy over the
netlink socket any second now. Unfortunately the netlink event has
already been and gone, and we just ignored an event we weren't supposed
to.
The two problems as I see them are:
1) The fact that receiving a netlink event disables sequence numbered queue_insert( ) ?
udevsend events, when there's already code to deal with de-duping
events anyway. Is there actually any need for this additional check,
can't we just queue both events and have them ignored by
msg_
2) That this ignoring of events is done at receipt, rather than in queue
order. This means that the "later" parport_pc netlink event is able
to disable queueing of udevsend events with a lower sequence number.
I can envisage that #1 is necessary in case the time between receiving
the udevsend and netlink event is so long that we've already processed
and removed one of the events by the time the second is queued. In
which case the problem becomes fixing #2, however unless the kernel
promises strict ordering of events over the netlink socket (which I
doubt, otherwise it wouldn't need sequence numbers), we can't assume
that we've received all of the pre-netlink events we are going to.
I suspect the right solution is actually to implement history of what
events we've already processed, and de-dupe them that way; rather than
ignoring messages on receipt.
Scott
[0] A common "fix" has been to simply install breezy fresh; this happens
to change the /etc/modules order slightly and thus hide the bug.
[1] And if we deliberately start udevd before we begin any of this
module loading, it sees the netlink event, and thus again hides the=20
bug.
--=20
Scott James Remnant
<email address hidden>
--=-iu5esJA+ UWFxRG9iv+ ol pgp-signature; name=signature.asc Description: This is a digitally signed message part
Content-Type: application/
Content-
-----BEGIN PGP SIGNATURE-----
iFMl4yK4RAibQAK CisLcYPxPGnCY/ J5KFx+WC/ K1xhwCcCN1I Xlc8xAVs=
Version: GnuPG v1.4.1 (GNU/Linux)
iD8DBQBDMMeOSnQ
aJT9YmNBpq436B5
=10BY
-----END PGP SIGNATURE-----
--=-iu5esJA+ UWFxRG9iv+ ol--