"restart udev" hangs

[Jürgen Kreileder]

Binary package hint: udev

On two headless servers "restart udev" just hangs. When the udev package gets updated the only way to get aptitude to complete the upgrade is to kill the 'restart udev' process from the outside.

The last thing 'strace restart udev' shows is:
ocket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_FILE, path=@"/com/ubuntu/upstart"}, 22) = 0
fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
geteuid() = 0
getsockname(3, {sa_family=AF_FILE, NULL}, [2]) = 0
poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, revents=POLLOUT}])
write(3, "\0", 1) = 1
sendto(3, "AUTH EXTERNAL 30\r\n", 18, MSG_NOSIGNAL, NULL, 0) = 18
poll([{fd=3, events=POLLIN}], 1, -1) = 1 ([{fd=3, revents=POLLIN}])
read(3, "OK d71fcfbb5e2671f0f0575cf800000"..., 2048) = 37
poll([{fd=3, events=POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "NEGOTIATE_UNIX_FD\r\n", 19, MSG_NOSIGNAL, NULL, 0) = 19
poll([{fd=3, events=POLLIN}], 1, -1) = 1 ([{fd=3, revents=POLLIN}])
read(3, "AGREE_UNIX_FD\r\n", 2048) = 15
poll([{fd=3, events=POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "BEGIN\r\n", 7, MSG_NOSIGNAL, NULL, 0) = 7
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\2\1\t\0\0\0\1\0\0\0_\0\0\0\1\1o\0\23\0\0\0/com/ubu"..., 112}, {"\4\0\0\0udev\0", 9}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 121
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\2\1\1\"\0\0\0\1\0\0\0\17\0\0\0\5\1u\0\1\0\0\0\10\1g\0\1o\0\0"..., 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 66
recvmsg(3, 0x7fffee846a60, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\2\1\10\0\0\0\2\0\0\0i\0\0\0\1\1o\0\35\0\0\0/com/ubu"..., 128}, {"\0\0\0\0\1\0\0\0", 8}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 136
poll([{fd=3, events=POLLIN}], 1, -1

I can only reproduce the problem on servers (with sshd, apache2, mysql, ... no dbus). On desktop installations the restart command works fine.
I first saw the problem on lucid but it still occurs on maverick.

[Jeremy Davis (jedmeister)]

This looks like the same bug that a number of people have reported in Turnkey Linux (an Ubuntu Lucid based server distro: http://www.turkenylinux.org/).

See:
Bug report: https://bugs.launchpad.net/turnkeylinux/+bug/653422
TKL forum thread: http://www.turnkeylinux.org/forum/support/20100928/tkl-lucid-core-apt-get-upgrade-freeze-installing-udev-151-121

[Dude4Linux (dude4linux)]

On a hunch, I decided to try testing for an undocumented dependency between udev and dbus.

Installing dbus failed to fix udev hang upon start.

# dpkg-query -l dbus udev
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-================-================-================================================
un dbus <none> (no description available)
ii udev 151-12.3 rule-based device node and kernel event manager

# apt-get install dbus
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  dbus-x11
Recommended packages:
  consolekit
The following NEW packages will be installed:
  dbus
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 189kB of archives.
After this operation, 618kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ lucid/main dbus 1.2.16-2ubuntu4 [189kB]
Fetched 189kB in 1s (157kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously deselected package dbus.
(Reading database ... 31546 files and directories currently installed.)
Unpacking dbus (from .../dbus_1.2.16-2ubuntu4_i386.deb) ...
Processing triggers for man-db ...
Setting up dbus (1.2.16-2ubuntu4) ...
Adding system user `messagebus' (UID 107) ...
Adding new group `messagebus' (GID 111) ...
Adding new user `messagebus' (UID 107) with group `messagebus' ...
Not creating home directory `/var/run/dbus'.
dbus start/running, process 4833

# status udev
udev start/running, process 348
# stop udev
udev stop/waiting
# status udev
udev stop/waiting
r# start udev
Terminated
# status udev
udev start/starting

[Dude4Linux (dude4linux)]

Stack traces for either 'start udev' or 'restart udev' hang at exactly the same point as reported by Jürgen Kreileder. The traces stop midway during a poll. I had to kill the 'start udev' task from a second terminal which resulted in the Terminated response above.

[Anatol Pomozov (anatol)]

Ubuntu 10.04.

I am trying to implement my own upstart script and "start mydaemon" also hangs on this point.

getuid() = 0
socket(PF_FILE, SOCK_STREAM, 0) = 3
connect(3, {sa_family=AF_FILE, path=@"/com/ubuntu/upstart"}, 22) = 0
fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
fcntl(3, F_GETFD) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
geteuid() = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f5200d05af0}, {SIG_DFL, [], 0}, 8) = 0
poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, revents=POLLOUT}])
write(3, "\0", 1) = 1
write(3, "AUTH EXTERNAL 30\r\n", 18) = 18
poll([{fd=3, events=POLLIN}], 1, -1) = 1 ([{fd=3, revents=POLLIN}])
read(3, "OK 695795fefc908e7694d65a6d4f623"..., 2048) = 37
poll([{fd=3, events=POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
write(3, "BEGIN\r\n", 7) = 7
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{"l\1\2\1\17\0\0\0\1\0\0\0_\0\0\0\1\1o\0\23\0\0\0/com/ubu"..., 112}, {"\n\0\0\0checkpatch\0", 15}], 2) = 127
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
read(3, "l\2\1\1(\0\0\0\1\0\0\0\17\0\0\0\5\1u\0\1\0\0\0\10\1g\0\1o\0\0"..., 2048) = 72
read(3, 0x7f520329fbe0, 2048) = -1 EAGAIN (Resource temporarily unavailable)
writev(3, [{"l\1\2\1\10\0\0\0\2\0\0\0q\0\0\0\1\1o\0#\0\0\0/com/ubu"..., 136}, {"\0\0\0\0\1\0\0\0", 8}], 2) = 144
poll([{fd=3, events=POLLIN}], 1, -1

I do not understand why it hangs. Where I can find the upstart logs? How to debug this problem? Is there any workaround for the issue?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[David Baehrens (david-baehrens)]

I confirm this issue on Ubuntu Server 12.04.2 LTS with udev version 175-0ubuntu9.3

Here, apt-get dist-upgrade hangs in udev start of update post-install phase with the same strace as in the description.

Reading through the Turnkey Linux forum thread referenced in comment #1, I was able to resolve the deadlock by "service udev stop" in another terminal, while the update still hangs in udev start. This hangs too, but now the update continues and succeeds. Last, I had to terminate the stop command by hitting Ctrl-C