Boot scripts hang in plymouth --ping (was : console-setup installation hangs)

[Scott Merrilees]

Binary package hint: console-setup

System is Xen/xVM domU amd64 lucid, minimal build, no X11.

# lsb_release -rd
Description: Ubuntu lucid (development branch)
Release: 10.04
# apt-cache policy console-setup
console-setup:
  Installed: 1.34ubuntu14
  Candidate: 1.34ubuntu14
  Version table:
 *** 1.34ubuntu14 0
        500 http://au.archive.ubuntu.com/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status

Recently did an aptitude upgrade, and the install hung on installation of console-setup.

Did a pstree, and found it seemed to be hung calling plymouth. Killed plymouth process, and install seemed to complete.

However, if I do an aptitude reinstall console-setup, I get the same issue.

Same if I run from hvc0 or pts/n.

# ps -ft hvc0
UID PID PPID C STIME TTY TIME CMD
root 701 1 0 Apr14 hvc0 00:00:00 /bin/login --
root 4378 32560 0 11:10 hvc0 00:00:00 aptitude reinstall console-setup
root 32560 701 0 11:07 hvc0 00:00:00 -bash
# pstree -pa 701
login,701 --
  bash,32560
      aptitude,4378 reinstall console-setup
          dpkg,4585 --status-fd 32 --configure console-setup
             frontend,4586 -w /usr/share/debconf/frontend...
                 console-setup.p,4761 ...
                     plymouth,4777 --ping
          {aptitude},4379
# cat /proc/cmdline
root=/dev/mapper/xe00vg-root ro ipv6.disable=1 console=hvc0 quiet
# strace -p 4777
Process 4777 attached - interrupt to quit
clock_gettime(CLOCK_MONOTONIC, {180706, 738772316}) = 0
epoll_wait(4, ^C <unfinished ...>
Process 4777 detached

Still hung after 1 hour.

[Scott Merrilees (scott.merrilees)]

Tried to apport-cli -u however bug 561421 precludes this action.

[aporter (aporter)]

Is there a workaround for this?

[Scott Merrilees (scott.merrilees)]

If try to remove console-setup, and accept the default to install console-setup-mini, I get this:

# pstree -ap 17906
aptitude,17906 purge console-setup
  ├─dpkg,18150 --status-fd 32 --configure console-setup-mini
  │ └─frontend,18151 -w /usr/share/debconf/frontend...
  │ └─console-setup-m,18320 ...
  │ └─plymouth,18365 --ping
  └─{aptitude},17907

So this hangs in plymouth too. Maybe plymouth shouldn't be involved in a serial/hvc0 console setup?

I can kill 18365, and the install continues, however I'm unsure as to what state this leaves the system.

# aptitude purge console-setup
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
The following packages are BROKEN:
  kbd ubuntu-minimal
The following packages will be REMOVED:
  console-setup{p}
0 packages upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
Need to get 0B of archives. After unpacking 1,430kB will be freed.
The following packages have unmet dependencies:
  ubuntu-minimal: Depends: console-setup but it is not installable
  kbd: Depends: console-setup but it is not installable
The following actions will resolve these dependencies:

Remove the following packages:
console-terminus

Install the following packages:
console-setup-mini [1.34ubuntu14 (lucid)]

Score is 104

Accept this solution? [Y/n/q/?] y
The following NEW packages will be installed:
  console-setup-mini{a}
The following packages will be REMOVED:
  console-setup{p} console-terminus{a}
0 packages upgraded, 1 newly installed, 2 to remove and 0 not upgraded.
Need to get 1,012kB of archives. After unpacking 864kB will be freed.
Do you want to continue? [Y/n/?] y
Writing extended state information... Done
Get:1 http://au.archive.ubuntu.com/ubuntu/ lucid/universe console-setup-mini 1.34ubuntu14 [1,012kB]
Fetched 1,012kB in 0s (1,894kB/s)
Preconfiguring packages ...
dpkg: console-setup: dependency problems, but removing anyway as you requested:
 ubuntu-minimal depends on console-setup.
 kbd depends on console-setup.
(Reading database ... 25797 files and directories currently installed.)
Removing console-setup ...
Purging configuration files for console-setup ...
Processing triggers for ureadahead ...
Processing triggers for man-db ...
(Reading database ... 25591 files and directories currently installed.)
Removing console-terminus ...
Selecting previously deselected package console-setup-mini.
(Reading database ... 25422 files and directories currently installed.)
Unpacking console-setup-mini (from .../console-setup-mini_1.34ubuntu14_all.deb) ...
Processing triggers for man-db ...
Setting up console-setup-mini (1.34ubuntu14) ...
Terminated
update-initramfs: deferring update (trigger activated)

Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-2.6.32-21-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
Writing extended state information... Done

#

[Scott Merrilees (scott.merrilees)]

# ps -ef|grep setup
root 26759 18404 0 10:11 pts/3 00:00:00 aptitude reinstall console-setup-mini
root 26959 26759 0 10:11 pts/2 00:00:00 /usr/bin/dpkg --status-fd 32 --configure console-setup-mini
root 26960 26959 0 10:11 pts/2 00:00:00 /usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/console-setup-mini.postinst configure 1.34ubuntu14
root 27135 26960 0 10:11 pts/2 00:00:00 /bin/sh /var/lib/dpkg/info/console-setup-mini.postinst configure 1.34ubuntu14
root 27192 17886 0 10:14 pts/1 00:00:00 grep setup
# ps -ef|grep plym
root 214 1 0 Apr14 ? 00:00:00 /sbin/plymouthd --mode=boot --attach-to-session
root 476 1 0 Apr14 ? 00:00:00 /bin/plymouth update-root-fs --read-write
root 550 1 0 Apr14 ? 00:00:00 /bin/plymouth show-splash
root 702 1 0 Apr14 ? 00:00:00 /bin/plymouth quit
root 27179 27135 0 10:11 pts/2 00:00:00 plymouth --ping
root 27194 17886 0 10:14 pts/1 00:00:00 grep plym
#

Seems this maybe more plymouth related than console setup?

I took the chance to reboot the server, and get the following trace:

init: ureadahead-other main process (289) terminated with status 4
init: ureadahead-other main process (309) terminated with status 4
 * Starting AppArmor profiles [ OK ]
 * Setting sensors limits [ OK ]
 * Starting domain name service... bind9 [ OK ]
 * ipvsadm is not configured to run. Please run dpkg-reconfigure ipvsadm
 * Starting nagios-nrpe nagios-nrpe [ OK ]
 * Starting DHCP server dhcpd3 * check syslog for diagnostics.
                                                                         [fail]
 * Starting web server apache2 [ OK ]
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Plymouth command failed

Ubuntu lucid (development branch) svr hvc0

However the mount table looks normal.

[Scott Merrilees (scott.merrilees)]

I had done an aptitude upgrade before then reboot which did some plymouth updates.

After the reboot, I has able to complete an aptitude reinstall cosnole-setup-mini without killing plymouth like I have been.

So I then did an aptitude install console-setup, and it successfully reinstalled console-setup and console-terminus.

I rebooted again I stll get the mountall Plymouth messages.

[Scott Merrilees (scott.merrilees)]

Last night I build another minimal server, and it had the same plymouth bug.

This morning I applied the current setup of updates, and the plymoth bug disappeared on both servers.

aptitude reinstall console-setup works ok too, so I can't see the original bug either anymore.

[aporter (aporter)]

I don't see the original bug anymore either. I assume a fix was released?

[Wladimir Mutel (mwg)]

This bug affects me on real hardware with non-GUI x86_64 server Ubuntu upgraded recently from Maverick to Natty
Plymouth ping just hangs, epolling for file handle 3 which lsof shows as :

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
plymouth 1449 root 3u 0000 0,9 0 5392 anon_inode

I can't understand what this means.
On boot, these plymouth-related processes are started :

/sbin/plymouthd --mode=boot --attach-to-session
plymouth-upstart-bridge

Fortunately, sshd is also started on boot, and I can access the system remotely.
Now checking if linking `which plymouth` to `which true` is of any help.

This is clearly a plymouth bug. Dislike this useless stuff very much .

[Wladimir Mutel (mwg)]

well, here are process trees with hung plymouth :

 1225 ? Ss 0:00 /bin/sh /etc/init.d/rc 2
 1443 ? S 0:00 \_ /bin/sh -e /etc/rc2.d/S15bind9 start
 1449 ? S 0:00 \_ plymouth --ping

 1720 ? Sl 0:01 \_ aptitude safe-upgrade
 1755 pts/2 Ss+ 0:00 \_ /usr/bin/dpkg --status-fd 42 --unpack --auto-deconfigure /var/cache/apt/archives/postfix_2.8.2-1ubuntu2.1_amd64.deb
 1761 pts/2 S+ 0:00 \_ /bin/sh -e /var/lib/dpkg/info/postfix.prerm upgrade 2.8.2-1ubuntu2.1
 1762 pts/2 S+ 0:00 \_ /bin/sh /usr/sbin/invoke-rc.d postfix stop
 1778 pts/2 S+ 0:00 \_ /bin/sh -e /etc/init.d/postfix stop
 1791 pts/2 S+ 0:00 \_ plymouth --ping

[Wladimir Mutel (mwg)]

ok, in the script /etc/lsb-base-logging.sh I found a controlling variable called 'loop' which turns off plymouth use when set to 'y'
set this variable in /etc/default/rcS and will check its effect on nexxt reboot.

[Steve Langasek (vorlon)]

> Plymouth ping just hangs, epolling for file handle 3
> I can't understand what this means.

It means that the plymouth daemon is hung on your system for some reason. Since you are running in a different hardware environment from the original submitter, please file a separate bug report by running 'ubuntu-bug plymouth' from the affected system, then attach the output of 'strace' run on the plymouthd process.

[Steve Langasek (vorlon)]

Scott, as the original submitter if you are still affected by this issue, please run 'apport-collect 564360'.

[Steve Langasek (vorlon)]

(oh, I see this bug was marked 'fix released' over a year ago; no idea why Wladimir chose this bug to follow up to, then...)

[Scott Merrilees (scott.merrilees)]

Linux server 2.6.35-24-server #42-Ubuntu SMP Thu Dec 2 03:58:11 UTC 2010 x86_64 GNU/Linux
Ubuntu 10.10

Welcome to the Ubuntu Server!
 * Documentation: http://www.ubuntu.com/server/doc
New release 'natty' available.
Run 'do-release-upgrade' to upgrade to it.

You have new mail.
Last login: Tue May 17 15:31:00 2011 from ta.home
$ sudo su -
# aptitude reinstall console-setup
The following packages will be REINSTALLED:
  console-setup
0 packages upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 493kB of archives. After unpacking 0B will be used.
Get:1 http://au.archive.ubuntu.com/ubuntu/ maverick/main console-setup all 1.34ubuntu15 [493kB]
Fetched 493kB in 0s (2,748kB/s)
Preconfiguring packages ...
(Reading database ... 63042 files and directories currently installed.)
Preparing to replace console-setup 1.34ubuntu15 (using .../console-setup_1.34ubuntu15_all.deb) ...
Unpacking replacement console-setup ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
Setting up console-setup (1.34ubuntu15) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-2.6.35-28-server
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.

#

So working fine under maverick on the server where I encountered the bug. Seems like Wladmir should open a new bug.