plymouthd spinning at 100% CPU after I log in

[Daniel van Vugt]

plymouthd spinning at 100% CPU. Probably triggered by use of this PPA:
https://launchpad.net/~mir-team/+archive/system-compositor-testing
although that does not change Plymouth.

ProblemType: Bug
DistroRelease: Ubuntu 13.10
Package: plymouth 0.8.8-0ubuntu7
ProcVersionSignature: Ubuntu 3.9.0-6.13-generic 3.9.6
Uname: Linux 3.9.0-6-generic x86_64
ApportVersion: 2.10.2-0ubuntu1
Architecture: amd64
Date: Tue Jun 18 15:06:38 2013
DefaultPlymouth: /lib/plymouth/themes/ubuntu-logo/ubuntu-logo.plymouth
ExecutablePath: /sbin/plymouthd
InstallationDate: Installed on 2013-06-11 (6 days ago)
InstallationMedia: Ubuntu 13.10 "Saucy Salamander" - Alpha amd64 (20130611)
MachineType: LENOVO 4286CTO
MarkForUpload: True
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.9.0-6-generic root=UUID=25c27832-0efa-4fc4-8a14-41af68d008dc ro quiet splash vt.handoff=7
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
ProcFB:
 0 inteldrmfb
 1 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.9.0-6-generic root=UUID=25c27832-0efa-4fc4-8a14-41af68d008dc ro quiet splash vt.handoff=7
SourcePackage: plymouth
TextPlymouth: /lib/plymouth/themes/ubuntu-text/ubuntu-text.plymouth
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/11/2013
dmi.bios.vendor: LENOVO
dmi.bios.version: 8DET68WW (1.38 )
dmi.board.asset.tag: Not Available
dmi.board.name: 4286CTO
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvr8DET68WW(1.38):bd04/11/2013:svnLENOVO:pn4286CTO:pvrThinkPadX220:rvnLENOVO:rn4286CTO:rvrNotAvailable:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 4286CTO
dmi.product.version: ThinkPad X220
dmi.sys.vendor: LENOVO

[Daniel van Vugt (vanvugt)]

Spinning where, you ask?

root@think:~# ./dstack plymouthd
Process ID: 186
Program name: plymouthd
Program path: /sbin/plymouthd
Thread count: 1

81 ../sysdeps/unix/syscall-template.S: No such file or directory.
Dstack of plymouthd pid 186 (/sbin/plymouthd)

[Switching to thread 1 (Thread 0x7f7d99de7740 (LWP 186))]
#0 0x00007f7d994e94b3 in __epoll_wait_nocancel ()
    at ../sysdeps/unix/syscall-template.S:81
81 in ../sysdeps/unix/syscall-template.S
#0 0x00007f7d994e94b3 in __epoll_wait_nocancel ()
    at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f7d999d082a in ply_event_loop_process_pending_events ()
   from /lib/x86_64-linux-gnu/libply.so.2
#2 0x00007f7d999d11c0 in ply_event_loop_run ()
   from /lib/x86_64-linux-gnu/libply.so.2
#3 0x0000000000404bfd in ?? ()
#4 0x00007f7d99410ea5 in __libc_start_main (main=0x4042f0, argc=3,
    ubp_av=0x7fff00df4b78, init=<optimised out>, fini=<optimised out>,
    rtld_fini=<optimised out>, stack_end=0x7fff00df4b68) at libc-start.c:260
#5 0x0000000000405879 in ?? ()
A debugging session is active.

 Inferior 1 [process 186] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]
root@think:~# ./dstack plymouthd
Process ID: 186
Program name: plymouthd
Program path: /sbin/plymouthd
Thread count: 1

81 ../sysdeps/unix/syscall-template.S: No such file or directory.
Dstack of plymouthd pid 186 (/sbin/plymouthd)

[Switching to thread 1 (Thread 0x7f7d99de7740 (LWP 186))]
#0 0x00007f7d994e94b3 in __epoll_wait_nocancel ()
    at ../sysdeps/unix/syscall-template.S:81
81 in ../sysdeps/unix/syscall-template.S
#0 0x00007f7d994e94b3 in __epoll_wait_nocancel ()
    at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f7d999d082a in ply_event_loop_process_pending_events ()
   from /lib/x86_64-linux-gnu/libply.so.2
#2 0x00007f7d999d11c0 in ply_event_loop_run ()
   from /lib/x86_64-linux-gnu/libply.so.2
#3 0x0000000000404bfd in ?? ()
#4 0x00007f7d99410ea5 in __libc_start_main (main=0x4042f0, argc=3,
    ubp_av=0x7fff00df4b78, init=<optimised out>, fini=<optimised out>,
    rtld_fini=<optimised out>, stack_end=0x7fff00df4b68) at libc-start.c:260
#5 0x0000000000405879 in ?? ()
A debugging session is active.

 Inferior 1 [process 186] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]
root@think:~# ./dstack plymouthd
Process ID: 186
Program name: plymouthd
Program path: /sbin/plymouthd
Thread count: 1

81 ../sysdeps/unix/syscall-template.S: No such file or directory.
Dstack of plymouthd pid 186 (/sbin/plymouthd)

[Switching to thread 1 (Thread 0x7f7d99de7740 (LWP 186))]
#0 0x00007f7d994e94b3 in __epoll_wait_nocancel ()
    at ../sysdeps/unix/syscall-template.S:81
81 in ../sysdeps/unix/syscall-template.S
#0 0x00007f7d994e94b3 in __epoll_wait_nocancel ()
    at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f7d999d082a in ply_event_loop_process_pending_events ()
   from /lib/x86_64-linux-gnu/libply.so.2
#2 0x00007f7d999d11c0 in ply_event_loop_run ()
   from /lib/x86_64-linux-gnu/libply.so.2
#3 0x0000000000404bfd in ?? ()
#4 0x00007f7d99410ea5 in __libc_start_main (main=0x4042f0, argc=3,
    ubp_av=0x7fff00df4b78, init=<optimised out>, fini=<optim...

[Daniel van Vugt (vanvugt)]

WORKAROUND:
sudo plymouth deactivate

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:~mir-team/lightdm/unity at revision 1619, scheduled for release in lightdm, milestone Unknown

[Daniel van Vugt (vanvugt)]

Verified Fix Released in lightdm 1.7.2bzr1621saucy0.57
https://launchpad.net/~mir-team/+archive/system-compositor-testing

[Daniel van Vugt (vanvugt)]

The fix in lightdm is enough right now. No need to fix plymouth, especially if a Mir-native-plymouth would never have the bug anyway.

[Christopher Townsend (townsend)]

I've been investigating the plymouth part of this and what I have discovered is that plymouth is waiting on an event from lightdm on VT 7(tty7) to tell it to stop. However, with Xmir, as far as I can tell, nothing is happening onVT 7, so the event never occurs causing plymouth to spin out of control.

I wonder if there is some way to force Xmir to use VT 7 instead of a different VT????

[Steve Langasek (vorlon)]

On Thu, Jul 11, 2013 at 08:26:31PM -0000, Christopher Townsend wrote:
> I've been investigating the plymouth part of this and what I have
> discovered is that plymouth is waiting on an event from lightdm on VT
> 7(tty7) to tell it to stop. However, with Xmir, as far as I can tell,
> nothing is happening onVT 7, so the event never occurs causing plymouth
> to spin out of control.

No, it has nothing to do with signals on any VT. It's waiting for lightdm
to call 'plymouth quit [--retain-splash]', which is not VT-dependent. Is
lightdm failing to call 'plymouth quit' correctly before starting Mir?

[Christopher Townsend (townsend)]

After some more investigation, I'm not sure what I said is exactly true, but there is something that is (obviously) not telling plymouth to quit with this setup.

[Daniel van Vugt (vanvugt)]

Oops, I forgot to post my original findings here. You might find them helpful...

plymouthd is spinning on an fd of "/dev/tty7" with errno==EIO. Waiting for the "plymouth deactivate" to properly complete (hence drmDropMaster), avoids the bug.

So it looks like plymouth's DRM code is not very clever with its error handling. Once it starts getting EIO it should probably give up. Instead, it doesn't understand EIO and retries constantly.

Also, enhancing plymouth so that "plymouth --wait" works with the deactivate command would have been very helpful. Right now, plymouth --wait does not wait for the deactivate to complete.

[Christopher Townsend (townsend)]

Thanks Daniel.

I have also discovered that plymouthd is crashing sometime during boot/log in (not sure exactly when). I built a version of plymouth with symbols and have a stack trace that shows that it's falling over at on_deactivate w/ a segfault. So I think when plymouthd crashes, a new plymouthd is spawned, but the deactivate from lightdm has already occurred, so plymouthd just spins waiting on a deactivate to occur.

Since I have a good stack trace, I think trying to figure out why it's segfaulting is possible. Also, as you point out, perhaps putting in some robustness for EIO may be helpful as well.

[Steve Langasek (vorlon)]

> So I think when plymouthd crashes, a new plymouthd is spawned,

The upstart job never does this.

[Christopher Townsend (townsend)]

Sigh...I made a lightdm package that reverted this hack to see the 100% CPU state, but didn't realize that it got reverted from the latest Mir updates today. Anyways, as Steve pointed out, plymouth is *not* respawned in this case, so my last update is bogus for this particular bug.

However, it's strange that plymouth segfaults in the hacked lightdm case. Also, if I send a "plymouth deactivate", the spinning plymouth segfaults the same way, but that is a different bug...

[Christopher Townsend (townsend)]

Ok, after a couple of false starts on root causing this, I believe I found the issue causing the spin:)

- In the Xmir startup, nothing is calling "plymouth quit", so that is why plymouthd keeps running. I know this because I enabled logging in both Xmir and X only, and in X only, I can see a call to "plymouth quit" whereas in the Xmir version, no such call is logged.
- Fir x only, the call to "plymouth quit" comes from a signal from the X server indicating that it has started. This is in the lightdm source in src/xserver-local.c in the got_signal_cb() function.
- In Xmir, the compositor is responsible for deactivating/quitting plymouth.
- The compositor handling of plymouth in lightdm is a bit lacking on the quitting part.

To test my findings, I created a package that reverted the changes in lp:~vanvugt/lightdm/fix-1192051. I then added some code in seat-unity.c that handles "plymouth quit". In my test, plymouth is no longer running on log in and no crash of plymouthd is observed.

I'm going to propose this change to the Mir version of lightdm and try to see about getting it upstream into lightdm. Also, plymouth is really not a fault here so I'll probably remove it from this bug.

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:~mir-team/lightdm/unity at revision 1636, scheduled for release in lightdm, milestone Unknown

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:lightdm at revision None, scheduled for release in lightdm, milestone Unknown

[Launchpad Janitor (janitor)]

This bug was fixed in the package lightdm - 1.7.6-0ubuntu1

---------------
lightdm (1.7.6-0ubuntu1) saucy; urgency=low

  * New upstream release:
    [ 1.7.6 ]
    - Restore greeter hints that were regressed in 1.7.5.
    - Don't run greeters through session wrapper - regression in 1.7.5
    [ 1.7.5 ]
    - Quit Plymouth correctly when using the unity seat type (LP: #1192051)
    - Release the VT when the system compositor fails to start
    - Load sessions and greeters from /usr/share/lightdm/sessions and
      /usr/share/lightdm/greeters. The existing directories are checked
      if the sessions are not in these directories.
    - Refactor the Display class so that it merges with the Seat class
    - Support running the greeter and session in different display servers
      instead of re-using the same one during a login.
    - Add more regression tests
    - Documentation fixes
 -- Robert Ancell <email address hidden> Mon, 22 Jul 2013 17:13:19 +1200

[Damiön la Bagh (kat-amsterdam)]

There is a regression of this bug in 16.04.4LTS

Linux computer 4.4.0-119-generic #143-Ubuntu SMP Mon Apr 2 16:08:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  388 root 20 0 201472 100776 8516 R 95,7 2,9 19:57.46 plymouthd

ii liblightdm-gobject-1-0:amd64 1.18.3-0ubuntu1.1 amd64 LightDM GObject client library
ii lightdm 1.18.3-0ubuntu1.1 amd64 Display Manager

[Damiön la Bagh (kat-amsterdam)]

I've attached an apport report of the runaway process

[Damiön la Bagh (kat-amsterdam)]

I found the conflict with the eGalax Touch Daemon driver.
I moved the driver startup from /etc/init.d/eGtouch.sh
to userspace as I don't need the touchscreen at login.

The strange thing is I have 17 machines with this configuration and only one is exhibiting this behaviour with plymouthd

[Damiön la Bagh (kat-amsterdam)]

Didn't help after all. Plymouth is still using 100% CPU on reboot.

Now trying all the suggestions in
https://bugs.launchpad.net/ubuntu/+source/ubuntu-gnome-default-settings/+bug/1536771

Disabled plymouth in Grub and dpkg-reconfigured it.

[Daniel van Vugt (vanvugt)]

This bug was closed 5 years ago.

If you're experiencing any issues at all today then please open a new bug.