Kernel Panic when 'telinit u', or (re)installing packages

Bug #1269405 reported by Aurel Branzeanu on 2014-01-15
This bug report is a duplicate of:  Bug #1269731: init crashed with SIGSEGV. Edit Remove
40
This bug affects 7 people
Affects Status Importance Assigned to Milestone
eglibc (Ubuntu)
Undecided
Unassigned
upstart (Ubuntu)
Critical
Unassigned

Bug Description

My system, 64-bit Trusty, is throwing kernel panic when running 'sudo dpkg --configure -a'.

dpkg 1.17.1ubuntu1

"--no-log" in grub didn't help.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: upstart 1.11-0ubuntu1
ProcVersionSignature: Ubuntu 3.13.0-2.17-generic 3.13.0-rc7
Uname: Linux 3.13.0-2-generic x86_64
ApportVersion: 2.13.1-0ubuntu1
Architecture: amd64
CurrentDesktop: Unity
Date: Wed Jan 15 14:45:11 2014
ProcKernelCmdline: BOOT_IMAGE=/boot/vmlinuz-3.13.0-2-generic root=UUID=bdeb49ec-f78d-4aa9-b948-d857b8a8fd57 ro quiet splash --no-log vt.handoff=7
SourcePackage: upstart
UpgradeStatus: Upgraded to trusty on 2013-12-20 (26 days ago)
UpstartBugCategory: System
UpstartRunningSessionVersion: init (upstart 1.11)
UpstartRunningSystemVersion: init (upstart 1.11)
modified.conffile..etc.at.deny: [inaccessible: [Errno 13] Permission denied: '/etc/at.deny']
modified.conffile..etc.default.whoopsie: [modified]
modified.conffile..etc.rsyslog.conf: [modified]
modified.conffile..etc.xdg.autostart.pulseaudio.desktop: [modified]
mtime.conffile..etc.default.whoopsie: 2013-12-20T14:42:32.340664
mtime.conffile..etc.rsyslog.conf: 2014-01-15T11:41:20.325384
mtime.conffile..etc.xdg.autostart.pulseaudio.desktop: 2013-06-14T10:22:10.739084

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in upstart (Ubuntu):
status: New → Confirmed
James Hunt (jamesodhunt) wrote :

Hi Aurel,

Do you know if you have any chroots on your system?

To help us diagnose the issue, please boot your system with the following changes to the kernel command-line:

- add "--debug"
- remove "quiet"
- remove "splash"

Then,

- Login via control-alt-f1
- Download get_state.sh from http://people.canonical.com/~jhunt/upstart/scripts/get_state.sh
- Run the script as:
  sudo bash get_state.sh | json_pp > /root/upstart.state && sync

- Attach /root/upstart.state to this bug if possible.

- Force upstart to restart (this may be what is causing the panic):

   sudo telinit u

- Attach a photograph of the screen if an error is shown.

Thanks!

Michael Marley (mamarley) wrote :

I have this same issue, and when I run the get_state.sh script, it causes a kernel panic immediately. There is no stack trace.

Michael Marley (mamarley) wrote :

I tried again and got a different message. I have attached the picture. There is a part of the text that is cut off, but it is just a standard DBUS error message.

TJ (tj) wrote :

Added eglibc since this bug can be triggered by "sudo telinit u", upgrade/reinstall of libc6, and others.

TJ (tj) on 2014-01-15
summary: - Kernel Panic running 'sudo dpkg --configure -a'
+ Kernel Panic when 'telinit u', or (re)installing packages
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in eglibc (Ubuntu):
status: New → Confirmed
Adam Conrad (adconrad) wrote :

Not a glibc issue, and confirmed locally that a glibc upgrade works fine for me. So, likely a local issue with upstart configs on the affected machines.

Changed in eglibc (Ubuntu):
status: Confirmed → Invalid
James Hunt (jamesodhunt) wrote :

I cannot recreate this issue on either 32-bit or 64-bit fully-updated trusty systems. What would be useful is to know if it also affects the Session Init (upstart running as the user) since that would allow debug to be collected without the kernel panicking.

To find out, please can those affected boot to a *desktop* and run the following as a non-root user:

$ get_state.sh | json_pp > upstart.state
$ start re-exec

Then, attach the following to this bug if a crash results:

$HOME/.xsession-errors
$HOME/.cache/upstart/upstart.state

If the Session Init does crash, we might be lucky and get a crash file in /var/crash/.

James Hunt (jamesodhunt) wrote :

Also, could those affected please:

- attach output produced by "sudo initctl list".
- comment if they have added any of their own jobs or modified existing jobs.
- comment if they have any chroots on their system.

Michael Marley (mamarley) wrote :

Here is the output of get_state.sh when running as a non-root user.

Michael Marley (mamarley) wrote :

There was no crash when running get_state.sh as non-root.

I attached the output of initctl list.

One of my systems (the one I am posting all the output from) has no chroots, no modified jobs, and one custom job that runs once during startup. The other system has two chroots, no modified jobs, and quite a few different custom jobs. The bug affects both systems exactly the same way.

Michael Marley (mamarley) wrote :

Oops, here's the file.

James Hunt (jamesodhunt) wrote :

Thanks Michael. Did 'start re-exec' work (appear to do nothing) or fail (crash your desktop session)?

The following isn't a conclusive test, but it will allow us to rule out whether somehow a system job in "unrepresentable" when serialised. To check this, we can copy the system jobs into the session area and see if all those jobs can be serialised like so:

$ mkdir $HOME/.config/upstart/tmp-system-jobs/
$ cp /etc/init/* $HOME/.config/upstart/tmp-system-jobs/
$ get_state.sh | json_pp > upstart.state
$ start re-exec

Don't forget to remove the 'tmp-system-jobs/' directory afterwards.

James Hunt (jamesodhunt) wrote :

Also, it would be extremely useful to know if this issue is repeatable every time. If so, please:

1) $ sudo initctl list > initctl.list.
2) attach that file to this bug before running 'sudo telinit u' / 'sudo get_state.sh' to demonstrate the crash is repeatable.

Thanks!

Michael Marley (mamarley) wrote :

Immediately after I run the get_state.sh command there, KDE krashes and dumps me back at the KDM login screen. The only output in the file is the error message from json_pp indicating that the input piped to it isn't valid JSON.

Yes, the issue is repeatable every time.

Giovanni Pardini (gio.pardini) wrote :

Hi, this bug also occurs to me.
I also tried the instructions from #15, and issuing get_state.sh causes Unity to close, bailing out to the login screen.
The resulting upstart.state file is empty.

The output of "initctl list" is attached.

Asher Wood (zeklandia) wrote :

Here are some pictures of the kernel panic that happens after dpkg tries to configure libc6:amd64.

Asher Wood (zeklandia) wrote :

Here is the output of initctl list.

Steve Langasek (vorlon) on 2014-01-15
Changed in upstart (Ubuntu):
importance: Undecided → Critical
Asher Wood (zeklandia) wrote :

Can we get a workaround or a fix soon, please? I need to be able to use my laptop.

Asher Wood (zeklandia) wrote :

Workaround:

Choose the recovery mode in GRUB (under Advanced Options for Ubuntu) and choose 'Fix broken Packages'.

After using the recovery-option in grub you mentioned above nothing works after reboot.

The gnome-session/GDM is not starting anymore and "telinit u" still crashes the machine.

[UPDATE]

Solved my gnome-shell-issue from above by creating a symbolic link for the libwayland-egl.so.1 :

 # sudo ln -s /usr/lib/x86_64-linux-gnu/mesa-egl/libwayland* /usr/lib

But "telinit u" still crashes the machine!

BTW: The interface of this forum is realy bad and is IMO unuseable - it's also worth a bug-entry.

get_state.sh is causing a kernel panic here too

James Hunt (jamesodhunt) wrote :

Whilst we investigate this issue, you can still run apt-get / dpkg without triggering the bug if you do the following:

$ sudo su -
# mkdir /root/bin
# ln -s /bin/true /root/bin/telinit
# chmod 755 /root/bin/telinit
# export PATH=/root/bin:$PATH
# dpkg --configure -a

When dpkg runs it will use the fake telinit (which does nothing). There are more invasive ways of disabling 'telinit u' but -- until we fix this issue -- as long as you remember to 'export PATH=/root/bin:$PATH' before running apt-get / dpkg, this is the safest I believe.

James Hunt (jamesodhunt) wrote :

This problem is the same as bug 1269731.

Thank you, James, for the workaround!

Michael Marley (mamarley) wrote :

It isn't fixed; this is just a workaround. Programs (especially ones as vital to system operation as upstart) should never crash when fed malformed data.

Any news on this one? I have the same issue, see question https://answers.launchpad.net/ubuntu/+question/247836

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Related questions