Boot into emergency mode on RPi2 with proposed kernel 4.4.0-1043.50 after system update

[Taihsiang Ho]

Hardware: RPi2
Image: http://cdimage.ubuntu.com/ubuntu/releases/16.04/release/ubuntu-16.04-preinstalled-server-armhf+raspi2.img.xz

[Steps to Reproduce]
1. Install the image in an SD card. Use it to boot into the system.
2. Enable the proposed pocket[1] and update the system[2]
3. Reboot

[Expected Result]
The system boots into the system with proposed kernel 4.4.0-1043.50

[Actual Result]
The system boots into emergency mode. SSH connection loses. However, the network is still connected.

[Additional Info]

* Reproducing rate 6 out of 7

* The previous system update works with updated-kernel 4.4.0-1042-raspi2.

* More system information which could reproduce this issue:

==========================================================================
Kernel: 4.4.0-1043-raspi2
Ubuntu 16.04.2 LTS
==========================================================================
==== dpkg -l | grep -e uboot -e flash-kernel -e firmware -e linux-raspi2 -e u-boot =================================
ii flash-kernel 3.0~rc.4ubuntu62.2 armhf utility to make certain embedded devices bootable
ii linux-firmware 1.157.8 all Firmware for Linux kernel drivers
ii linux-firmware-raspi2 1.20161020-0ubuntu1~0.1 armhf RaspberryPi2 GPU firmware and bootloaders
ii linux-raspi2 4.4.0.1043.42 armhf Complete Linux kernel for the BCM2709 architecture.
ii linux-raspi2-headers-4.4.0-1009 4.4.0-1009.10 armhf Header files related to Linux kernel version 4.4.0
ii linux-raspi2-headers-4.4.0-1042 4.4.0-1042.49 armhf Header files related to Linux kernel version 4.4.0
ii linux-raspi2-headers-4.4.0-1043 4.4.0-1043.50 armhf Header files related to Linux kernel version 4.4.0
ii u-boot-rpi:armhf 2016.01+dfsg1-2ubuntu3 armhf A boot loader for Raspberry PI systems
ii u-boot-tools 2016.01+dfsg1-2ubuntu3 armhf companion tools for Das U-Boot bootloader
==========================================================================

==========================================================================
Please refer to this URL to access full dpkg info.
http://paste.ubuntu.com/24006297/
==========================================================================

[1] Add this line in sourcelist
deb http://ports.ubuntu.com/ubuntu-ports xenial-proposed main universe multiverse restricted

[2] sudo apt-get update && sudo apt-get dist-upgrade -y

[Paolo Pisati (p-pisati)]

There's no emergency mode in that image, what you exactly mean for emergency mode?

Can you connect a serial cable or lcd screen and tell us what you see?

[Taihsiang Ho (tai271828)]

Here you go the LCD screenshot of the term "Welcome to emergency mode".

[Thadeu Lima de Souza Cascardo (cascardo)]

Can you log in and run journalctl -xb?

Thanks.
Cascardo.

[Taihsiang Ho (tai271828)]

The following were collected by root in emergency mode:

/var/log/syslog http://paste.ubuntu.com/24006850/
dmesg http://paste.ubuntu.com/24006852/
journalctl -xb http://paste.ubuntu.com/24006861/

[Thadeu Lima de Souza Cascardo (cascardo)]

It looks like there is an oops when udevadm trigger writes "add" to some module uevent. Can you edit /lib/systemd/system/systemd-udev-trigger.service and add --verbose after subsystems?

The line would be like this:

ExecStart=/bin/udevadm trigger --type=subsystems --verbose --action=add ; /bin/udevadm trigger --type=devices --action=add

Then, reboot, and get journal logs from systemd-udev-trigger. I guess journalctl --unit=systemd-udev-trigger might show it.

Thanks.
Cascardo.

[Thadeu Lima de Souza Cascardo (cascardo)]

Hi, Taihsiang Ho.

Can you reproduce the problem, or did it happen only once? That way, we can understand how easy it should be to reproduce, and if there is some kind of race involved.

Thanks.
Cascardo.

[Taihsiang Ho (tai271828)]

journalctl --unit=systemd-udev-trigger verbose log suggested as comment5

[Taihsiang Ho (tai271828)]

I can reproduce this issue. Reproducing rate: 5 out of 5.

This issue could reproduce via the following both ways:

* Download the image and apt-get update to the latest stack.
* apt-get update with an existing system.

[Taihsiang Ho (tai271828)]

I keep rebooting the system. The reproducing rate is now 6 out of 7.

[Taihsiang Ho (tai271828)]

log dumped by
"udevadm trigger --type=subsystems --action=add --verbose"
with 4.4.0-1042-raspi2

http://paste.ubuntu.com/24013536/

[Taihsiang Ho (tai271828)]

Switch to 4.4.0-1042-raspi2 and reboot 10 times. I could always boot into the system successfully.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-raspi2 - 4.4.0-1044.51

---------------
linux-raspi2 (4.4.0-1044.51) xenial; urgency=low

  [ Thadeu Lima de Souza Cascardo ]

  * Release Tracking Bug
    - LP: #1667422

  * Boot into emergency mode on RPi2 with proposed kernel 4.4.0-1043.50 after
    system update (LP: #1665280)
    - Revert "PM / wakeirq: Fix dedicated wakeirq for drivers not using
      autosuspend"
    - Revert "mm, oom: prevent premature OOM killer invocation for high order
      request"
    - Revert "mm, oom: protect !costly allocations some more for
      !CONFIG_COMPACTION"
    - Revert "mm, oom, compaction: prevent from should_compact_retry looping for
      ever for costly orders"
    - Revert "mm: consider compaction feedback also for costly allocation"

  [ Ubuntu: 4.4.0-64.85 ]

  * CVE-2017-6074 (LP: #1665935)
    - dccp: fix freeing skb too early for IPV6_RECVPKTINFO

 -- Thadeu Lima de Souza Cascardo <email address hidden> Thu, 23 Feb 2017 15:44:15 -0300

[Stefan Bader (smb)]

Sorry wrong bug report.