Device appears to be off after updating to 15Sept2016's rc-proposed image

[Andrea Bernabei]

Krillin, rc-proposed/bq-aquaris.en r429

Description:
Many people, myself included, are reporting that their devices seemed to have turned off after upgrading to today's rc-proposed

In my case it was an updated from r428 to r429.
The device rebooted to install the upgrade, then at one point I noticed the display was off and not responding to PwrKey.
I had for force shutdown via press-holding PwrKey.

A few community members reported the same problem on Arale (MX4) and Freiza (M10), so it seems to be device-independent.

[Michal Predotka (mpredotka)]

I can confirm this issue on M10 rc-proposed. Update to r189. Not sure if that's relevant but I had double notification about the update like on the screenshot attached.

[dinamic (dinamic6661)]

i think this is the second time that it happened on arale rc-proposed, the phone takes 10+minutes to boot. the boot animation (...) after a while appears frozen and then the screen turns off

[dinamic (dinamic6661)]

maybe... don't turn off the screen during boot

[Pat McGowan (pat-mcgowan)]

anyone have a syslog from the first boot after install?
just updated 415 to 421 with no issue on mx4

[Andrea Bernabei (faenil)]

nope sorry, too much spam in syslog, it just keeps the last 5-10 mins :/
I had just woken up when it happened, forgot to save it straight away after the fact :)

[Pat McGowan (pat-mcgowan)]

I got this on an M10. The log shows ubuntu-location and unity8 core dump once, then unity8-dash and maliit-server repeatedly dumping core. A reboot fixed it. The log contains a "failed to write core dump" for each.

[Andrea Bernabei (faenil)]

Community is reporting the same problem after updating M10 and E5

See https://lists.launchpad.net/ubuntu-phone/msg22369.html and the "Ubuntu App Dev" Telegram group

[Anupam (anupam207)]

Same with my E5, stable channel

[Peter Bittner (peter-bittner)]

Here's my report. (TL;DR My phone is alive!)

I've updated my bq Aquaris E5 from OTA-12 to OTA-13 today (always stable channel since I bought the device). It was not attached to electric power, but the battery was charged at about 95% when I started, so I had no worries. It turned off the screen after rebooting, and then it had the screen off being unresponsive. No power button could awaken it, not even a long-press. I was sure it was bricked. I attached it to the power cord, still no reaction, neither for charging nor for booting. Then after a few minutes it made a light bleep and it was alive again.

[GTriderXC (gtriderxc)]

It happened to me on E4.5 krillin and M10cooler. Devices seemed to turn off and didn't react to nothing like short or long pressing a power button. Device behaves as if the battery was dead. The dislay backed to life in both devices after connecting the battery charger.

[Kugi Eusebio (kugi-igi)]

I got this too on my bq E5 HD when I updated to OTA13.
After the update progress bar completed, the screen just turned off.
Since I read about the issue already, I decided not to do the same which is to do hard reboot or long press of the power button.
As I read here, I connected my phone to power then the screen turned on for a couple of seconds but it was blank and turned off again.
I pulled the plug then that's when my phone completely rebooted.

This issue seems to be encountered by many and I think this should have been captured by the testers.

[Pat McGowan (pat-mcgowan)]

Could you attach /var/log/upstart/apparmor.log.n created during the upgrade

[Antoni (wynajem-pn)]

I just updated M10 and had exactly the same symptoms as in comment #11. I attach apparmor.log as requested. Hope it helps. I you want some more info just ask.

[Louis Holbrook (nolash74)]

Also happened on my Aquaris E4.5, switched off and nothing happened for half an hour. First I couldn't start it with long press, but finally it came on.

I've attached the requested syslog

[Antoni (wynajem-pn)]

syslog from M10.

[Jean-Baptiste Lallement (jibel)]

I reproduced on krillin, plugging/unplugging a power supply twice makes it boot.

[Marcin Wójcik (masheleni)]

It happened to me too when I installed the update this morning.

I have a BQ E4.5.

The update installed and after the install bar completed, the system rebooted.
It sat on the white BQ logo screen for a bit and then the screen went dim.
Tapping on the screen had no effect.

Shortly after it dimming, the screen turned off.
It did not respond to any taps or to any presses of the power key.

At the time this happened, my phone was 60% charged and was on WiFi, connected to the internet.
I pressed the power key down for 1 - 2 seconds and still nothing, as I thought it may have turned off.

Then I was concerned that it might have still been updating and that I might have just interrupted the update process.

I then plugged the phone into power. The screen then went on with a black background for 1 second before turning off again.

After waiting for a few seconds, I tapped on the power button and the screen turned on, displaying the Ubuntu loading screen.

It seems like OTA-13 is now installed, but I am not sure if it installed correctly.

[Jean-Baptiste Lallement (jibel)]

The OTA is installed correctly. It's the first boot sequence after the installation that doesn't work correctly when the phone is not connected to a power supply apparently. Next boots are fine and so are next upgrades. Still investigating...

[Jean-Baptiste Lallement (jibel)]

Here are all the logs I could grab after reproducing this issue.

[Jean-Baptiste Lallement (jibel)]

excerpt of syslog corresponding to first boot after upgrade.

From syslog there is a 2 minutes gap in the syslog timestamp compared to the kernel timestamp
Sep 21 07:38:19 ubuntu-phablet kernel: [ 74.454170]PM: suspend exit 2016-09-21 05:38:19.199271077 UTC
Sep 21 07:38:19 ubuntu-phablet kernel: [ 74.454218]PM: suspend entry 2016-09-21 05:38:19.199318385 UTC
Sep 21 07:40:05 ubuntu-phablet kernel: [ 74.454229]PM: Syncing filesystems ... done.
Sep 21 07:40:05 ubuntu-phablet kernel: [ 74.465804][pm_notifier_call_chain]: there are 4 notify callbacks, event = 3

From unity8.log.1.gz
[2016-09-21 07:40:52.105060] mirserver: Starting

[Jean-Baptiste Lallement (jibel)]

I suspect that the phone suspends during boot if the boot takes too long and because unity8 is not started there is no way to wake it up apart from plugging a power supply.

[Jean-Baptiste Lallement (jibel)]

apparmor activity from syslog

[Jean-Baptiste Lallement (jibel)]

My theory is:
- A change in apparmor triggers a recompilation of the profiles on upgrade
- repowerd starts early in the boot process and before u-s-c and unity8 (in the log above it starts at 7:37 and u-s-c at 7:40)
- After the upgrade, the system boots, repowerd starts, the compilation of the profiles starts too (at 7:36)
- The compilation of the profiles takes more than 1 minute (it stops at 7:37:09 according to the logs, 1 minute after it started)
- The device suspends when it reaches the suspend timeout.
- Because u-s-c is not started, there is no way to wake the device up with the power button.
- Plugging a power supply resumes the device and it proceeds with the boot.

For anyone with this problem, the easiest workaround it to connect the device to a power source, otherwise long press the power button, it should be harmless but it'll interrupt apparmor. In any case, the OTA is correctly applied.

[Matthias Apitz (gubu)]

I updated 2 devices BQ E4.5 last nigth; both staid black after the installation process and I have to press ~20 seconds the power button to make them reboot; lucky that they came up; I will attach the syslog and the apparmar.log file; the latter is full of error messages like:

ERROR: Could not parse click manifest. Skipping 'com.ubuntu.telegram_sctelegram_2.2.24.0.json'
ERROR: Could not parse click manifest. Skipping 'com.ubuntu.telegram_telegram_2.2.24.0.json'
ERROR: Could not parse click manifest. Skipping 'com.canonical.scopes.songkick_songkick_1.1.0.json'
ERROR: Could not parse click manifest. Skipping 'twitter.canonicalpartners_twitter_2.6.json'

[Petras (klavishas)]

m10 cooler here, I had usb charger plugged while updating. After the update screen was dark (I could notice that backlight was actually on, and didnt panicked and didnt hold any buttons, just pressed once to check). I remebered that it was intended at ota13 powerd overhaul, so I thought that maybe it is due to charger connected and such scenario for update were not tested. I disconected the charger and my login screen apeared on.Thats my story.

[Davilinux (davidfl)]

Hello.

I have a BQ E4.5

Yesterday updated my phone with the last OTA-13 and, unfortunately, with the same results after flashing... that means I had to plug the power cord to boot the device on, because it seems to be bricked.

The worst: The battery is consuming a lot of power.
Usually, I have to connect to the power each 3 days more or less, but now (with OTA-13) between 11:00AM and 18:30PM of today, the battery has consumed 50% of his energy and I'm sure this night I'll have to connect again if I want to have it alive.

I attached the syslog excerpt with the power consumption data.

Thanks in advance.

David.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Pat McGowan (pat-mcgowan)]

@Davilinux can you open a new bug for that issue and if it persists run top to see what if anything is consuming the cpu

[Pat McGowan (pat-mcgowan)]

@alf can you compare what powerd used to do with what repowerd is doing now in this scenario

[Alexandros Frantzis (afrantzis)]

> can you compare what powerd used to do with what repowerd is doing now in this scenario

Based on jibel's theory from comment #23:

With powerd the timeout logic was in USC, so if USC hasn't started (e.g. because of apparmor recompilation in this case?) the device won't suspend.

With repowerd all the timeout logic is in repowerd itself; USC is used only as a source of events (e.g. power button, touches). In this case, since repowerd doesn't get any requests to disallow suspending, it correctly suspends after the normal timeout.

I think a sensible solution would be to disallow suspending until at least USC (and possibly unity8) is up and running. This could be achieved by acquiring and releasing a suspend blocker from repowerd at the init system level, with a prohibit_suspend_at_boot upstart/systemd job: on "repowerd started" acquire a suspend blocker, on "usc/unity8 started" release the blocker.

Even without the suspend issue, the initial delay after first boot due to apparmor recompilation or whatnot does not provide a good experience, since there is no visual indication that something is going on for a significant amount of time. This brings up the following questions:

0. Why do we need to recompile the profiles at first boot?
1. Could we start USC before the recompilation and other time consuming tasks start, so at least the user would be able to see the dots animating (and incidentally this would also solve the suspend issue)?

[Davilinux (davidfl)]

Done, @pat-mcgowan.

URL --> https://bugs.launchpad.net/canonical-devices-system-image/+bug/1626672

I'll have a look to the top information.

Thanks for your attention.

Regards.

David.

[danilo (tatonilo)]

No need to annoy you with the same operatins and events list.
Take as my reference the experience of Marcin.
Sane events sequence, same fear, same good end.
Thanks for further investigating and to never repeat it again...Ok?!?!.
;-)
Thanks and
Regards.
D

[Pat McGowan (pat-mcgowan)]

@alf per comment #31

we need to regenerate the compiled profiles for installed apps when new rules are provided. This needs to be done before any apps can be launched, so unity8 cannot be started. We could consider starting usc but unsure the dependencies there.

Can repowerd simply hold a lock until the screen dbus service is connected?

[Alexandros Frantzis (afrantzis)]

> Can repowerd simply hold a lock until the screen dbus service is connected?

(I am assuming you are referring to a display-on lock, which also disallows suspends).

repowerd could hold a lock, but, IMO:

1. repowerd is the wrong place to put this logic. repowerd shouldn't have to know, even implicitly, about the boot time dependencies. As an example of why this is fragile, if we decide to move USC before or in parallel with apparmor reprofiling (which I suggest we do), this logic will stop having the desired effect. It would be much better to use a suspend block during the whole boot process (repowerd started -> unity8 started) using the init system facilities, as described in comment #31.

2. this lock by itself would only improve the user experience very slightly, since it would brighten the backlight but the screen would still remain black and unresponsive to power keys for some time until USC starts.

I think the best solution from a UX and cleanliness perspective would be:

1. Start USC before or in parallel with apparmor reprofiling, so the user gets some visual indication that the system is working and can at least turn the screen on/off. This is optional, but I think improves UX a lot.
2. Acquire a suspend blocker (using an init job) when repowerd starts, release when unity8 (not USC) has started or failed to start.

I'll try to find a way to recreate the "profiles need recompilation" scenario without flashing a new image, so we can test easily. I will provide a prototype for the proposed solution for people to try out.

[Pat McGowan (pat-mcgowan)]

You can trigger the compilation by
sudo rm /var/cache/apparmor/*
sudo rm /var/lib/apparmor/profiles/*

[Alexandros Frantzis (afrantzis)]

> You can trigger the compilation by
> sudo rm /var/cache/apparmor/*
> sudo rm /var/lib/apparmor/profiles/*

Thanks Pat. I also found another way to do the same, tricking the apparmor boot script into thinking this is the first it runs:

sudo rm /var/lib/apparmor/profiles/.apparmor.md5sums

[Alexandros Frantzis (afrantzis)]

I have attached two upstart .conf files implementing a fix that keeps the display on until the phone has booted. In particular it acquires a display-on lock when repowerd has started and releases it when lightdm has started. After the display-on lock is released the display stays on for 30s-1m (depending on the user timeout).

To test this, copy *both* attached files into /etc/init/ on the device (overwrite the existing repowerd.conf, keep-display-on-during-boot.conf is a new file) and reboot.

If you want to try the scenario described in this bug, do:

sudo rm /var/lib/apparmor/profiles/.apparmor.md5sums

before rebooting.

Please try it out and let me know what you think, so I can push the fixes to the respective packages.

[Alexandros Frantzis (afrantzis)]

Note that the fix above doesn't implement my earlier suggestion to start USC before apparmor finishes reprofiling. Unfortunately the way the job dependencies are structured makes this change very difficult.

I would suggest that we clearly and prominently mention in our release notes (and also perhaps in phone upgrade UI?) that the first boot after an upgrade may take a significantly longer time than usual, so the users expect the delay.

[Pat McGowan (pat-mcgowan)]

I tested it and it is much better although obviously still lacks any indication things are happening. Along with a note for people to expect it to take some time on the boot screen should be ok until we can address it further.

[Alexandros Frantzis (afrantzis)]

The repowerd part of the fix has landed in repowerd 2016.10. Now waiting for the upstart script addition to land: https://code.launchpad.net/~afrantzis/ubuntu-touch-session/keep-display-on-during-boot

[Launchpad Janitor (janitor)]

This bug was fixed in the package repowerd - 2016.10+16.10.20161007-0ubuntu1

---------------
repowerd (2016.10+16.10.20161007-0ubuntu1) yakkety; urgency=medium

  * New 2016.10 release
    - Fix typo in brightness config value (LP: #1618391)
    - Update repowerd-cli help text
    - Handle SIGTERM gracefully in repowerd-cli
    - Wait for the repowerd DBus API to be accessible before marking
      the upstart job as started (LP: #1623853)
  * 2016.10 release

 -- Alexandros Frantzis <email address hidden> Fri, 07 Oct 2016 11:50:44 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-touch-session - 0.108+16.10.20161014-0ubuntu1

---------------
ubuntu-touch-session (0.108+16.10.20161014-0ubuntu1) yakkety; urgency=medium

  [ Alexandros Frantzis ]
  * Keep the display on during boot (LP: #1623853) (LP: #1623853)

 -- Łukasz Zemczak <email address hidden> Fri, 14 Oct 2016 07:48:51 +0000