unresponsive shell including power button doesn't blank/unblank display

[Pat McGowan]

Note: based on descriptions we spawned bug 1495871 to cover cases where the power button still works, this bug will only be for cases where the ui input is frozen as well as the power button being unresponsive

RC proposed MX4 r100

This happened twice in the last two days
Receive an incoming text
Screen turns on and notification is shown
notification goes away
try to interact with the phone and nothing works
another text message is received and the notification is displayed

Adding incoming calls as also suspected to be triggering the issue

Note this has been impossible to reproduce, and in use only happens very rarely

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alexey Balmashnov (a.balmashnov)]

Observed similar issue on BQ E4.5., r25 (after OTA6 update).

Dialer locked UI.

Happened after dialing back via activation of dialer from missed call notification (without unlocking the phone). Dialer remains on the foreground and phone UI seems to be stuck. At the same time, icons in the notification area are being updated (clock, battery, network status), but there seem to be no way to access notification area or unlock the phone.

[Alexey Balmashnov (a.balmashnov)]

Attaching unity8.log as advised by Pat McGowan.

[Pat McGowan (pat-mcgowan)]

not sure which component this really is, seems like focus is getting confused

[Michał Sawicz (saviq)]

Indeed, we need to track where is the input lost. When you experience it again, can you please check if volume buttons still work? Unity8 isn't stuck altogether because you can see notifications and indicators updating, so somehow no input reaches unity8.

I wonder if unity-system-compositor could be involved.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alexey Balmashnov (a.balmashnov)]

Actually, I tried buttons, there was no visible reaction on the presses:
* volume buttons (even attempted to get a screenshot, but it is not present
on the phone, so it did not work)
* power button. could not even reboot the phone (or maybe have not waited
long enough? pressed on it for about a minute), had to do reboot via shell

The only part that still was updating on the screen - notification area.
Noticed, because charging was indicated upon connection of the phone to the
PC.

On Fri, Sep 4, 2015 at 10:59 AM, Launchpad Bug Tracker <
<email address hidden>> wrote:

> Status changed to 'Confirmed' because the bug affects multiple users.
>
> ** Changed in: unity-system-compositor (Ubuntu)
> Status: New => Confirmed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1491566
>
> Title:
> Greeter and edges not responsive after an incoming SMS
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/canonical-devices-system-image/+bug/1491566/+subscriptions
>

[Michał Sawicz (saviq)]

We have not been able to reproduce this yet, I'm afraid. We're looking into ways of debugging this "remotely", we'll be preparing a silo with more debugging enabled for unity8 and unity-system-compositor to pinpoint the problem.

In the mean time, if you find yourself in such a state, please come to #ubuntu-unity on freenode and we'll try and see what can be done.

[Michał Sawicz (saviq)]

The two suspects here now are: Qt's input mechanism getting stuck or unity8 losing focus in unity-system-compositor.

Daniel's preparing a set of steps to enable input debugging, if you saw this issue and willing to help, please follow them and report back with the requested logs.

[Daniel d'Andrada (dandrader)]

Steps to get input logging from unity-system-compositor and unity8:

- Apply usc-wrapper_add_logging.diff to /usr/share/ubuntu-touch-session/usc-wrapper
- Apply unity8_conf_add_logging.diff to /usr/share/upstart/sessions/unity8.conf
- phablet@ubuntu-phablet:~$ sudo restart lightdm

Then check logs at:
/var/log/lightdm/unity-system-compositor.log
/home/phablet/.cache/upstart/unity8.log

You can check then live with:
phablet@ubuntu-phablet:/var/log/lightdm$ sudo tail -f unity-system-compositor.log
phablet@ubuntu-phablet:~/.cache/upstart$ tail -f unity8.log

[Alexandros Frantzis (afrantzis)]

> * power button. could not even reboot the phone (or maybe have not waited
> long enough? pressed on it for about a minute), had to do reboot via shell

Power button not responding indicates a problem at the level of USC (since the power button is handled internally by USC). A backtrace of USC when the phone is in this state could be helpful:

$ sudo gdb -p $(pidof unity-system-compositor)
...
(gdb) thread apply all bt full

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Enabling input logging in unity-system-compositor" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Alberto Mardegan (mardy)]

I've seen this happening also after notifications of incoming Telegram messages. My impression is that this bug happens if I press the power button to wake up the phone immediately after the notification has arrived. And I think it has been happening since OTA-5, not just with the last one.

[Pat McGowan (pat-mcgowan)]

This time a different set of symptoms but seems related
Incoming call rings, but there is no notification to accept the call
Inputs do not work
I push power twice to turn scren off and on, then twice again, and the notification showed up so I could answer the call

[Pat McGowan (pat-mcgowan)]

Unity8 log at the time of the problem with debug enabled
Unfortunately u-s-c logs do not get retained over boot so that one is gone
In order to debug this we should change that behavior since getting into the phone usually requires you to open the greeter but with greeter frozen you need to reboot. I have since enabled adb access without unlock but rolling the logs is preferred.

[Pat McGowan (pat-mcgowan)]

Unity8 log at the time of the problem with debug enabled
Unfortunately u-s-c logs do not get retained over boot so that one is gone
In order to debug this we should change that behavior since getting into the phone usually requires you to open the greeter but with greeter frozen you need to reboot. I have since enabled adb access without unlock but rolling the logs is preferred.

[Pat McGowan (pat-mcgowan)]

I note two crash files from the day before, pulseaudio and unity-system-compositor-spinner, so I wonder if they prevented seeing a newer crash saved at the time I noticed the issue.

[JoiHap (astronomy)]

Hi. I've just run into this issue and I am unable to reboot the phone and it is fully charged. I do not have the phone in developer mode. (Battery is non-removable right?). I've installed phablet-shell but what steps should I follow?

[Gerry Boland (gerboland)]

Unsure if related, but if I mash the vol up/down keys on arale a bit, it makes unity8 freeze

[JoiHap (astronomy)]

"power +sound up + sound down" buttons simultaneously. (sorry for the noise on the bug).

[Dylan Coakley (dylan-coakley)]

Please fix this one. I use my Ubuntu phone every day and texting quite a bit. Meaning this bug happens often.. meaning I have to restart my phone often.

It seems from my experience that the phone gets slower and slower the more you text until it locks up completely. After restarting everything is much faster.

[Daniel d'Andrada (dandrader)]

@pat the important thing to know in unity8 is whether it still prints entries like the following in the log when you touch the screen after it gets into this unresponsive state:

"""
qtmir.mir.input: Received MirTouchInputEvent((id=586,action=up,x=586.491,y=468.756))
qtmir.mir.input: Sending to Qt (id=586,state=released,normalPosition=(0.509107,0.244144))
"""

[Daniel d'Andrada (dandrader)]

Still can't reproduce it, by the way. So help on that is still welcome and needed

[Alexey Balmashnov (a.balmashnov)]

Wanted to apply patches from #12 and #13, but files are read-only. Looks like I miss some knowledge. Any pointers are welcome.

[kevin gunn (kgunn72)]

@a.balmashov full disclosure, anytime you make your phone writable and helping with stuff like this, you might lose some of your data or need to reflash. _only_ if you are ok with that should you attempt. if you are, then you can turn on developer mode in system settings (under "about this phone") then you can log on to the phone shell via adb (or phablet-shell) and do sudo touch /userdata/writable_image
if you wish to keep your phone safe, that's completely understandable too, thanks for the willingness to help.

[Pat McGowan (pat-mcgowan)]

Just got the issue again, debug logging should be enabled

[Pat McGowan (pat-mcgowan)]

stack trace from u-s-c

[Pat McGowan (pat-mcgowan)]

While the screen is unresponsive, nothing is logged to either log file when touching the screen.
I do see the process mtk-tpd start using cpu in top.

A restart of lightdm returns it to the normal state

[Daniel d'Andrada (dandrader)]

This means that while the device was unresponsive neither unity-system-compostior nor qtmir ( in unity8 process) got any input events at all.

So this rules out at least qtmir and unity8.

I think the next step is to directly monitor the evdev files to see if anything at all comes out of them while the device is in this unresponsive state.

[Daniel d'Andrada (dandrader)]

To monitor an evdev file you have to apt-get install evemu-tools and then run (on MX4):
$ sudo evemu-record /dev/input/event1

hint: to ensure that you're monitoring the correct event file you can check them all with:
$ sudo evemu-describe

[Pat McGowan (pat-mcgowan)]

In the spirit of wag

in the stack I see the following in threads:

usc::MirScreen::set_screen_power_mode(MirPowerMode, PowerStateChangeReason) ()
usc::MirScreen::keep_display_on_temporarily()
usc::MirScreen::dimmer_alarm_notification()

is it possible a couple of these threads deadlock due to an unlucky timing of events, like the screen dimming as an sms comes in

[Alexandros Frantzis (afrantzis)]

My take on the backtrace is that the set_screen_power_mode request hangs somewhere in Mir, and since it holds the lock, it blocks both dimmer_alarm_notification and keep_display_on_temporarily (which is a result of user input). If my assumption is correct, we need to find why and where in Mir we block, so it would be great if we had debug symbols for mir (and libstdc++ and libc) for the USC backtrace.

[Alexandros Frantzis (afrantzis)]

I have been continuing my efforts to reproduce, but without any success so far. I have created a script to automate the process which I am sharing here in the hope that someone else will have more success with it.

The script either uses simulated sms messages (powerd receives the SMS event and tells USC to turn the screen on) or talks directly to USC to turn the screen on. Just uncomment the line for the method you prefer (and comment the other). For the simulated message approach to work you need to setup the system properly, read: https://wiki.ubuntu.com/Touch/Testing/PhoneAndMessagingSimulation .

The scripts include some reasonable sleeps to let things settle. Increase them or decrease them to simulate different scenarios better.

I invoke the script with:

while true; do sh bug_1491566.sh; done

Note: the slide gesture doesn't always slide in the right direction, not sure why, but in any case you can see some effect on screen

[Alexandros Frantzis (afrantzis)]

Note: For the slide gesture to work you need the python3-evdev package.

[Alexandros Frantzis (afrantzis)]

> Note: the slide gesture doesn't always slide in the right direction, not sure why, but in any case you can see some effect on
screen

Increasing the first sleep in the script (sleep 2) to something longer (e.g. sleep 3) seems to fix this, so probably it's just a matter of trying to touch the screen too soon.

[kevin gunn (kgunn72)]

been running incoming text simulation for over 4 hrs, running at ~1 min interval to allow for inactivity timer to fire after interaction & sleep to allow for notification to disappear. no freeze yet.

[kevin gunn (kgunn72)]

altered script to include sms, phone call, sms in short order, ran over night 8+hrs no issue

[kevin gunn (kgunn72)]

ran combinations of phone call, sms and swiping throughout the weekend, no issue.

[Jamie Strandboge (jdstrand)]

I'm not sure if this is related or not, but the symptoms seem too similar to not comment....

With the blabble game in the store if you play the game (eg, start it, then start finding words as fast as you can by tap, dragging and lifting), after a while the UI gets stuck where several tiles are highlighted after you lift your finger and you can't tap anything any more (this UI state is not supposed to be possible-- when you lift your finger the word is processed and the tiles reset to normal colors-- it is like the app lost tap events). Using the power button to off/on or side swiping to bring up the spread and selecting blabble makes the app responsive again. This game has been in the store more than a year and a half and this did not start happening until recent updates. Seen on mako and arale on post-OTA6 rc-proposed.

Unfortunately, it is hard to reproduce-- when it happens, it usually happens a few times in close succession and then you might not see it again for minutes.

[kevin gunn (kgunn72)]

these are the current debug symbols for u-s-c and mir, for rc-proposed images circa sept 14
adding this zip file to make it easy, b/c at the moment the phone images rely on the stable-phone-overlay ppa so you cannot get these dbg symbols to automagically install

for libc and libstdc++ just install
sudo apt-get install libstdc++6-4.9-dbg libc6-dbg

[kevin gunn (kgunn72)]

note about the comment above - be careful, if you do not wish to have to reflash your phone and lose user data, do not modify.
Also, the mir in rc-proposed is currently 15.1, and stable is 14.1 so loading these symbols from the zip file won't help, you would need to get the matching symbols. Lastly, for the mir symbol files, you can adb push these onto the phone and then install with dpkg -i

[Michal Predotka (mpredotka)]

Experienced this bug today on bq 4.5 with OTA-6. I got a text message. The screen turned on, envelope went green and the led light started to blink green. I was driving, so I didn't touch the phone and was just waiting for the screen to dim. I've noticed the screen wss staying on longer than it should but the time on the phone was changing. After about 5 minutes I stopped the car and tried to interact with the phone. It was not responding to touches, swipes or power button presses. Eventually I pressed power button and volume down button together for maybe 20 seconds and the phone restarted.

[kevin gunn (kgunn72)]

i succeeded in getting my device stuck in a frozen state. it seems alive, i can interact with the shell, top shows processes running and updating.
trouble is, gdb t a a bt yeilds no output. bt and bt-full result in "No stack."

I did check the unity8.log and noticed the log is fairly normal, until the end where is spews "cannot open file <name> too many open files" so i grabbed a dump from lsof as well.

[kevin gunn (kgunn72)]

here's the list of open files dump

[kevin gunn (kgunn72)]

ok, i was able to attach gdb to usc, here's the bt

[Alexandros Frantzis (afrantzis)]

> i succeeded in getting my device stuck in a frozen state. it seems alive, i can interact with the shell

How can you interact with the shell? Does it respond to input? Can you turn the screen on/off with the power button?

> so i grabbed a dump from lsof as well.
> here's the list of open files dump

Good call! unity8 has clearly reached its fd limit. It seems to be leaking fds (can't tell if it's unity8 itself leaking or another component inside unity8, e.g., mir):

...
unity8 2421 phablet 1017r CHR 13,70 0t0 1122358 /dev/input/event6 (deleted)
unity8 2421 phablet 1018r CHR 13,70 0t0 1117702 /dev/input/event6 (deleted)
unity8 2421 phablet 1019r CHR 13,70 0t0 1120899 /dev/input/event6 (deleted)
unity8 2421 phablet 1020r 0000 0,9 0 5513 anon_inode
unity8 2421 phablet 1021r 0000 0,9 0 5513 anon_inode
unity8 2421 phablet 1022r CHR 13,70 0t0 1120534 /dev/input/event6 (deleted)
unity8 2421 phablet 1023r CHR 13,70 0t0 1117848 /dev/input/event6 (deleted)

Unfortunately we didn't get a list of files for USC since it's running as root. If you still have the phone in that state could you rerun lsof with sudo?

> ok, i was able to attach gdb to usc, here's the bt

The backtrace seems normal. USC is just waiting for work to do, nothing suspicious in this case.

So, the question is if this is indeed an instance of the same bug we have been investigating or a different bug you have uncovered.

I will follow the fd leak lead and see if I can get more information.

[Jean-Baptiste Lallement (jibel)]

I reproduce the issue by following the "fd leak" lead with the following test case:

1. Open 2 apps
2. Switch from the app scope to app1
3. Switch from app1 to the app scope
4. Switch from the app scope to app2
5 lock the screen (optional but it leaks even more fd)

At the same time monitor the number of file descriptors used by unity8:
$ watch -n1 "ls /proc/$(pidof unity8)/fdinfo/| wc -l "

After a while unity8 runs out of fd and the system hangs. Eventually unity8 will restart a moment later.

[Alexandros Frantzis (afrantzis)]

Another way to leak fds is just opening and closing an app. It leaks an fd of type 'anon_inode:[eventfd]' per app connection.

Still don't have enough info to verify (or disprove) that the fd leaks are the cause of the original bug. The leak seems to be unity8 specific and in previous reports there was some indication of USC involvement.

I filed https://bugs.launchpad.net/ubuntu/+source/unity8/+bug/1495871 so we can manage the bugs independently until we get more proof that they are indeed related.

[kevin gunn (kgunn72)]

ok, verified that u-s-c is still ok, power button works and screen blanks...but ui is still completely frozen.
attaching sudo lsof just in case

tried to look at backtrace of unity8, but it seems corrupted

[kevin gunn (kgunn72)]

and output from 'ls -lv /proc/$(pidof unity8)/fd'

[kevin gunn (kgunn72)]

Ok, so following up.
bug 1495871 - u-a-l leaking fd's is truly critical and is what people are experiencing in terms of a completely frozen ui. In the case of this bug, people are able to blank/unblank the screen with their power button.

we are going to leave this bug open to see if there is a 2nd bug somewhere.
we are basing this on the fact that some reports included descriptions of the power button not blanking/unblanking the screen which would imply a potential issue at the u-s-c level. so please provide any info or hail us on irc in #ubuntu-mir if you get a device in a state which is unresponsive on the touchscreen and the power button.

[Pat McGowan (pat-mcgowan)]

had some trouble getting this, not sure if its valid by the time I did
gdb of usc while froze

[Alexandros Frantzis (afrantzis)]

> had some trouble getting this, not sure if its valid by the time I did gdb of usc while froze

Unfortunately the command used in gdb was 't a a b' which inserted breakpoints in all threads, instead of the correct 't a a bt' (note 'bt' vs 'b') which would print the backtraces :) So the log is not very informative.

The lsof output shows that unity8 has *not* exhausted its fd namespace, so this is not an instance of https://bugs.launchpad.net/bugs/1495871. lsof was not run as root, so we didn't get fd information about USC (which runs as root).

For reference, the exact invocations to use to get all the information next time this occurs:

$ sudo gdb -p $(pidof unity-system-compositor)
(gdb) thread apply all backtrace full

$ sudo lsof

[Andrea Bernabei (faenil)]

Hello people :)

here's a full bt of all threads! :)

I have qtcore, qtdeclarative, and libunity debug packages installed, I hope this provides some insight!

[Andrea Bernabei (faenil)]

additional info: I currently have the bug on my Krillin, updated with the latest rc-proposed less than 24h ago iirc.

I will try to post more debug info later

[Andrea Bernabei (faenil)]

Here's a backtrace of unity-system-compositor! :)

I have qtcore, qtdeclarative and libunity debug packages installed.

Moreover, I installed the following packages as suggested by kgunn:
libmirserver33-dbgsym libmirclient9-dbgsym libmircommon5-dbgsym libmirplatform9-dbgsym libmirprotobuf1-dbgsym mir-client-platform-android3-dbgsym

NOTE: when I first tried to take the bt, the compositor had PID 1574.

I attached gdb to it, and while it was loading the dbgsyms I think the compositor died.
I checked top, compositor now had PID 1575.

UI still frozen, power button still not responding.

I attached gdb to it again, and got the bt that you can find attached.

[Andrea Bernabei (faenil)]

And here is unity8.log

[Launchpad Janitor (janitor)]

This bug was fixed in the package unity-system-compositor - 0.1.3+15.10.20150925.1-0ubuntu1

---------------
unity-system-compositor (0.1.3+15.10.20150925.1-0ubuntu1) wily; urgency=medium

  [ Alexandros Frantzis ]
  * Fix "unresponsive shell including power button doesn't blank/unblank
    display" (LP: #1491566)

  [ CI Train Bot ]
  * No-change rebuild.

 -- Alexandros Frantzis <email address hidden> Fri, 25 Sep 2015 14:09:43 +0000