poweroff not working after apt upgrade

[Günter Neiß]

System-Information:
Description: Ubuntu 18.04.4 LTS
Release: 18.04
######################################
Working packages:
- Kernel 5.3.0.61.114
- linux-firmware 1.173.18

Problem occurs when upgrading (anyone of) these packages
- Kernel 5.3.0-62.56
- linux-firmware 1.173.19

I use an Acer Aspire E5-771G Notebook with a fresh installation of Ubuntu 18.04.
Till the last upgrade everything works as expected.
The last upgrade brings two new kernel versions (as mentioned above).
Now the system stucks at the very end of the power down sequence.
Even SysReq-O will not work (only SysReq-B will work).

Because I use Systemback, I time shift to the previous system and then install package by package to see what will bring in the problem.
If I install any of the mentioned kernel updates the problem will show up.

I had ask the above question on "askubuntu.com".
I was directed to upgrade the BIOS, But even after doing so the problem still persist.

I am not sure if I do it right (APT pinning).
I currently create a file /etc/apt/prefereces.d/kernel.

I add an entry for every of the following packages to block the version mentioned above (Pin: version: <see above> Pin-Priority: -1):

linux-generic-hwe-18.04
linux-headers-generic-hwe-18.04
linux-image-generic-hwe-18.04
linux-modules-nvidia-440-generic-hwe-18.04
linux-firmware

After doing so "apt list --upgradeable" will not list the packages in question and the PC will power down normally (and SysReq-O WILL work).

In the meanwhile there are newer versions of the kernel online, so the above mentioned pinning will allow the newer kernels to get installed and the problem will show up again.
So I block these versions too.

BTW: Due to the stuck I am able to see some other error messages that are not reported in the journal. The very last 3-5 lines show:
shutdown [1]: Failed to wait for process: Protocol error
These lines are present although in the working kernels, but hard to see, because the system powers down some ms after these lines are displayed!
This problem seams to be solved more than one year ago in the upstream kernel!
---
ProblemType: Bug
ApportVersion: 2.20.9-0ubuntu7.15
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: gneiss 1929 F.... pulseaudio
 /dev/snd/controlC0: gneiss 1929 F.... pulseaudio
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2019-05-02 (448 days ago)
InstallationMedia:

MachineType: Acer Aspire E5-771G
NonfreeKernelModules: nvidia_modeset nvidia
Package: linux (not installed)
ProcEnviron:
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
 PATH=(custom, no user)
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.3.0-61-generic root=UUID=0b245afe-758b-4e45-a473-ad2077260d8f ro noplymouth resume=UUID=79a9e809-6ba1-491b-acfc-1816057c2dee
ProcVersionSignature: Ubuntu 5.3.0-61.55~18.04.1-generic 5.3.18
RelatedPackageVersions:
 linux-restricted-modules-5.3.0-61-generic N/A
 linux-backports-modules-5.3.0-61-generic N/A
 linux-firmware 1.173.18
Tags: bionic
Uname: Linux 5.3.0-61-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip input lpadmin plugdev sambashare sudo video
_MarkForUpload: True
dmi.bios.date: 09/02/2015
dmi.bios.vendor: Insyde Corp.
dmi.bios.version: V1.23
dmi.board.asset.tag: Type2 - Board Asset Tag
dmi.board.name: EA70_HB
dmi.board.vendor: Acer
dmi.board.version: Type2 - A01 Board Version
dmi.chassis.type: 10
dmi.chassis.vendor: Chassis Manufacturer
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnInsydeCorp.:bvrV1.23:bd09/02/2015:svnAcer:pnAspireE5-771G:pvrV3.72:rvnAcer:rnEA70_HB:rvrType2-A01BoardVersion:cvnChassisManufacturer:ct10:cvrChassisVersion:
dmi.product.family: BDW
dmi.product.name: Aspire E5-771G
dmi.product.sku: Aspire E5-771G_0884_1_23
dmi.product.version: V3.72
dmi.sys.vendor: Acer

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1888570/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1888570

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

apport information

[Günter Neiß (gneiss)]

I have done as sugested (call apport-collect 1888570). But this was done while running the kernel that works. Should I install the not working kernes and report again ?

[Günter Neiß (gneiss)]

Just one more information:
Befor the bug ocours I had two kernels that gets maintaind by apt: The one without "hwe" and the newer hwe versions. Because the non HWE kernel wasn't used (pulled in via linux-generic).
While searching the problem I found that i can remove this non used kernel by doing:
apt purge linux-generic linux-headers-generic linux-image-generic linux-modules-nvidia-440-generic
Now I had only kernel 5.3.0-61-generic running. The newer ones 5.3.0-62-generic and 5.4* will all result in the mentioned poweroff problem.

[Alex Hung (alexhung)]

Thanks for reporting this, and your help is needed to identify the patch that causes this regression.

There are 260 patches between 5.3.0-61 & 5.3.0-62, and it requires some testing with bisects (see below).

$ git bisect start
$ git bisect good Ubuntu-5.3.0-61.55
$ git bisect bad Ubuntu-5.3.0-62.56
Bisecting: 260 revisions left to test after this (roughly 8 steps)
[19357b3b0835b96039e907f65ae203f7adc48ef6] epoll: atomically remove wait entry on wake up

A test kernel is available @ https://people.canonical.com/~alexhung/LP1888570/. Let me know whether it is pass or fail so I can build a next version.

[Alex Hung (alexhung)]

Correction to #19. The kernel is available @ https://people.canonical.com/~alexhung/LP1888570/19357b3b0835/

[Günter Neiß (gneiss)]

Sorry, I did not know what exactly I havee to do now.
As far as I can see, bisect is a tool to do a binary search inside a repro to find a commit that causes a failure (a special behaviour).

So to use it I have to have a (checked out) repro.
So I believe the commands given above (#19) must be given on a checkout of the kernel repro..

On the other hand the link You gave (#19) shows 6 deb-packages (compiled by You) assuumed by doing the above steps on such a repro.

I will now try to install these 6 deb-packages, hope this is what I should do.

[Günter Neiß (gneiss)]

OK, it seams I had understood You right.
After installing the 6 deb-packages from #20 the problem still persistist.

You might now provide a kernel with the next step ....
Or if You tell me the correct git-repro I had to checkout to meake the steps by my own, I will do the 8 steps myself and tell You only the final result of bisect (Your choise).

[Alex Hung (alexhung)]

@gneiss,

Thanks. Next one is https://people.canonical.com/~alexhung/LP1888570/d37a638086c1/

[Günter Neiß (gneiss)]

@alexhung,
Thanks, same result: bad

[Alex Hung (alexhung)]

Thanks. Next one is https://people.canonical.com/~alexhung/LP1888570/0affe1fa22b2/

[Günter Neiß (gneiss)]

Sorry this one can't be installed.
Here is the output, in german, but I think You see what happens.

root@GNAspire:~/Schreibtisch/Temp/debs# sudo apt install ./*.deb
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
Hinweis: »linux-buildinfo-5.3.0-62-generic« wird an Stelle von »./linux-buildinfo-5.3.0-62-generic_5.3.0-62.56_amd64.deb« gewählt.
Hinweis: »linux-cloud-tools-5.3.0-62-generic« wird an Stelle von »./linux-cloud-tools-5.3.0-62-generic_5.3.0-62.56_amd64.deb« gewählt.
Hinweis: »linux-headers-5.3.0-62« wird an Stelle von »./linux-headers-5.3.0-62_5.3.0-62.56_all.deb« gewählt.
Hinweis: »linux-headers-5.3.0-62-generic« wird an Stelle von »./linux-headers-5.3.0-62-generic_5.3.0-62.56_amd64.deb« gewählt.
Hinweis: »linux-image-unsigned-5.3.0-62-generic« wird an Stelle von »./linux-image-unsigned-5.3.0-62-generic_5.3.0-62.56_amd64.deb« gewählt.
Hinweis: »linux-modules-5.3.0-62-generic« wird an Stelle von »./linux-modules-5.3.0-62-generic_5.3.0-62.56_amd64.deb« gewählt.
Hinweis: »linux-modules-extra-5.3.0-62-generic« wird an Stelle von »./linux-modules-extra-5.3.0-62-generic_5.3.0-62.56_amd64.deb« gewählt.
Hinweis: »linux-tools-5.3.0-62-generic« wird an Stelle von »./linux-tools-5.3.0-62-generic_5.3.0-62.56_amd64.deb« gewählt.
Einige Pakete konnten nicht installiert werden. Das kann bedeuten, dass
Sie eine unmögliche Situation angefordert haben oder, wenn Sie die
Unstable-Distribution verwenden, dass einige erforderliche Pakete noch
nicht erstellt wurden oder Incoming noch nicht verlassen haben.
Die folgenden Informationen helfen Ihnen vielleicht, die Situation zu lösen:

Die folgenden Pakete haben unerfüllte Abhängigkeiten:
 linux-cloud-tools-5.3.0-62-generic : Hängt ab von: linux-cloud-tools-5.3.0-62 ist aber nicht installierbar
 linux-tools-5.3.0-62-generic : Hängt ab von: linux-tools-5.3.0-62 ist aber nicht installierbar
E: Probleme können nicht korrigiert werden, Sie haben zurückgehaltene defekte Pakete.
root@GNAspire:~/Schreibtisch/Temp/debs# ll
insgesamt 72848
drwxr-xr-x 2 gneiss gneiss 4096 Jul 29 17:12 ./
drwxrwxr-x 4 gneiss gneiss 4096 Jul 25 17:23 ../
-rw-rw-r-- 1 gneiss gneiss 506060 Jul 29 17:11 linux-buildinfo-5.3.0-62-generic_5.3.0-62.56_amd64.deb
-rw-rw-r-- 1 gneiss gneiss 289952 Jul 29 17:11 linux-cloud-tools-5.3.0-62-generic_5.3.0-62.56_amd64.deb
-rw-rw-r-- 1 gneiss gneiss 11237476 Jul 29 17:12 linux-headers-5.3.0-62_5.3.0-62.56_all.deb
-rw-rw-r-- 1 gneiss gneiss 1465188 Jul 29 17:11 linux-headers-5.3.0-62-generic_5.3.0-62.56_amd64.deb
-rw-rw-r-- 1 gneiss gneiss 8953196 Jul 29 17:12 linux-image-unsigned-5.3.0-62-generic_5.3.0-62.56_amd64.deb
-rw-rw-r-- 1 gneiss gneiss 13593492 Jul 29 17:12 linux-modules-5.3.0-62-generic_5.3.0-62.56_amd64.deb
-rw-rw-r-- 1 gneiss gneiss 38240616 Jul 29 17:12 linux-modules-extra-5.3.0-62-generic_5.3.0-62.56_amd64.deb
-rw-rw-r-- 1 gneiss gneiss 290080 Jul 29 17:12 linux-tools-5.3.0-62-generic_5.3.0-62.56_amd64.deb

[Günter Neiß (gneiss)]

After moving the DEBs
linux-cloud-tools-5.3.0-62-generic_5.3.0-62.56_amd64.deb
and
linux-tools-5.3.0-62-generic_5.3.0-62.56_amd64.deb
away.
The installation works.
I assume this is OK, because formerly You suply only 6 files, but now 8, so I use only the same files as before.
I have to reboot now to tell the result....

[Günter Neiß (gneiss)]

Hm, the output of
sudo apt install ./*.deb
still states:
Hinweis: »linux-buildinfo-5.3.0-62-generic« wird an Stelle von »./linux-buildinfo-5.3.0-62-generic_5.3.0-62.56_amd64.deb« gewählt.
which means (I assume) that I did not install the packages You provided, instead I install the "normal" packages.
I don't know why this happens and how to prevent this.
It seams that apt simply ignores the given local DEB-files and althought ignore the apt-pinning I had made for this packages.
So maybe my former tests are done against the normal packages :-(

[Günter Neiß (gneiss)]

OK, I hope I do it now right.
Instead of using apt, I use
sudo dpkg -i ./*.deb
inside the folder where I have downloaded Your files.
uname -a
Then shows me a date that is close to Your message date (so it feels OK).

The problem with the 2 extra DEBs from Msg 25 are the same, so I can't install them.

The results regarding the power off problem are (for all 3 versions): BAD

Sorry for long messages, but I will give as mutch information as possible.

[Alex Hung (alexhung)]

@gneisss,

I forgot to remove to *tool*.deb. They aren't needed.

next one: https://people.canonical.com/~alexhung/LP1888570/115f51d0fa11/

[Günter Neiß (gneiss)]

OK, Result for #30: BAD

[Alex Hung (alexhung)]

#19 and forward assume kernel 5.3.0-61 is good, according to bug description; however, it may worthwhile to give it: https://people.canonical.com/~alexhung/LP1888570/Ubuntu-5.3.0-61.55/

Especially, all tries were bad kind of worrying...

If it is good, the next one is https://people.canonical.com/~alexhung/LP1888570/875f10fe899f/

[Günter Neiß (gneiss)]

OMG, Your comment: "#19 and forward assume..."
Points into the right direction.
When I install the "good" kernel-DEBs You provided the problem pops up...
This happens although the "real" kernel deb (linux-image-unsigned-5.3.0-61-generic_5.3.0-61.55_amd64.deb) was not installed due to conflicts against my signed kernel of the same version.

I then compare the snapshots made by SystemBack of these two versions.
There are a lot of differences mainly the initrd (in boot) and althougth all *.ko inside modules.

I remember that every try to install something that resuts in a kernel update (dkms and/or update-initrd) the problem ocurs.

I therefore do a manual
sudo update-initramfs -k all -u
without installing anything else.
-> Problem pops up!

This happens even when I use my oldest system snapshot I have :-(

So how to proceed ?
(I am thinking about doing a fresh install of Ubuntu 20.4 LTS, but have to check if all tools I use will be available... FreeCad wasn't the first time I checked).

[Alex Hung (alexhung)]

A fresh install would certainly help. It is also possible that additional drivers can cause this; NV proprietary driver can be an example of it.

[Günter Neiß (gneiss)]

That's something I already tried (removing the NV driver), but that didn't help either.
I belive that some part of the tools needed for kernel preparation is broken..
I will try the instalation of Ubuntu 20.4 LTS next weekend.
As far as I could see now all programms I use are available for this release.

[Alex Hung (alexhung)]

Ubuntu 20.04.1 is going to be released on Aug 06 (tomorrow). You can also give it a try too.

[Günter Neiß (gneiss)]

Today I had donloaded the ubuntu-20.04.1-desktop-amd64.io and do a fresh install.
unfortunaly the problem exist there too :-(
I will do some more tests and inform You about the findings..
(will take some time...)

[Günter Neiß (gneiss)]

I don't know what happens. Because I had installed Version 20.04.1 prior I had to reinstall a dsk image of my former system and althougth reinstall grub2...
I then update package by package and everything works as it should be..
Very corious, but now all seams to be OK..
Maybe the real reason of the falure was inside grub ???

I belive tat we could not find the real reason for the probem, but this case could be closed now.
Thanks for all Your help!