landscape-broker segfaults killing network connection and touchscreen driver

[Damiön la Bagh]

Every week on Sunday morning one of my Ubuntu 16.04 machines goes into a non-usable state.
The log file shows the last entry in the log is

landscape-broke[1164]: segfault at 0 ip 000000000049469d sp 00007ffe06803db0 error 4 in python2.7[400000+2de000]

All logging stops
the screen only shows the background
and the only way out is to 5 second press the power button on the computer
which then restores the computer to a usable state again.

Sunday mornings the computer is idle, and sleep state has been turned off

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: landscape-client 16.03-0ubuntu2.16.04.4
ProcVersionSignature: Ubuntu 4.4.0-127.153-generic 4.4.128
Uname: Linux 4.4.0-127-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.18
Architecture: amd64
Date: Sun Jun 10 11:35:46 2018
InstallationDate: Installed on 2015-04-29 (1137 days ago)
InstallationMedia: Ubuntu 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=nl_NL.UTF-8
 SHELL=/bin/bash
SourcePackage: landscape-client
UpgradeStatus: Upgraded to xenial on 2017-03-05 (461 days ago)

[Andreas Hasenack (ahasenack)]

Could you please attach /var/log/kern.log and /var/log/kern.1.log from that machine?

[Damiön la Bagh (kat-amsterdam)]

Here are the requested log files including the Landscape error

[Andreas Hasenack (ahasenack)]

This looks like a kernel bug:
Jun 9 18:37:37 tilburg-kassa kernel: [41840.340109] landscape-broke[1217]: segfault at 0 ip 000000000049469d sp 00007ffff276f720 error 4 in python2.7[400000+2de000]
Jun 9 18:38:25 tilburg-kassa kernel: [41888.700394] BUG: unable to handle kernel paging request at 000000000000ff60
...
Jun 9 19:05:33 tilburg-kassa kernel: [43516.157000] traps: sshd[10982] trap invalid opcode ip:7ff469ac8d40 sp:7fffb13fe488 error:0 in pam_unix.so[7ff469ac1000+e000]
Jun 9 19:33:50 tilburg-kassa kernel: [45213.343686] BUG: unable to handle kernel paging request at 0000000000002960
...

and other similar ones.

How much memory do you have on this box? Please show output of:

free -h
cat /proc/swaps

landscape-client is known to consume RAM, maybe it's triggering other problems when the system becomes starved.

[Damiön la Bagh (kat-amsterdam)]

Heres a 3-day graph from landscape showing that memory never reaches more that 57 used.
The gap of course being the downtime caused by this bug.

~$ free -h
              total used free shared buff/cache available
Mem: 3,8G 959M 2,0G 28M 819M 2,6G
Swap: 3,9G 0B 3,9G

Filename Type Size Used Priority
/dev/dm-1 partition 4112380 0 -1

[Damiön la Bagh (kat-amsterdam)]

Here is Swap usage over 3 days.

[Damiön la Bagh (kat-amsterdam)]

Jun 9 18:36:29 machine sshd[10309]: Failed password for root from $EVILIP port 60394 ssh2
Jun 9 18:38:25 machine sshd[10335]: Server listening on 0.0.0.0 port 22.
Jun 9 18:38:25 machine sshd[10335]: Server listening on :: port 22.
Jun 9 18:38:39 machine sshd[10339]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=$EVILIP user=root

Jun 9 18:38:25 tilburg-kassa kernel: [41888.700394] BUG: unable to handle kernel paging request at 000000000000ff60

I notice on the same timestamp that ssh does a Server listening on 0.0.0.0 port 22

But for the SSHD error nothing in the log.

Jun 9 19:05:33 tilburg-kassa kernel: [43516.157000] traps: sshd[10982] trap invalid opcode ip:7ff469ac8d40 sp:7fffb13fe488 error:0 in pam_unix.so[7ff469ac1000+e000]

[Launchpad Janitor (janitor)]

[Expired for landscape-client (Ubuntu) because there has been no activity for 60 days.]