Greeter/lockscreen fails to authenicate in an endless loop. Repeated "No space left on device" errors and strange swap file configuration

[Martin D. Weinberg]

After a recent install of the 18.04 (why is a separate story), I notice that after some number of successful screen locks, I become locked out of the desktop with an authentication failure. During those unsuccessful attempts to log in, syslog shows:

Mar 13 22:46:04 magpie /usr/lib/gdm3/gdm-x-session[1340]: (**) Option "fd" "21"
Mar 13 22:46:04 magpie kernel: [83554.367969] rfkill: input handler enabled
Mar 13 22:46:04 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) event2 - Power B
utton: device removed
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (**) Option "fd" "25"
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) event4 - Video B
us: device removed
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (**) Option "fd" "27"
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) event1 - Sleep B
utton: device removed
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (**) Option "fd" "24"
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) event17 - Integra
ted Camera: Integrated C: device removed
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (**) Option "fd" "29"
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) event3 - AT Tran
slated Set 2 keyboard: device removed
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (**) Option "fd" "28"
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) event5 - SynPS/2 Synaptics TouchPad: device removed
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (**) Option "fd" "30"
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) event6 - TPPS/2 IBM TrackPoint: device removed
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (**) Option "fd" "62"
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) event7 - ThinkPad Extra Buttons: device removed
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) AIGLX: Suspending AIGLX clients for VT switch
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) systemd-logind: got pause for 13:81
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) systemd-logind: got pause for 13:68
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) systemd-logind: got pause for 226:0
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) systemd-logind: got pause for 13:71
Mar 13 22:46:05 magpie /usr/lib/gdm3/gdm-x-session[1340]: (II) systemd-logind: got pause for 13:66

one new line for each attempted log in. This is occurring once or twice a day. Sometimes, it works again. Last time, I needed to reboot.

Since this is a fresh install, albeit with my desktop files being restored via CrashPlan (which could be part of the problem?), I thought that this warranted a report.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: gdm3 3.27.92-0ubuntu1
ProcVersionSignature: Ubuntu 4.15.0-10.11-generic 4.15.3
Uname: Linux 4.15.0-10-generic x86_64
ApportVersion: 2.20.8-0ubuntu10
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Mar 14 07:58:11 2018
EcryptfsInUse: Yes
InstallationDate: Installed on 2018-03-12 (1 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Alpha amd64 (20180114)
SourcePackage: gdm3
UpgradeStatus: No upgrade log present (probably fresh install)

[Martin D. Weinberg (martin-weinberg-5)]

Addendum: I was running the Xorg ("Ubuntu") session when the problems occurred. I'm trying the Wayland session now, so the autogenerated tag is not correct, and maybe not relevant.

[Martin D. Weinberg (martin-weinberg-5)]

Also a problem with the Wayland session. Not a big surprise. For now, I've turn off the screen lock.

Not sure what else to do.

[Daniel van Vugt (vanvugt)]

Martin,

Can you please reproduce the problem with Wayland and then:

  journalctl -b0 > todays-journal.txt

and attach the resulting file here?

[Martin D. Weinberg (martin-weinberg-5)]

Daniel,

I took a few days, but it did again fail with the same issue. I'm attaching the log, collected shortly after the failure by switch to console while running the Wayland session, as you requested. Had to reboot afterwards, to be able to log on to the desktop.

Hope this helps you spot the issue.

[Daniel van Vugt (vanvugt)]

Thanks. I can see two recurring issues that might be the cause:

1. Something called 'variety' is failing:

Mar 17 08:16:14 magpie variety.desktop[1860]: /home/weinberg/.config/variety/scripts/set_wallpaper: line 121: [: ==: unary operator expected

Try fixing your script, or try uninstalling 'variety'.

2. Multiple crashes in extensions:

/<email address hidden>/*
/<email address hidden>/*

But also:

3. Please check that you don't have any gnome-shell extensions installed other than the two default Ubuntu ones (mentioned above). Since they all run in the same process, any buggy extension can break the entire shell.

[Martin D. Weinberg (martin-weinberg-5)]

Thanks.  That variety script was from an old version of variety, it
seems.  I removed that and checked that the current scripts  in
/usr/share/variety/scripts do not have that bug (looked like an
operation on a shell variable that may not exist).  I did notice a few
variety crashes, but they do not seem to coincide with screenlock failure.

I did have "clipboard indicator" and "media player indicator" enabled.
Disabled those.  Will see what happens.  I can uninstall variety if I
see more issues with that.

On 03/19/2018 10:07 PM, Daniel van Vugt wrote:
> Thanks. I can see two recurring issues that might be the cause:
>
> 1. Something called 'variety' is failing:
>
> Mar 17 08:16:14 magpie variety.desktop[1860]:
> /home/weinberg/.config/variety/scripts/set_wallpaper: line 121: [: ==:
> unary operator expected
>
> Try fixing your script, or try uninstalling 'variety'.
>
> 2. Multiple crashes in extensions:
>
> /<email address hidden>/*
> /<email address hidden>/*
>
> But also:
>
> 3. Please check that you don't have any gnome-shell extensions installed
> other than the two default Ubuntu ones (mentioned above). Since they all
> run in the same process, any buggy extension can break the entire shell.
>
>
> ** Also affects: gnome-shell-extension-appindicator (Ubuntu)
> Importance: Undecided
> Status: New
>
> ** Changed in: gnome-shell-extension-appindicator (Ubuntu)
> Status: New => Incomplete
>
> ** Also affects: gnome-shell-extension-ubuntu-dock (Ubuntu)
> Importance: Undecided
> Status: New
>
> ** Changed in: gnome-shell-extension-ubuntu-dock (Ubuntu)
> Status: New => Incomplete
>

--
Martin Weinberg
6 Grass Hill Rd
West Whately, MA
010039

[Martin D. Weinberg (martin-weinberg-5)]

I'm still getting failures. This is with a fixed variety and no gnome-shell extensions other than the default ubuntu extensions.

I don't see but reports similar to this, so I gather that others are not generally plagued by this problem. Nonetheless, I appreciate any advice or comment you have on what to do here.

I'd like to contribute to improving Ubuntu, but at some point, I going to have to do something to make this stop just for my own sanity; having to reboot at random times so that I can log in to my laptop is really not a good thing. Switch to a different session type?

I will attach another journanctl -b0 log

Thanks!

[Daniel van Vugt (vanvugt)]

Your log is showing "No space left on device"

So one of your disks is full, which may be the problem. Please check by running:

  df -h

[Martin D. Weinberg (martin-weinberg-5)]

I saw that.  But the disk is far from full.  Don't know what's up with
the /snap loopback devices.  The hard drive itself is at 35%.  Here's
the df output:

Filesystem               Size  Used Avail Use% Mounted on
udev                     3.8G     0  3.8G   0% /dev
tmpfs                    767M  1.9M  765M   1% /run
/dev/sda5                450G  148G  279G  35% /
tmpfs                    3.8G  4.0K  3.8G   1% /dev/shm
tmpfs                    5.0M  4.0K  5.0M   1% /run/lock
tmpfs                    3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/loop1               163M  163M     0 100% /snap/spotify/6
/dev/loop0                82M   82M     0 100% /snap/core/4206
tmpfs                    767M   16K  767M   1% /run/user/120
tmpfs                    767M   36K  767M   1% /run/user/1000
/home/weinberg/.Private  450G  148G  279G  35% /home/weinberg

On 03/25/2018 10:59 PM, Daniel van Vugt wrote:
> Your log is showing "No space left on device"
>
> So one of your disks is full, which may be the problem. Please check by
> running:
>
> df -h
>

--
Martin Weinberg
6 Grass Hill Rd
West Whately, MA
010039

[Daniel van Vugt (vanvugt)]

"No space left on device" is coming up way too often in your log. I can't explain that but it is still possibly relevant since a lot of things are failing with that message.

Please monitor your disk usage carefully, before and after the problem occurs again. And when it occurs again, please send us a fresh log file as well as more 'df -h' output.

[Daniel van Vugt (vanvugt)]

Come to think of it, some code might be returning ENOSPC ("No space left on device") for a RAM-based operation.

So please also check how much RAM your system has before and after the problem (run 'free -m'), and check 'ps' to see if any process is unusually large.

[Martin D. Weinberg (martin-weinberg-5)]

Thanks.  Your comments led me to look at swap.  There is no swap!

I installed from iso image several weeks ago (I am traveling and had a
bad ecryptfs corruption, not sure how this happened, and needed to
reinstall, and had this on my memory stick for testing).

I selected encrypted and had an prexisting swap partition.   The
installer made a 2GB swapfile (okay) but the ecryptfs swap did not get
configured correctly, so no swap.  So there is some inconsistency with
the installer, clearly, that should be ironed out.

I made normal swap on both the swapfile and the swap partition, followed
by swapon -a.  Now there is plenty of swap.  I wonder if this is the
cause of the lockscreen behavior?  Difficultly with swap.

I will observe.  Any pointers on correctly configuring my ecryptfs swap
at this point?  Thanks!!

On 03/26/2018 12:21 AM, Daniel van Vugt wrote:
> Come to think of it, some code might be returning ENOSPC ("No space left
> on device") for a RAM-based operation.
>
> So please also check how much RAM your system has before and after the
> problem (run 'free -m'), and check 'ps' to see if any process is
> unusually large.
>
> ** Changed in: gdm3 (Ubuntu)
> Status: New => Incomplete
>
> ** No longer affects: gnome-shell-extension-appindicator (Ubuntu)
>
> ** No longer affects: gnome-shell-extension-ubuntu-dock (Ubuntu)
>

--
Martin Weinberg
6 Grass Hill Rd
West Whately, MA
010039

[Martin D. Weinberg (martin-weinberg-5)]

I've been studying the swap mess. It seems that the installer made a swapfile called /swapfile and entered the line in /etc/crypttab as:

cryptswap1 /target/swapfile /dev/urandom swap,offset=1024,cipher=aes-xts-plain64

which is the wrong, non-existent filename. So that seems to be an issue with the installer scripts.

Also, I do not believe that the installer even asked me about a swap partition. I recall being surprised at the time.

Anyway, after fixing that and rerunning ecryptfs-setup-swap, I was able to use cryptdisks_start to get the encrypted swap going and reboot with encrypted swap successfully. I wonder if this has been the root of my problem all along?

[Daniel van Vugt (vanvugt)]

I can confirm the default swapfile behaviour in 18.04 looks weird, but I don't know what the intention there is. Please log a separate bug for that if you think it's a problem that needs fixing.

[Launchpad Janitor (janitor)]

[Expired for gdm3 (Ubuntu) because there has been no activity for 60 days.]

[Daniel van Vugt (vanvugt)]

I'm a little rusty and lost now. Can you please reword the bug title to better describe the problem, if any?

[Martin D. Weinberg (martin-weinberg-5)]

We can close this bug report at this point. But I do appreciate the follow up!

It turns out that there were two problems. The first was that 18.04 had misconfigured the cryptoswap so that was not working at all. I reported this separately before the release of bionic and I believe that this bug has been fixed by the maintainers. The second was that the kernel was running out of inotify watches and this prevented (for some reason) gdm3 from allowing logins to complete. The inotify watches were being consumed by the Crashplan java app. I increased the fs.inotify.max_user_watches to 1048576 and that solved the problem. I realize that this consumes some unswappable ram, but I rather have Crashplan than not.