Unable to login when disk space is exhausted

[Marius Gedminas]

I track Dapper. I did an apt-get upgrade today, rebooted, and was unable to log in. After I entered my username and password, I'd briefly see the first text console, and gdm would show me the login prompt again.

I tried logging into /dev/tty1 and using startx, and that's where I got an error message mentioing something about being unable to create an authentication socket in /tmp because the disk was full. sudo apt-get autoclean fixed things.

It would be nice if gdm could tell the user when the session fails because there is no free disk space in /tmp.

[Sebastien Bacher (seb128)]

Thanks for your bug

[Sebastien Bacher (seb128)]

The issue is similar to http://bugzilla.gnome.org/show_bug.cgi?id=144473 upstream

[Vincenzo Ciancia (vincenzo-ml)]

A way to log in must be provided anyway, I don't think it is acceptable for all ubuntu users to be able to log in inside a virtual console and type commands to remove files - at least, my mother would not be able to do so but she's able to use a graphical interface. We should identify what files must be necessarily created in order to log in and mount a ramfs ad-hoc for those files. Or the free disk space reserved to root could be perhaps used for the purpose (by writing some setuid program, but this would perhaps create more problems than it would solve).

The priority of this bug should be raised: if my disk is full, I can't log in (at least in the GUI) so how do I delete files? This is similar to nautilus-cd-burner not burning on the fly: I decide to write my files to DVD since I am running out of disk space, and I can't write files to DVD since I am running out of disk space.

Not willing to be polemic in any case :) just want to point out that this is an important bug.

[seanh (seanh)]

I had a user with this same problem. In his case, he filled up the disk by downloading stuff, then the next time he booted the computer GNOME fails to start, just dumps you back at GDM with no explanation. He had an essay to write and had absolutely no idea why his computer had completely stopped working, or what to do about it. Luckily I was on hand to fix it for him. Teaching him to use the text terminal to correct the problem himself in future is not really feasible, too much knowledge needed.

This is a serious bug, for normal users it amounts to complete system failure if they are neglectful and let their disk get full.

Windows XP keeps working when this happens, btw.

[seanh (seanh)]

I don't seem able to change the priority of this bug. I really don't think it should be set to 'low' though. The bug can effect anyone who fills up their disk, and leaves them completely unable to login, even to search for help, assuming they don't know how to use text mode, which most don't. They just better hope they can telephone a friend who knows more about Linux than they do.

[Chris Wagner (chris-wagner)]

Also see bug 41170. Someone will probably want to mark it as a duplicate of this bug, but I do believe it is actually a bit different. I think the problem that I came across, when reporting 41170, was due to GDM not being able to write to the user's home directory:
  "GDM could not write to your authorization file. This could mean that you are out of disc space or that your home directory could not be opened for writing. In any case, it is not possible to log in. Please contact your administrator."
That's a more helpful method of failure than described in this bug's description, but still not sufficient in solving the problem.

[Jordan Hall (jordan-hall)]

I would recommend this bug be raised to at least High priority.

A good friend of mine recently had this problem and we were almost at the point of formatting reinstalling the system, until I happened to remember hearing about this bug on the Ubuntu forums.

To fix this bug, we installed a free ext2/3 driver in Windows XP (the system is dual boot) and moved some files from the Linux partition to the Windows partition in order to free space and allow a graphical login.

No system message was given when my friend attempted to login with a full disk. The screen momentarily flickered and then he was returned to the GDM login screen.

One solution would be to remove unnecessary files (the package cache, various unimportant log files etc.) if attempting to login under these conditions. A more permanent and possibly easier to implement solution is to simply disallow users to write to the last few megabytes (50 to 100 ?) of the disk in order to completely prevent this problem, and hopefully reduce other issues that could occur from a low disk space situation.

[Vincenzo Ciancia (vincenzo-ml)]

There already is reserved space (for root, and gdm is launched as root) however this space can be filled by root itself using e.g. apt or pbuilder. Reserving space for a single program might be interesting but would require kernel modification.

I have a simpler solution in mind, but I am still waiting to find the time to try this :) The idea is to make gdm write _all_ of its temporary files, which are small, on a ramfs. Mounting a ramfs is really easy, however I didn't have the time to modify gdm in order to write all files in a user-defined directory. The next step would then be to check how many programs of the default gnome session crash if disk space is exhausted and perhaps use the same trick (even if I suspect non-root applications writing to a temporary filesystem could lead to denial of service attacks).

In "man mount" one can find

===
Mount options for ramfs
       Ramfs is a memory based filesystem. Mount it and you have it. Unmount
       it and it is gone. Present since Linux 2.3.99pre4. There are no mount
       options.

===

Actually, it is not necessary to use a ramfs. A loopback ext2 mount on a fixed-size file would probably be easier, and raise less security problems. The fixed-size file would act as the space "reserved for gdm". A temporary directory for the logged in user could be created, even with quotas, to allow writing temporary sockets to all session applications.

In both cases, the main problem remains, to make gdm write into a precise directory and not where she wants, i.e. in /tmp. First of all, let's see what are the files that gdm absolutely needs to write. Quoting from the upstream bug, the files gdm writes at login are:

===
+ /tmp/.gdm_socket for the GDM daemon and GUI program to talk to each other
  properly. Without the socket being setup, GDM probably would still work
  though the GUI program wouldn't be able to access the config file and would
  instead revert to the compiled-in defaults (e.g. your theming would probably
  be wrong). Features that require the socket to be working (such as
  automatic/timed login or any gdmflexiserver --commands) would also not work.
  Perhaps moving the socket to /var/tmp might make it less likely you'd have
  problems here?

+ GDM uses /tmp for the fallback Xauth key directory if the normal xauth
  directory (UserAuthDir in the GDM configuraiton) is also full. If GDM can't
  write xauth files, then it probably wouldn't let the user login. If both
  UserAuthDir and /tmp are full, this could be a problem with GDM not allowing
  users to log in at all.

+ /tmp/.ICE-unix, /tmp/.X11-lock, and /tmp/.X11-unix are also used. Not sure
  how GDM will break if it couldn't write these files. I'm guessing this might
  be the problem that causes GDM to fail to allow login when /tmp is full.

+ If the user's $HOME is full, it will put the .xsession-errors file in /tmp
  instead. Probably not a big deal if these don't get written, though I'm not
  sure what GDM does if it can't write them.
===

Then, I e-mailed Brian Cameron directly, asking him where are the locations of all these files decided in the code, this was the reply:

===
You'ld need to edit the ...

[Martin Olsson (mnemo)]

I'm glad that talented people like Vincenzo etc are actually thinking about ways of solving this. This bug has caused me a lot of problems and I would love to see it eliminated.

[Ian (superian)]

I agree that this should be assigned a higher priority,

Why, yes, it did catch me last night :)

It's made worse by the behaviour of ext3 when full: you can delete files using 'rm' or 'apt-get clear' *BUT* the free space doesn't increase?!? (Because the journalling system couldn't write anything before making the changes that would give it some space?)

A combination of reboot, oh no, delete some files, reboot, oh no, try deleting via Windows and ext2fs, reboot, oh no, run fschk, delete some files, reboot, oh no, run fschk etc worked. In the end.

But, basically, the behaviour when the file system fills up is Not Good.

I leave it to others to decide what the best solution is - stop the file system getting in this state? Having a specific utility to get free space (e.g. by emptying caches) in such a way that ext3 actually creates some or just increasing the helpfullness of the onscreen messages.

About the only good bit is that, browsing the bug database, it looks like KDE is even worse :/

[Ian (superian)]

Ah yes, my other comment about the current position is that it's particularly annoying when you've (sensibly) put /home on a different partition: by picking a different session, you can get a simple graphical login working (albeit without menus) *BUT* there's no way to change to root in order to delete files anywhere other than your /home directory... and deleting files there will have no effect.

You can see what you need to do, but you're stopped from doing it... argh.

[Beaumont Muni (beaumont)]

I agree ... this should be a high priority bug. I have tried other suggestions from earlier discussions where it was suggested that for some reason the ownership on the .ICEauthority file in my home directory has changed. And all that was needed to do to resolve the situation was to was to remove the file or change the owner ship to me instead of root. This didn't work either. Not being able to access your system in spite of trying to resolve things via safe terminal mode is pretty frustrating. I have more than enough space on my system (55 G HD). Further more I feel uneasy deleting stuff from the / files system since I this may affect booting up in a drastic way. I continue to get this issue with a new installations of ubuntu since version 6.06. I am really thinking of moving to another version of linux where this is not being experienced.

[Sebastien Bacher (seb128)]

The gdm code is mostly upstream you will likely get the same bug with any other linux distribution

[Sebastien Bacher (seb128)]

Note that it's easy enough to make some space, just uninstall something you are not using

[carvell (andy-tenup)]

The fact that you can't log in to uninstall something because you've run out of space is more the point, I think.

[Sebastien Bacher (seb128)]

That's going to be worked for gutsy, https://wiki.ubuntu.com/BootLoginWithFullFilesystem

[TonyO (okuthorr9)]

I have the same issue, except the fact with mine is that I freed more than 4 gigs of space by deleting some stuff and removing a ton of sofware with synaptic and apt-get, checked it with "df -h", only to reboot and find that my computer had somehow reclaimed that space and my root partition returned to 100% use.....is ther a fix yet for feisty?

[Sebastien Bacher (seb128)]

Does anybody still get the issue on gutsy?

[Vincenzo Ciancia (vincenzo-ml)]

I experienced the problem of disk space exhausted recently. It worked, i.e. I was able to log in, but nobody warned me so I had to interrupt my work and reboot when I noticed that the tmpfs had been mounted. I don't understand if the tmpfs can be mounted also while users are logged (I guess not or you would have to mv relevant files with all the connected problems) anyways situation is as follows: I logged in and worked until I started having usual /tmp full problems because the ramfs has a fixed size.

We must be sure that user is warned to reboot as soon as possible when the ramfs is mounted, and the fake device name of the ramfs (tmpfs, whatever) should be the name of a readme file explaining what happened. If I had not followed the thing from the beginning here, I would never have understood who had mounted that tmpfs, which I saw using "df".

Maybe I didn't get a warning because I have autologin and login arrived _before_ tmpfs being mounted?

[Sebastien Bacher (seb128)]

Do you consider the bug fixed in gutsy then? The tmpfs looks like a different issue

[ultim8k (ultimatekonstantinos)]

i confirm this bug

[uncle (old-uncle-z)]

I also confirm this BUG and waiting for any solution!

[Vincenzo Ciancia (vincenzo-ml)]

I am now testing hardy, I reported bug #186633 which seems to be different than this bug, but may be confused for it. Does it look like the same bug to some of you?

[Martin Pitt (pitti)]

Since we have /etc/init.d/mountoverflowtmp, there is a chacne that this is already fixed in Hardy. Ted agreed to test this throroughly to verify.

[Ted Gould (ted)]

I created a virtual machine running Hardy. Then I switched to a VT and filled up the disk with dd. When trying to login, GDM prompted me with an error that the disk was full, and it could not create the authentication tokens. There was no way to log in.

I rebooted the machine.

Upon reboot /etc/init.d/mountoverflowtmp kicked in and I was able to log into a normal session and get both a terminal and launch other programs to free up the space.

While I believe that I should have been able to log in the first time, I believe atleast this bug should be moved to wishlist for Hardy and after. It would be nice to have this work cleaner, but it does work.

[Vincenzo Ciancia (vincenzo-ml)]

Can't the check to mount overflow tmp just be added to gdm each time it restarts?

[Ted Gould (ted)]

On Sat, 2008-03-08 at 12:43 +0000, Vincenzo Ciancia wrote:
> Can't the check to mount overflow tmp just be added to gdm each time it
> restarts?

Well, not directly. GDM runs as the gdm user, and so wouldn't have root
permissions. Now, if there were a set uid script that did this, that
could do it.

I guess the question is whether it's worth moving this from an init
script to a patch for GDM with a set uid script that did the mounting.
That way when GDM sees the out of disk space condition, it would then
fix it by doing the mount overflow. I wouldn't see any harm in that,
assuming there was a reasonable security audit.

[ultim8k (ultimatekonstantinos)]

2008/3/8, Ted Gould <email address hidden>:
>
> On Sat, 2008-03-08 at 12:43 +0000, Vincenzo Ciancia wrote:
> > Can't the check to mount overflow tmp just be added to gdm each time it
> > restarts?
>
>
> Well, not directly. GDM runs as the gdm user, and so wouldn't have root
> permissions. Now, if there were a set uid script that did this, that
> could do it.
>
> I guess the question is whether it's worth moving this from an init
> script to a patch for GDM with a set uid script that did the mounting.
> That way when GDM sees the out of disk space condition, it would then
> fix it by doing the mount overflow. I wouldn't see any harm in that,
> assuming there was a reasonable security audit.
>
>
> --
> Unable to login when disk space is exhausted
> https://bugs.launchpad.net/bugs/35217
> You received this bug notification because you are a direct subscriber
> of the bug.
>

I don't know much about programming but i can imagine that "system" should
keep some disk space or something, "untouchable" by a simple user just for
the login process...

[Ted Gould (ted)]

As discussed this morning in the Desktop Team meeting we're going to mark this bug as fixed. Adding the ability for GDM to handle this is a different bug.

[toddq (toddq)]

It appears to me this bug is back. That is, the same thing happened to be in Ubuntu 8.10 today

[toddq (toddq)]

I should have also noted that Ted Gould's solution doesn't work in 9.10. That is, simply rebooting the computer has no effect. This problem began with the latest upgrade.

[toddq (toddq)]

I figured out a four part solution to this problem which has been going on since 2006.The history itself (which is given below) reveals the solution.

Part I https://bugzilla.gnome.org/show_bug.cgi?id=144473 has a partial fix for gdm which has not been assigned to anyone.

Part II The next part of the fix is that the gnome warning that your disk space is low is too weak. It should state that if you don't fix the problem before rebooting, you will lose everything. There should be additional warnings about the problem placed in ubuntu documentation.

Part III, ubuntu should abandon encryption until a method is found that allows you to recover a encrypted drive when you have a full-disk. At this point, it is my view that an encrypted drive is just do dangerous if you can lose everything because you have a full disk. My proposed solutions of course would make some people unhappy but I believe they are more appropriate than what has been going on since 2006, which is basically to endlessly put off the solution to the problem.

Part IV So my final proposed solution has to do with the tracker team. I'm sure we all appreciate the terrific job the tracker teams do in identifying and addressing problem. So why wasn't the problem solved in 2006? It appears the nature of problem and solution has befuddled the tracker team. The problem was difficult to recognize because a typical debug report is not possible because the user can't login. My proposed solution is to suggest that when a problem has been around for over a year that it is assigned to someone who goal is to identify similar problems in forums and other bug reports. If there is a pattern of a lot of similar bugs and/or questions not being dealt with and no one seems to have a constructive solutions, I would suggest that one person be assigned to come up with a workable solution to the more general problem. This approach would actually save time because instead of endlessly putting out small fires, you would identify an important underlying issue that is contributing to user dissatisfaction. It was quite striking to me when reading all this material, to see so many comments by experienced and highly competent progammers trivialize this issue which has clearly affected a sizeable portion of the Ubuntu community without anyone recognizing it's genuine important or realizing that a simple fix is possible. That's because the problem and solution are more about social psychology than they are about programming. You need a tracking process that addresses this issue. Sometimes the solution might be as simply as updating the documentation. For example, there is documentation about how to handle full-disks. However, this doesn't address all the issues raised in the discussions belows (e.g., how to address the problem when one has an encrypted drive).

THE HISTORY
Many links below including this one are assigned to gnome-bugs #144473 which means its assigned as a more general linuz problem. Unfortunately, the problem has little to do with linux it has to do with Ubuntu gdm. Next, legitimate confirmed high-priority bugs are linked to this black hole as duplicates. If someone brings it ...

[shankao (shankao)]

I can confirm that this bug is back in maverick

[David (lofidevops)]

This bug still occurs with the Unity greeter in 11.10. (Greeter is not displayed, cannot log in.)

[Sebastien Bacher (seb128)]

don't change years old bugs like that, there is already a bug about lightdm and things got improved in the precise version