ext4 "no space left on device" problems

[Andreas]

I'm suffering from very odd "no space left on device" problems.

I installed Karmic Alpha 2 with the following partition table:

Model: ATA Hitachi HTS54251 (scsi)
Disk /dev/sda: 160GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
 1 20.5kB 210MB 210MB fat32 EFI System Partition boot
 2 210MB 6652MB 6442MB hfs+ Untitled
 3 6652MB 7002MB 350MB ext4 root
 4 7002MB 13.0GB 6000MB ext4 usr
 5 13.0GB 16.0GB 3000MB ext4 var
 6 16.0GB 28.1GB 12.1GB linux-swap(new) swap
 7 28.1GB 28.5GB 400MB ext4 tmp
 8 28.5GB 160GB 132GB ext4 local

the disk usage is as follows:

Filesystem Size Used Avail Use% Mounted on
/dev/sda3 324M 185M 123M 61% /
tmpfs 2.0G 0 2.0G 0% /lib/init/rw
varrun 2.0G 116K 2.0G 1% /var/run
varlock 2.0G 0 2.0G 0% /var/lock
udev 2.0G 352K 2.0G 1% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/sda8 123G 2.0G 115G 2% /local
/dev/sda7 370M 11M 341M 3% /tmp
/dev/sda4 5.6G 2.7G 2.6G 51% /usr
/dev/sda5 2.8G 282M 2.4G 11% /var

Today I could not even login via gdm because of the "no space left on device problem". This is my .xsession-errors file:

gdm-session-worker[3532]: WARNING: could not save session and language settings: Failed to create file '/home/koto/.dmrc.EGLPWU': No space left on device
/etc/gdm/Xsession: Beginning session setup...
Setting IM through im-switch for locale=en_US.
Start IM through /etc/X11/xinit/xinput.d/all_ALL linked to /etc/X11/xinit/xinput.d/default.
E: core-util.c: Failed to symlink /home/koto/.pulse/3f6106b0be07f68824b215244a50a16c:runtime.tmp to /tmp/pulse-f4lUtkX4Itak: No space left on device
E: core-util.c: Failed to symlink /home/koto/.pulse/3f6106b0be07f68824b215244a50a16c:runtime.tmp: No space left on device
W: lock-autospawn.c: Cannot access autospawn lock.

HOWEVER: all of above listed output was generated by piping the command outputs (df, parted) to /tmp. And as you can see from the df, /tmp is not full.

I executed a fsck -f on all partitions but this did not help.
I've seen this problem before on my big 123GB partition when I was trying to check out a big subversion trunk. But again - the drive is practically empty.

[Strong (bwwwaaaa)]

I have similar problems only on my disk this point is already reached at 30%.

If I do a '# debugfs /dev/md1' stats I see that there are only a few inodes left so I suspect that during the upgrade to Karmic's ext4 file system far too less inodes were created.

Does anyone know how to increase them without loosing data?

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md1 493G 142G 346G 30% /

cp: cannot create directory `./Yazoo': No space left on device

# debugfs /dev/md1

stats

Filesystem volume name: main
Last mounted on: /
Filesystem UUID: 20423a23-b921-41b1-82bf-e27a826dd64d
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 504320
Block count: 129090240
Reserved block count: 1290902
Free blocks: 92021041
Free inodes: 35

[Jeremy Foshee (jeremyfoshee)]

Hi Andreas,

This bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? Can you try with the latest development release of Ubuntu? ISO CD images are available from http://cdimage.ubuntu.com/releases/ .

If it remains an issue, could you run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux 395864

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

[This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[Andreas (andreas-kotowicz)]

I've downloaded the latest cd image from the link above and did a clean install. I did not face the problem anymore, so it seems to be fixed.

[Andreas (andreas-kotowicz)]

I'm facing this problem again on ubuntu 9.10:

$ df -h
/dev/sdc1 932G 458G 427G 52% /local

$ df -i
/dev/sdc1 953984 953984 0 100% /local

as you can see, there are no inodes left. How can I fix this?

[Andreas (andreas-kotowicz)]

AplayDevices: aplay: device_list:223: no soundcards found...
Architecture: amd64
ArecordDevices: arecord: device_list:223: no soundcards found...
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/dsp', '/dev/snd/pcmC0D1p', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/pcmC0D2c', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D1c', '/dev/snd/pcmC0D0p', '/dev/snd/hwC0D2', '/dev/snd/seq', '/dev/snd/timer', '/dev/sequencer', '/dev/sequencer2'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 9.10
HibernationDevice: RESUME=UUID=f4d54674-0095-420f-8c81-e902dc905cce
MachineType: System Manufact System Product
NonfreeKernelModules: nvidia
Package: linux (not installed)
ProcCmdLine: root=UUID=f5e021c9-550f-4c2c-895c-4687ad19f364 ro quiet splash quiet
ProcEnviron:
 SHELL=/bin/bash
 LANG=en_US.UTF-8
ProcVersionSignature: Ubuntu 2.6.31-20.58-generic
RelatedPackageVersions:
 linux-backports-modules-2.6.31-20-generic N/A
 linux-firmware 1.26
RfKill:

Uname: Linux 2.6.31-20-generic x86_64
UserGroups: adm admin cdrom dialout libvirtd lpadmin plugdev sambashare vboxusers
WpaSupplicantLog:

dmi.bios.date: 01/14/2009
dmi.bios.vendor: Phoenix Technologies, LTD
dmi.bios.version: 6.00 PG
dmi.board.asset.tag: A1234-0
dmi.board.name: DFI LP JR X58
dmi.board.vendor: DFI Inc.
dmi.board.version: 1.0
dmi.chassis.type: 3
dmi.chassis.vendor: Enclosure Manufact
dmi.chassis.version: Enclosure Product
dmi.modalias: dmi:bvnPhoenixTechnologies,LTD:bvr6.00PG:bd01/14/2009:svnSystemManufact:pnSystemProduct:pvr1.0:rvnDFIInc.:rnDFILPJRX58:rvr1.0:cvnEnclosureManufact:ct3:cvrEnclosureProduct:
dmi.product.name: System Product
dmi.product.version: 1.0
dmi.sys.vendor: System Manufact

[Michael Joyner (michael-newsrx)]

mount |grep ext4
/dev/sda2 on / type ext4 (rw,errors=remount-ro,commit=0)
/dev/sdb1 on /media/EXTERNAL type ext4 (rw,nosuid,nodev,uhelper=udisks)

df -h /media/EXTERNAL/
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 917G 562G 309G 65% /media/EXTERNAL

touch /media/EXTERNAL/test.file
touch: cannot touch `/media/EXTERNAL/test.file': No space left on device

[Lisa Simpson (lisa-p)]

I am not on Karmic but Oneric and the problem is still present. I went to patch the kernel and started getting Out of disk space error messages from apt, aptitude, and dpkg.

When I looked at df, it was showing 83% usage. I couldn't understand why the disk was "full". Then I found this thread which describes the problem EXACTLY on Lucid - http://ubuntuforums.org/showthread.php?t=1454052

When I looked at df -i it was 100%. I had to remove some of the old kernels in order to complete the update. This freed up enough inodes that the update was finally successful.