Comment 6 for bug 1176977

Revision history for this message
Paolo Pisati (p-pisati) wrote :

Try this kernel:

http://people.canonical.com/~ppisati/linux-image-3.5.0-29-highbank_3.5.0-29.49~7c4cebe8_armhf.deb

it's a Q/generic highbank kernel + the attached patch and it fixes the fs corruption.

On the other hand, even with this fix applied, when the xfs partition reaches ~25% of its capacity, a test like this:

while /bin/true; do swift-bench -x -c 20 -s 10 -n 100 -g 100 /etc/swift/swift-bench.conf; done

collapses and swift-storage node reports 'out of space' while there's clearly still space and inodes (see /dev/sda5 below):

flag@c13:~$ uname -a
Linux c13 3.5.0-29-highbank #49~7c4cebe8 SMP Tue May 14 18:44:12 UTC 2013 armv7l armv7l armv7l GNU/Linux

flag@c13:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 225G 2.1G 212G 1% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 2.0G 4.0K 2.0G 1% /dev
tmpfs 405M 260K 404M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda1 228M 30M 186M 14% /boot
/dev/sda5 2.0G 569M 1.5G 28% /mnt/sdb1

flag@c13:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 14958592 74462 14884130 1% /
none 182027 1 182026 1% /sys/fs/cgroup
udev 177378 1361 176017 1% /dev
tmpfs 182027 807 181220 1% /run
none 182027 3 182024 1% /run/lock
none 182027 1 182026 1% /run/shm
none 182027 1 182026 1% /run/user
/dev/sda1 124496 35 124461 1% /boot
/dev/sda5 524288 237184 287104 46% /mnt/sdb1

flag@c13:~$ less /var/log/swift/storage1.error
...
May 14 15:33:27 c13 container-server ERROR __call__ error with PUT /sdb1/62777/AUTH_test/777cd4065dc9446da68d511a1aeb5e8a_1 : [Errno 28] No space left on device: '/srv/1/node/sdb1/containers/62777' (txn: tx276b80075bb9429d9bebb-005192916a)

we might have uncovered another issue here.