flag@c13:~$ less /var/log/swift/storage1.error
...
May 14 15:33:27 c13 container-server ERROR __call__ error with PUT /sdb1/62777/AUTH_test/777cd4065dc9446da68d511a1aeb5e8a_1 : [Errno 28] No space left on device: '/srv/1/node/sdb1/containers/62777' (txn: tx276b80075bb9429d9bebb-005192916a)
Try this kernel:
http:// people. canonical. com/~ppisati/ linux-image- 3.5.0-29- highbank_ 3.5.0-29. 49~7c4cebe8_ armhf.deb
it's a Q/generic highbank kernel + the attached patch and it fixes the fs corruption.
On the other hand, even with this fix applied, when the xfs partition reaches ~25% of its capacity, a test like this:
while /bin/true; do swift-bench -x -c 20 -s 10 -n 100 -g 100 /etc/swift/ swift-bench. conf; done
collapses and swift-storage node reports 'out of space' while there's clearly still space and inodes (see /dev/sda5 below):
flag@c13:~$ uname -a
Linux c13 3.5.0-29-highbank #49~7c4cebe8 SMP Tue May 14 18:44:12 UTC 2013 armv7l armv7l armv7l GNU/Linux
flag@c13:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 225G 2.1G 212G 1% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 2.0G 4.0K 2.0G 1% /dev
tmpfs 405M 260K 404M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda1 228M 30M 186M 14% /boot
/dev/sda5 2.0G 569M 1.5G 28% /mnt/sdb1
flag@c13:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 14958592 74462 14884130 1% /
none 182027 1 182026 1% /sys/fs/cgroup
udev 177378 1361 176017 1% /dev
tmpfs 182027 807 181220 1% /run
none 182027 3 182024 1% /run/lock
none 182027 1 182026 1% /run/shm
none 182027 1 182026 1% /run/user
/dev/sda1 124496 35 124461 1% /boot
/dev/sda5 524288 237184 287104 46% /mnt/sdb1
flag@c13:~$ less /var/log/ swift/storage1. error AUTH_test/ 777cd4065dc9446 da68d511a1aeb5e 8a_1 : [Errno 28] No space left on device: '/srv/1/ node/sdb1/ containers/ 62777' (txn: tx276b80075bb94 29d9bebb- 005192916a)
...
May 14 15:33:27 c13 container-server ERROR __call__ error with PUT /sdb1/62777/
we might have uncovered another issue here.