df reports negative disk usage

[tor-henning]

Binary package hint: coreutils

I am in the process of converting a md-RAID5 to RAID6, when checking df during the process i notice that the disk usage is negative:

http://pastebin.com/YHcMjiD7

A du -hs on the mounted device gives:

~$ sudo du -hs /home/samba/
2.1T /home/samba/

~$ lsb_release -rd
Description: Ubuntu 10.04.1 LTS
Release: 10.04

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: coreutils 7.4-2ubuntu3
ProcVersionSignature: Ubuntu 2.6.32-27.49-server 2.6.32.26+drm33.12
Uname: Linux 2.6.32-27-server x86_64
Architecture: amd64
Date: Fri Feb 11 18:56:58 2011
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: coreutils

[C de-Avillez (hggdh2)]

Adding the pastebin localy for survival:

torhenning@bais:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdf1 352G 205G 130G 62% /
none 4.0G 224K 4.0G 1% /dev
none 4.0G 0 4.0G 0% /dev/shm
none 4.0G 512K 4.0G 1% /var/run
none 4.0G 0 4.0G 0% /var/lock
none 4.0G 0 4.0G 0% /lib/init/rw
none 352G 205G 130G 62% /var/lib/ureadahead/debugfs
/dev/md0 5.4T -177G 5.6T - /home/samba

[tor-henning (tor-henning)]

dmesg reports successful reshape:

[158623.944746] md: md0: reshape done.
[158624.630207] RAID5 conf printout:
[158624.630217] --- rd:11 wd:10
[158624.630223] disk 0, o:1, dev:sdk
[158624.630228] disk 1, o:1, dev:sda
[158624.630232] disk 2, o:1, dev:sdi
[158624.630236] disk 3, o:1, dev:sdb
[158624.630240] disk 4, o:1, dev:sdg
[158624.630244] disk 5, o:1, dev:sdc
[158624.630248] disk 6, o:1, dev:sde
[158624.630253] disk 7, o:1, dev:sdd
[158624.630256] disk 8, o:1, dev:sdj
[158624.630261] disk 9, o:1, dev:sdh
[158624.630273] md0: detected capacity change from 6001207017472 to 13502715789312
[158624.729814] VFS: busy inodes on changed media or resized disk md0

df still reports negative numbers.

[tor-henning (tor-henning)]

I unmounted the device and performed fsck on it, which detected a nice bunch of problems:

Free blocks count wrong for group #5641 (32265, counted=11).
Fix<y>? yes

Free blocks count wrong for group #5642 (32311, counted=57).
Fix<y>? yes

Free blocks count wrong for group #5643 (33697, counted=1449).
Fix<y>? yes

Free blocks count wrong for group #5644 (34162, counted=1908).
Fix<y>? yes

Free blocks count wrong for group #5645 (36278, counted=4024).
Fix<y>? no

Free blocks count wrong for group #5646 (33303, counted=1049).
Fix<y>? no

Free blocks count wrong for group #5647 (32276, counted=22).
Fix<y>?

*etc*

After completing fsck and remounting the device, df has started displaying correct size again.

[Dave Gilbert (ubuntu-treblig)]

Can you describe the exact set of steps you took performing the conversion from RAID5 to RAID6 and the filesystem type please.

Dave

[tor-henning (tor-henning)]

It went something like this:

cat /proc/mdstat
sudo mdadm --stop /dev/md1
mdadm --zero-superblock /dev/sdh
mdadm --zero-superblock /dev/sdj
mdadm --zero-superblock /dev/sdd
mdadm --zero-superblock /dev/sde
mdadm --zero-superblock /dev/sdc
sudo mdadm --remove /dev/md1
umount /dev/md0
fsck.ext3 /dev/md0

mdadm --add /dev/md0 /dev/sdh
mdadm --add /dev/md0 /dev/sdk
mdadm --add /dev/md0 /dev/sdj
mdadm --add /dev/md0 /dev/sdd
mdadm --add /dev/md0 /dev/sdde
mdadm --add /dev/md0 /dev/sde
mdadm --add /dev/md0 /dev/sdc
mdadm --add missing

I then tried some grow commands, before i noticed that the given version of mdadm does not support grow, so i did a:

wget http://ftp.at.debian.org/debian/pool/main/m/mdadm/mdadm_3.1.4-1+8efb9d1_amd64.deb
dpkg -i mdadm_3.1.4-1+8efb9d1_amd64.deb

mdadm --grow /dev/md0 --level=6 --raid-devices=11

Once completed i did a:
resize2fs /dev/md0 12570G

After i re-mounted the device, the issue was gone.

I also noticed that munin reported negative disk usage, so it appears to be a issue with mdadm/the device, and *not* df.

[Launchpad Janitor (janitor)]

[Expired for coreutils (Ubuntu) because there has been no activity for 60 days.]