Comment 45 for bug 1423672

I'm running LoCo mirror for my country and this affects our file server for over a week already.
All our official file services got offline now.
We're also providing free VM/Hosting for OpenSource projects, I can't image how the disaster will become.

Any suggestion?

All VMs are VMware guest.

root@ncnu-ftp:/ubuntu/mirror# LANG=C free -m
             total used free shared buffers cached
Mem: 7984 1453 6531 0 66 352
-/+ buffers/cache: 1033 6950
Swap: 4091 0 4091

root@ncnu-ftp:/ubuntu/mirror# LANG=C touch /lala
touch: cannot touch '/lala': Read-only file system

root@ncnu-ftp:/ubuntu/mirror# LANG=C mount -o remount,rw /
mount: cannot remount block device /dev/mapper/ub--ftp-root read-write, is write-protected

root@ncnu-ftp:/ubuntu/mirror# dumpe2fs /dev/mapper/ub--ftp-root /dev/|grep -i Filesystem\ state
dumpe2fs 1.42.9 (4-Feb-2014)
Filesystem state: clean with errors

root@ncnu-ftp:/ubuntu/mirror# LC_ALL=C dmesg -T |grep error
[Thu Nov 5 00:35:46 2015] EXT4-fs (dm-0): re-mounted. Opts: errors=remount-ro
[Thu Nov 5 00:35:55 2015] EXT4-fs error (device dm-0): ext4_mb_generate_buddy:756: group 371, block bitmap and bg descriptor inconsistent: 25478 vs 25477 free clusters
[Thu Nov 5 00:35:55 2015] IP: [<ffffffff8125e511>] __ext4_error_inode+0x31/0x160
[Thu Nov 5 00:35:55 2015] RIP: 0010:[<ffffffff8125e511>] [<ffffffff8125e511>] __ext4_error_inode+0x31/0x160
[Thu Nov 5 00:35:55 2015] RIP [<ffffffff8125e511>] __ext4_error_inode+0x31/0x160
[Thu Nov 5 00:35:55 2015] EXT4-fs error (device dm-0): ext4_remount:4816: Abort forced by user
[Thu Nov 5 00:40:39 2015] EXT4-fs (dm-0): error count since last fsck: 15
[Thu Nov 5 00:40:39 2015] EXT4-fs (dm-0): initial error at time 1445904785: ext4_reserve_inode_write:4928
[Thu Nov 5 00:40:39 2015] EXT4-fs (dm-0): last error at time 1446654956: ext4_remount:4816
[Thu Nov 5 01:18:38 2015] EXT4-fs error (device dm-0): ext4_remount:4816: Abort forced by user
[Thu Nov 5 01:18:59 2015] EXT4-fs error (device dm-0): ext4_remount:4816: Abort forced by user
[Thu Nov 5 01:28:59 2015] EXT4-fs error (device dm-0): ext4_remount:4816: Abort forced by user
[Thu Nov 5 01:58:08 2015] EXT4-fs error (device dm-0): ext4_remount:4816: Abort forced by user
[Thu Nov 5 02:20:41 2015] EXT4-fs error (device dm-0): ext4_remount:4816: Abort forced by user

root@ncnu-ftp:/ubuntu/mirror# uname -a
Linux ncnu-ftp 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

root@ncnu-ftp:/ubuntu/mirror# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty