Comment 4 for bug 1414098

I believe I found the underlying issue here, this is a df of the root filesystem on each of our test env hosts

/dev/sda1 845G 808G 586M 100% /
/dev/sda1 845G 808G 556M 100% /
/dev/sda1 845G 805G 4.3G 100% /
/dev/sda1 845G 809G 55M 100% /
/dev/sda1 845G 809G 21M 100% /
/dev/sda1 845G 809G 264M 100% /
/dev/sda1 845G 805G 4.0G 100% /
/dev/sda1 845G 627G 183G 78% /
/dev/sda1 845G 549G 261G 68% /
/dev/sda1 845G 653G 157G 81% /
/dev/sda1 845G 784G 26G 97% /
/dev/sda1 845G 600G 209G 75% /
/dev/sda1 845G 696G 113G 87% /
/dev/sda1 845G 559G 251G 70% /

I'm going to do a little prep and then re provision them all, with disk size a less over subscribed

For reference the nodes that are full are running 6 testenvs each for the past 4 months and most of the space is being used up in /var/lib/libvirt/images

[heat-admin@testenv-testenv2-24jhfjm5ohp3 images]$ sudo du -sh /var/lib/libvirt/images/
806G /var/lib/libvirt/images/

[heat-admin@testenv-testenv2-24jhfjm5ohp3 images]$ sudo ls -lSr /var/lib/libvirt/images/
total 840080760
-rw-r--r--. 1 root root 1800011776 Feb 2 11:54 seed_4.qcow2
-rw-------. 1 root root 5409996800 Dec 16 00:58 baremetalbrbm1_6.qcow2
-rw-------. 1 root root 5525667840 Dec 10 10:10 baremetalbrbm6_10.qcow2
-rw-------. 1 root root 5742329856 Dec 16 00:52 baremetalbrbm6_4.qcow2
...
-rw-------. 1 root root 12849381376 Dec 16 17:23 baremetalbrbm1_3.qcow2
-rw-------. 1 root root 13022134272 Jan 15 00:19 baremetalbrbm5_13.qcow2
-rw-r--r--. 1 qemu qemu 13036879872 Feb 2 11:54 seed_6.qcow2
-rw-------. 1 qemu qemu 13138984960 Feb 2 11:54 baremetalbrbm6_3.qcow2
-rw-------. 1 root root 13518569472 Jan 7 22:04 baremetalbrbm6_0.qcow2
-rw-------. 1 root root 13648461824 Feb 2 08:36 baremetalbrbm2_3.qcow2
-rw-------. 1 root root 13943177216 Dec 17 21:25 baremetalbrbm3_5.qcow2
-rw-------. 1 root root 14780203008 Dec 16 03:52 baremetalbrbm6_7.qcow2
-rw-------. 1 root root 15622406144 Jan 14 22:59 baremetalbrbm3_2.qcow2
-rw-------. 1 root root 15805775872 Dec 17 21:25 baremetalbrbm3_14.qcow2
-rw-------. 1 root root 16497836032 Dec 16 12:39 baremetalbrbm1_0.qcow2
-rw-r--r--. 1 root root 17180000256 Feb 2 10:05 seed_1.qcow2
-rw-------. 1 root root 17854627840 Feb 2 09:55 baremetalbrbm4_0.qcow2
-rw-------. 1 qemu qemu 18130403328 Feb 2 11:54 baremetalbrbm6_12.qcow2
-rw-------. 1 root root 18411356160 Jan 15 00:19 baremetalbrbm5_0.qcow2
-rw-------. 1 root root 26098335744 Feb 1 19:31 baremetalbrbm2_0.qcow2