Keystone backends are down after filling root filesystem on main controller

[Volodymyr Shypyguzov]

Steps to reproduce:
1. Deploy any environment with 3 controllers and NeutronTUN or NeutronVLAN
2. Fill root filesystem on primary controller. For example, "fallocate -l 10000G /bigfile"
3. Wait for ~5 min
4. Verify networks fail (mcollective aggent fails on primary controller, network checker returns error: "Reason: systemu: Error - process interrupted!")
5. Run OSTF tests <<< FAIL

Expected result: OSTF tests passed
Actual result: All tests failed

In /var/log/apache2/keystone_wsgi_main_error.log:

[Wed Jun 08 15:22:02.028505 2016] [wsgi:error] [pid 54268:tid 140593522988800] (11)Resource temporarily unavailable: [client 10.109.3.3:46881] mod_wsgi (pid=54268): Unable to connect to WSGI daemon process 'keystone_main' on '/var/run/apache2/wsgi.46010.4.2.sock' after multiple attempts as listener backlog limit was exceeded.
[Wed Jun 08 15:22:03.192453 2016] [wsgi:error] [pid 53630:tid 140593756165888] (11)Resource temporarily unavailable: [client 10.109.3.6:52884] mod_wsgi (pid=53630): Unable to connect to WSGI daemon process 'keystone_main' on '/var/run/apache2/wsgi.46010.4.2.sock' after multiple attempts as listener backlog limit was exceeded.

root@node-2:~# haproxy-status.sh | grep DOWN
keystone-1 node-2 Status: DOWN/L7TOUT Sessions: 0 Rate: 0
keystone-1 node-3 Status: DOWN/L7TOUT Sessions: 0 Rate: 0
keystone-1 node-4 Status: DOWN/L7TOUT Sessions: 0 Rate: 0
keystone-1 BACKEND Status: DOWN Sessions: 0 Rate: 0

[Volodymyr Shypyguzov (vshypyguzov)]

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 427
cat /etc/fuel_build_number:
 427
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6347.noarch
 fuel-misc-9.0.0-1.mos8415.noarch
 fuel-bootstrap-cli-9.0.0-1.mos284.noarch
 fuel-migrate-9.0.0-1.mos8415.noarch
 rubygem-astute-9.0.0-1.mos747.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8718.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-mirror-9.0.0-1.mos137.noarch
 fuel-openstack-metadata-9.0.0-1.mos8718.noarch
 fuel-notify-9.0.0-1.mos8415.noarch
 nailgun-mcagents-9.0.0-1.mos747.noarch
 python-fuelclient-9.0.0-1.mos319.noarch
 fuelmenu-9.0.0-1.mos270.noarch
 fuel-9.0.0-1.mos6347.noarch
 fuel-utils-9.0.0-1.mos8415.noarch
 fuel-setup-9.0.0-1.mos6347.noarch
 fuel-library9.0-9.0.0-1.mos8415.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-agent-9.0.0-1.mos284.noarch
 fuel-ui-9.0.0-1.mos2710.noarch
 fuel-ostf-9.0.0-1.mos934.noarch
 python-packetary-9.0.0-1.mos137.noarch
 fuel-nailgun-9.0.0-1.mos8718.noarch

[Volodymyr Shypyguzov (vshypyguzov)]

Diagnostic snapshot https://drive.google.com/a/mirantis.com/file/d/0B7rLGkmwHi0ONktXSDdlZGVES28/view?usp=sharing

[Dina Belova (dbelova)]

after conversation with keystone team, making it a wishlist and marking as won't fix for 9.0. Due to their comment zero space on disk is gonna make things stop working - we may be out of RAM+swap... etc.

[Boris Bobrov (bbobrov)]

Yep. Things don't work with 0 disk space. The same way they don't work with 0 RAM+swap. If you want things to work, do not create 10TB files on your controller.

[Volodymyr Shypyguzov (vshypyguzov)]

And what if this happens accidentally? I just don't get it, how this could be called HA if running out of disk space brakes all cluster?