Keystone backends are down after filling root filesystem on main controller
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Confirmed
|
Wishlist
|
MOS Keystone | ||
Mitaka |
Won't Fix
|
Wishlist
|
MOS Keystone |
Bug Description
Steps to reproduce:
1. Deploy any environment with 3 controllers and NeutronTUN or NeutronVLAN
2. Fill root filesystem on primary controller. For example, "fallocate -l 10000G /bigfile"
3. Wait for ~5 min
4. Verify networks fail (mcollective aggent fails on primary controller, network checker returns error: "Reason: systemu: Error - process interrupted!")
5. Run OSTF tests <<< FAIL
Expected result: OSTF tests passed
Actual result: All tests failed
In /var/log/
[Wed Jun 08 15:22:02.028505 2016] [wsgi:error] [pid 54268:tid 140593522988800] (11)Resource temporarily unavailable: [client 10.109.3.3:46881] mod_wsgi (pid=54268): Unable to connect to WSGI daemon process 'keystone_main' on '/var/run/
[Wed Jun 08 15:22:03.192453 2016] [wsgi:error] [pid 53630:tid 140593756165888] (11)Resource temporarily unavailable: [client 10.109.3.6:52884] mod_wsgi (pid=53630): Unable to connect to WSGI daemon process 'keystone_main' on '/var/run/
root@node-2:~# haproxy-status.sh | grep DOWN
keystone-1 node-2 Status: DOWN/L7TOUT Sessions: 0 Rate: 0
keystone-1 node-3 Status: DOWN/L7TOUT Sessions: 0 Rate: 0
keystone-1 node-4 Status: DOWN/L7TOUT Sessions: 0 Rate: 0
keystone-1 BACKEND Status: DOWN Sessions: 0 Rate: 0
Changed in fuel: | |
milestone: | none → 10.0 |
importance: | Undecided → High |
assignee: | nobody → MOS Keystone (mos-keystone) |
tags: | added: area-keystone |
Changed in fuel: | |
milestone: | 10.0 → 9.1 |
description: | updated |
Changed in fuel: | |
milestone: | 9.1 → 9.2 |
[root@nailgun ~]# shotgun2 short-report build_number: openstack_ version: network- checker| nailgun| packetary| shotgun' : 9.0.0-1. mos6347. noarch 9.0.0-1. mos8415. noarch -cli-9. 0.0-1.mos284. noarch 9.0.0-1. mos8415. noarch astute- 9.0.0-1. mos747. noarch ing-scripts- 9.0.0-1. mos8718. noarch checker- 9.0.0-1. mos72.x86_ 64 9.0.0-1. mos137. noarch -metadata- 9.0.0-1. mos8718. noarch 9.0.0-1. mos8415. noarch mcagents- 9.0.0-1. mos747. noarch fuelclient- 9.0.0-1. mos319. noarch 9.0.0-1. mos270. noarch 0.0-1.mos6347. noarch 9.0.0-1. mos8415. noarch 9.0.0-1. mos6347. noarch 0-9.0.0- 1.mos8415. noarch 9.0.0-1. mos90.noarch 9.0.0-1. mos284. noarch 9.0.0-1. mos2710. noarch 9.0.0-1. mos934. noarch packetary- 9.0.0-1. mos137. noarch 9.0.0-1. mos8718. noarch
cat /etc/fuel_build_id:
427
cat /etc/fuel_
427
cat /etc/fuel_release:
9.0
cat /etc/fuel_
mitaka-9.0
rpm -qa | egrep 'fuel|astute|
fuel-release-
fuel-misc-
fuel-bootstrap
fuel-migrate-
rubygem-
fuel-provision
network-
fuel-mirror-
fuel-openstack
fuel-notify-
nailgun-
python-
fuelmenu-
fuel-9.
fuel-utils-
fuel-setup-
fuel-library9.
shotgun-
fuel-agent-
fuel-ui-
fuel-ostf-
python-
fuel-nailgun-