Keystone becomes not operable if there is no connectivity on br-mgmt
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
Critical
|
Aleksandr Didenko | ||
6.0.x |
Invalid
|
High
|
Aleksandr Didenko |
Bug Description
{"build_id": "2015-03-
Steps to reproduce:
1. Deploy Ha on Centos with neutron
- 3 controllers
- 2 computes
2. When cluster ready run ostf ha, smoke and sanity suites
3. As soon as tests passed ssh on any controller and block input/output traffic on br-mgmt
4. Wait until cluster recovers after fail-over (I waiting for ~30 minutes)
5. manually check rabbitmq health and galera health, check crm
6. Try to login in horizon
Actual result:
Authorization failed. ssh on node and execute . openrc nova list. Command failed with 401 from keystone.
execute telnet to memcached on each controller. telned on controller where we block traffic failed(and it is expected), on other 2 controllers we can successfully connect to memcached.
On controller where we block traffic check haproxy backends for keystone
[root@node-5 ~]# haproxy-status | grep keystone
2015/03/30 10:17:58 socat[4902] E connect(3, AF=1 "/var/lib/
[root@node-5 ~]#
check haproxy-backends for keystone from healthy controller:
root@node-2 ~]# haproxy-status | grep keystone
keystone-1 FRONTEND Status: OPEN Sessions: 0 Rate: 0
keystone-1 node-2 Status: UP/L7OK Sessions: 0 Rate: 0
keystone-1 node-4 Status: UP/L7OK Sessions: 0 Rate: 0
keystone-1 node-5 Status: DOWN/L4TOUT Sessions: 0 Rate: 0
keystone-1 BACKEND Status: UP Sessions: 0 Rate: 0
keystone-2 FRONTEND Status: OPEN Sessions: 0 Rate: 0
keystone-2 node-2 Status: UP/L7OK Sessions: 0 Rate: 0
keystone-2 node-4 Status: UP/L7OK Sessions: 0 Rate: 0
keystone-2 node-5 Status: DOWN/L4TOUT Sessions: 0 Rate: 0
keystone-2 BACKEND Status: UP Sessions: 0 Rate: 0
[root@node-2 ~]#
(node-5 out node with blocked traffic)
run command . openrc nova-list one more time from healthy controller - result 401 from keystone
edit keystone.conf on both healthy controller - remove here node with failed memcached from section [memchache] and [cache], restart keystone on both controllers, run command . openrc keystone token get - it is passed
rum command . openrc nova -list - it failed with 401 error from keystone, user can not pass authorization in horizon, services also failed to communicate (according keystone send all the time 401 error)
http://
summary: |
- Keystone becomes not operatable if there is not connectivity on br-mgmt + Keystone becomes not operable if there is not connectivity on br-mgmt |
summary: |
- Keystone becomes not operable if there is not connectivity on br-mgmt + Keystone becomes not operable if there is no connectivity on br-mgmt |
Changed in fuel: | |
assignee: | Fuel Library Team (fuel-library) → Aleksandr Didenko (adidenko) |
Changed in fuel: | |
status: | Incomplete → In Progress |
Changed in fuel: | |
status: | In Progress → Fix Committed |
create the same issue on mos-keystone https:/ /bugs.launchpad .net/mos/ +bug/1438279 (Because I am not sure that it is library part), So guys could you pls take a look and if it is related only to keystone close as duplicate