Keystone fails to respond to auth request when rabbitmq is having problems
Bug #1615127 reported by
Eugene Nikanorov
This bug affects 1 person
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Mirantis OpenStack |
Won't Fix
|
High
|
MOS Oslo | ||
| 7.0.x |
Won't Fix
|
High
|
MOS Maintenance | ||
| 8.0.x |
Won't Fix
|
High
|
MOS Maintenance | ||
| 9.x |
Won't Fix
|
High
|
MOS Oslo | ||
Bug Description
Encountered on MOS 7
Keystone fails to respond to auth requests in case it can't push a notification to the rabbitmq.
Instead of giving any kind of an error, it silently hangs. It is detected by apache which then gives 503.
The behavior is consistent with keystone running as a standalone process.
Restarting the service did not fix things, which indicate that it has something to do with rabbitmq state.
However in this case rabbitmq seem to be running and doing fine by all usual means (pcs status, rabbitmqctl cluster_status, rabbitmqctl list_queues)
Restarting rabbitmq solved the problem.
I'm attaching rabbitmq logs. The issue was observed on aug 17.
Revision history for this message
| Eugene Nikanorov (enikanorov) wrote | #1 |
| tags: | added: area-oslo |
| description: | updated |
Revision history for this message
| Dmitry Mescheryakov (dmitrymex) wrote | #2 |
| Changed in mos: | |
| status: | Confirmed → Won't Fix |
To post a comment you must log in.
Eugene, this is as designed. The notifications are meant to be reliable. If a service can not send a notification, it should not perform any action. Besides, what is the point in having Keystone operational if the rest of the services are down, as they can't work without RabbitMQ at all? The only reason I see is to ease troubleshoot of Keystone issues, but I am sure it does not worth it. Instead we should increase oslo.messaging logging in Keystone to a level, on which we will see that it fails because it can't connect to RabbitMQ.