keystone is down when rabbitmq in "Stopped" status

[Oleksiy Butenko]

Steps:

1. Deploy ha env with 3 controllers.
2. Destroy two controllers
3. On controller: check pcs status
( Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     Stopped: [ node-1.test.domain.local ])
3. Try get new token

Actual result:
Authorization Failed: Unable to establish connection to http://192.168.0.2:5000/v2.0/tokens

http://paste.openstack.org/show/485523/

[Oleksiy Butenko (obutenko)]

[root@nailgun ~]# cat /etc/fuel/8.0/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "478"
  build_id: "478"
  fuel-nailgun_sha: "ae949905142507f2cb446071783731468f34a572"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "481ed135de2cb5060cac3795428625befdd1d814"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "420c6fa5f8cb51f3322d95113f783967bde9836e"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "6c6b088a3d52dd0eaf43d59f3a3a149c93a07e7e

[Oleksiy Butenko (obutenko)]

Reproduced on env with one controller:
root@node-1:~# pcs resource disable p_rabbitmq-server
root@node-1:~# rabbitmqctl stop
root@node-1:~# keystone --debug token-get

On ISO 8.0 478, ISO 7.0 301

[Boris Bobrov (bbobrov)]

How is this a critical bug?

[Oleksiy Butenko (obutenko)]

ISO 7.0 301
http://paste.openstack.org/show/485525/
https://goo.gl/hi3VN2 - snapshot

[Roman Podoliaka (rpodolyaka)]

Oleksiy, I'm not saying we shouldn't find the root cause of this, but with 3 controllers running in HA mode we can only lose 1 - if 2 controllers are lost, it's like a split-brain, i.e. our cluster will not be available, and that's *expected*.

At the same time, this does not explain why Keystone goes down ( it is not managed by Pacemaker).

I suggest we downgrade this to Medium and do the RCA, as this use case is not supported and High/Critical seem to be an overkill to me.

[Timur Nurlygayanov (tnurlygayanov)]

Hi Roman, I agree with you about Medium priority of this issue because it is not a blocker for customers, if RabbitMQ dies all OpenStack services will crash.

But in the same time it looks like we should check the code of Keystone and identify why it crashed with tracebacks if it can't connect to RabbitMQ (other services don't crash in this case)

[Roman Podoliaka (rpodolyaka)]

We no longer fix Medium bugs in 8.0, closing as Won't Fix

[Alexander Makarov (amakarov)]

If keystone notifications use RabbitMQ as a transport then keystone will hang on sending the message - that's the expected behavior.

[Alexander Makarov (amakarov)]

What is the expected result?

[Dina Belova (dbelova)]

Added move-to-10.0 tag due to the fact bug was transferred from 9.0 to 10.0