Comment 7 for bug 1840182

The VIM on controller-0 is failing to connect to the nova notifications queue:

NotFound: Queue.declare: (404) NOT_FOUND - home node '<email address hidden>' of durable queue 'notifications.nfvi_nova_listener_queue' in vhost 'nova' is down or inaccessible

This error indicates that the queue exists, but cannot be connected to. The rabbitmq logs indicate that rabbitmq is shutting down on controller-0:

{"log":"2019-08-14 15:58:15.535 [info] \u003c0.472.0\u003e Autoheal request received from '<email address hidden>'\n","stream":"stdout","time":"2019-08-14T15:58:15.535800761Z"}
{"log":"2019-08-14 15:58:15.536 [info] \u003c0.472.0\u003e Autoheal decision\n","stream":"stdout","time":"2019-08-14T15:58:15.536559958Z"}
{"log":" * Partitions: [['<email address hidden>'],['<email address hidden>']]\n","stream":"stdout","time":"2019-08-14T15:58:15.536586158Z"}
{"log":" * Winner: '<email address hidden>'\n","stream":"stdout","time":"2019-08-14T15:58:15.536601564Z"}
{"log":" * Losers: ['<email address hidden>']\n","stream":"stdout","time":"2019-08-14T15:58:15.536613184Z"}
{"log":"2019-08-14 15:58:15.536 [warning] \u003c0.472.0\u003e Autoheal: we were selected to restart; winner is '<email address hidden>'\n","stream":"stdout","time":"2019-08-14T15:58:15.537049161Z"}
{"log":"2019-08-14 15:58:15.536 [info] \u003c0.20917.1\u003e RabbitMQ is asked to stop...\n","stream":"stdout","time":"2019-08-14T15:58:15.537061637Z"}
{"log":"2019-08-14 15:58:15.585 [info] \u003c0.20917.1\u003e Stopping RabbitMQ applications and their dependencies in the following order:\n","stream":"stdout","time":"2019-08-14T15:58:15.585461935Z"}
{"log":" rabbitmq_management\n","stream":"stdout","time":"2019-08-14T15:58:15.585474435Z"}
{"log":" amqp_client\n","stream":"stdout","time":"2019-08-14T15:58:15.585478042Z"}
{"log":" rabbitmq_web_dispatch\n","stream":"stdout","time":"2019-08-14T15:58:15.585481157Z"}
{"log":" cowboy\n","stream":"stdout","time":"2019-08-14T15:58:15.585484303Z"}
{"log":" cowlib\n","stream":"stdout","time":"2019-08-14T15:58:15.585491628Z"}
{"log":" rabbitmq_peer_discovery_k8s\n","stream":"stdout","time":"2019-08-14T15:58:15.58549482Z"}
{"log":" rabbitmq_peer_discovery_common\n","stream":"stdout","time":"2019-08-14T15:58:15.585497784Z"}
{"log":" rabbitmq_management_agent\n","stream":"stdout","time":"2019-08-14T15:58:15.585500727Z"}
{"log":" rabbit\n","stream":"stdout","time":"2019-08-14T15:58:15.585503554Z"}
{"log":" mnesia\n","stream":"stdout","time":"2019-08-14T15:58:15.585506253Z"}
{"log":" rabbit_common\n","stream":"stdout","time":"2019-08-14T15:58:15.585508974Z"}
{"log":" sysmon_handler\n","stream":"stdout","time":"2019-08-14T15:58:15.585512513Z"}
{"log":" os_mon\n","stream":"stdout","time":"2019-08-14T15:58:15.58551719Z"}
{"log":"2019-08-14 15:58:15.585 [info] \u003c0.20917.1\u003e Stopping application 'rabbitmq_management'\n","stream":"stdout","time":"2019-08-14T15:58:15.585521823Z"}
{"log":"2019-08-14 15:58:15.587 [warning] \u003c0.959.0\u003e RabbitMQ HTTP listener registry could not find context rabbitmq_management_tls\n","stream":"stdout","time":"2019-08-14T15:58:15.587999948Z"}
{"log":"2019-08-14 15:58:15.590 [info] \u003c0.20917.1\u003e Stopping application 'amqp_client'\n","stream":"stdout","time":"2019-08-14T15:58:15.591123618Z"}
{"log":"2019-08-14 15:58:15.592 [info] \u003c0.20917.1\u003e Stopping application 'rabbitmq_web_dispatch'\n","stream":"stdout","time":"2019-08-14T15:58:15.592915835Z"}
{"log":"2019-08-14 15:58:15.594 [info] \u003c0.20917.1\u003e Stopping application 'cowboy'\n","stream":"stdout","time":"2019-08-14T15:58:15.594802094Z"}
{"log":"2019-08-14 15:58:15.596 [info] \u003c0.20917.1\u003e Stopping application 'cowlib'\n","stream":"stdout","time":"2019-08-14T15:58:15.596375611Z"}
{"log":"2019-08-14 15:58:15.596 [info] \u003c0.20917.1\u003e Stopping application 'rabbitmq_peer_discovery_k8s'\n","stream":"stdout","time":"2019-08-14T15:58:15.596404374Z"}
{"log":"2019-08-14 15:58:15.596 [info] \u003c0.20917.1\u003e Stopping application 'rabbitmq_peer_discovery_common'\n","stream":"stdout","time":"2019-08-14T15:58:15.596423837Z"}
{"log":"2019-08-14 15:58:15.597 [info] \u003c0.20917.1\u003e Stopping application 'rabbitmq_management_agent'\n","stream":"stdout","time":"2019-08-14T15:58:15.598089513Z"}
{"log":"2019-08-14 15:58:15.602 [info] \u003c0.20917.1\u003e Stopping application 'rabbit'\n","stream":"stdout","time":"2019-08-14T15:58:15.602731143Z"}
{"log":"2019-08-14 15:58:15.602 [info] \u003c0.282.0\u003e Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping unregistration.\n","stream":"stdout","time":"2019-08-14T15:58:15.6027601Z"}

Further investigation is required to understand what is wrong with rabbitmq on controller-0.