It seems that even the services initially able to connect the rabbitmq, somehow degraded and lost the connectivity, for example here is a sippet for ironic-conductor:
2018-06-22 14:43:33.867 6 INFO oslo.messaging._drivers.impl_rabbit [-] [d4784c4c-0857-4659-aed9-61a5fdca1cf2] Reconnected to AMQP server on undercloud.internalapi.localdomain:5672 via [amqp] client with port 54148.
...
2018-06-22 14:49:40.124 6 ERROR oslo.messaging._drivers.impl_rabbit [req-2aa9f436-87a2-4e16-aaf3-cd3529292b26 - - - - -] Unable to connect to AMQP server on undercloud.internalapi.localdomain:5672 after None tries: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.: AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
It seems that even the services initially able to connect the rabbitmq, somehow degraded and lost the connectivity, for example here is a sippet for ironic-conductor:
2018-06-22 14:43:33.867 6 INFO oslo.messaging. _drivers. impl_rabbit [-] [d4784c4c- 0857-4659- aed9-61a5fdca1c f2] Reconnected to AMQP server on undercloud. internalapi. localdomain: 5672 via [amqp] client with port 54148. _drivers. impl_rabbit [req-2aa9f436- 87a2-4e16- aaf3-cd3529292b 26 - - - - -] Unable to connect to AMQP server on undercloud. internalapi. localdomain: 5672 after None tries: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.: AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
...
2018-06-22 14:49:40.124 6 ERROR oslo.messaging.