ACCESS_REFUSED on oslo service

[Mohammad]

Hi,
At first I configured my neutron service with two network (Provider and Selfservice) and I created a router for this purpose. The I got connection problem in instance I made and provider network.

To solve the problem I decided to remove router and Selfservie network. To do so first I deleted the Selfservie ports and then try to delete the router, but I got error for doing that.

After that my I get errors of oslo service in all of Openstack services like nova and cinder and RabbitMQ and etc.

Now I get error for command below:

    # openstack router list
    ResourceNotFound: 404: Client Error for url: http://10.0.0.101:9696/v2.0/routers, The resource could not be found.

-------------------------------------------------------------------------
nova-compute logs:

    2019-11-28 10:36:07.729 37106 ERROR nova (class_id, method_id), ConnectionError)
    2019-11-28 10:36:07.729 37106 ERROR nova amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
    2019-11-28 10:36:07.729 37106 ERROR nova
    2019-11-28 10:36:10.507 37136 INFO os_vif [-] Loaded VIF plugins: linux_bridge, noop, ovs
    2019-11-28 10:36:11.093 37136 CRITICAL nova [req-ed629209-bac1-47df-bff0-63d638cd0dc7 - - - - -] Unhandled error: amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.

    2019-11-28 10:36:11.093 37136 ERROR nova return self.connection
    2019-11-28 10:36:11.093 37136 ERROR nova File "/usr/lib/python3/dist-packages/kombu/connection.py", line 802, in connection
    2019-11-28 10:36:11.093 37136 ERROR nova self._connection = self._establish_connection()
    2019-11-28 10:36:11.093 37136 ERROR nova File "/usr/lib/python3/dist-packages/kombu/connection.py", line 757, in _establish_connection
    2019-11-28 10:36:11.093 37136 ERROR nova conn = self.transport.establish_connection()
    2019-11-28 10:36:11.093 37136 ERROR nova File "/usr/lib/python3/dist-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    2019-11-28 10:36:11.093 37136 ERROR nova conn.connect()
    2019-11-28 10:36:11.093 37136 ERROR nova File "/usr/lib/python3/dist-packages/amqp/connection.py", line 313, in connect
    2019-11-28 10:36:11.093 37136 ERROR nova self.drain_events(timeout=self.connect_timeout)
    2019-11-28 10:36:11.093 37136 ERROR nova File "/usr/lib/python3/dist-packages/amqp/connection.py", line 500, in drain_events
    2019-11-28 10:36:11.093 37136 ERROR nova while not self.blocking_read(timeout):
    2019-11-28 10:36:11.093 37136 ERROR nova File "/usr/lib/python3/dist-packages/amqp/connection.py", line 506, in blocking_read
    2019-11-28 10:36:11.093 37136 ERROR nova return self.on_inbound_frame(frame)
    2019-11-28 10:36:11.093 37136 ERROR nova File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 55, in on_frame
    2019-11-28 10:36:11.093 37136 ERROR nova callback(channel, method_sig, buf, None)
    2019-11-28 10:36:11.093 37136 ERROR nova File "/usr/lib/python3/dist-packages/amqp/connection.py", line 510, in on_inbound_method
    2019-11-28 10:36:11.093 37136 ERROR nova method_sig, payload, content,
    2019-11-28 10:36:11.093 37136 ERROR nova File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 126, in dispatch_method
    2019-11-28 10:36:11.093 37136 ERROR nova listener(*args)
    2019-11-28 10:36:11.093 37136 ERROR nova File "/usr/lib/python3/dist-packages/amqp/connection.py", line 639, in _on_close
    2019-11-28 10:36:11.093 37136 ERROR nova (class_id, method_id), ConnectionError)
    2019-11-28 10:36:11.093 37136 ERROR nova amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
    2019-11-28 10:36:11.093 37136 ERROR nova
    2019-11-28 10:36:14.182 37167 INFO os_vif [-] Loaded VIF plugins: linux_bridge, noop, ovs
    2019-11-28 10:36:14.907 37167 ERROR oslo.messaging._drivers.impl_rabbit [req-ccbadfdf-adcf-468d-932c-3bf35b8be403 - - - - -] Connection failed: [Errno 111] ECONNREFUSED (retrying in 2.0 seconds): ConnectionRefusedError: [Errno 111] ECONNREFUSED
    2019-11-28 10:36:16.925 37167 ERROR oslo.messaging._drivers.impl_rabbit [req-ccbadfdf-adcf-468d-932c-3bf35b8be403 - - - - -] Connection failed: [Errno 111] ECONNREFUSED (retrying in 4.0 seconds): ConnectionRefusedError: [Errno 111] ECONNREFUSED
    2019-11-28 10:36:20.946 37167 ERROR oslo.messaging._drivers.impl_rabbit [req-ccbadfdf-adcf-468d-932c-3bf35b8be403 - - - - -] Connection failed: [Errno 111] ECONNREFUSED (retrying in 6.0 seconds): ConnectionRefusedError: [Errno 111] ECONNREFUSED
    3bf35b8be403 - - - - -] Connection failed: timed out (retrying in 20.0 seconds): socket.timeout: timed out

-------------------------------------------------------------------------
Cinder-scheduler logs:

    2019-11-30 12:44:40.162 12645 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 55, in on_frame
    2019-11-30 12:44:40.162 12645 ERROR oslo_service.service callback(channel, method_sig, buf, None)
    2019-11-30 12:44:40.162 12645 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/amqp/connection.py", line 510, in on_inbound_method
    2019-11-30 12:44:40.162 12645 ERROR oslo_service.service method_sig, payload, content,
    2019-11-30 12:44:40.162 12645 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 126, in dispatch_method
    2019-11-30 12:44:40.162 12645 ERROR oslo_service.service listener(*args)
    2019-11-30 12:44:40.162 12645 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/amqp/connection.py", line 639, in _on_close
    2019-11-30 12:44:40.162 12645 ERROR oslo_service.service (class_id, method_id), ConnectionError)
    2019-11-30 12:44:40.162 12645 ERROR oslo_service.service amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.

----------------------------------------------------------------------------
RabbitMQ logs:

    =INFO REPORT==== 30-Nov-2019::13:32:36 ===
    Connection <0.29181.156> (10.0.0.112:49104 -> 10.0.0.101:5672) has a client-provided name: neutron-linuxbridge-agent:1935:0a424cb4-182b-4fdc-be57-e3c7f438ce92

    =ERROR REPORT==== 30-Nov-2019::13:32:36 ===
    Error on AMQP connection <0.29181.156> (10.0.0.112:49104 -> 10.0.0.101:5672, state: starting):
    AMQPLAIN login refused: user 'openstack' - invalid credentials

    =INFO REPORT==== 30-Nov-2019::13:32:36 ===
    accepting AMQP connection <0.29198.156> (10.0.0.112:49108 -> 10.0.0.101:5672)

    =INFO REPORT==== 30-Nov-2019::13:32:36 ===
    accepting AMQP connection <0.29192.156> (10.0.0.112:49110 -> 10.0.0.101:5672)

    =INFO REPORT==== 30-Nov-2019::13:32:36 ===
    Connection <0.29192.156> (10.0.0.112:49110 -> 10.0.0.101:5672) has a client-provided name: neutron-linuxbridge-agent:1935:0a424cb4-182b-4fdc-be57-e3c7f438ce92

    =ERROR REPORT==== 30-Nov-2019::13:32:36 ===
    Error on AMQP connection <0.29192.156> (10.0.0.112:49110 -> 10.0.0.101:5672, state: starting):
    AMQPLAIN login refused: user 'openstack' - invalid credentials

    =INFO REPORT==== 30-Nov-2019::13:32:36 ===
    closing AMQP connection <0.29198.156> (10.0.0.112:49108 -> 10.0.0.101:5672 - nova-compute:2127:c92acc32-b9ac-4f6c-823f-33facc7c0c42)

    =INFO REPORT==== 30-Nov-2019::13:32:36 ===
    accepting AMQP connection <0.29201.156> (10.0.0.112:49112 -> 10.0.0.101:5672)

    =INFO REPORT==== 30-Nov-2019::13:32:36 ===
    accepting AMQP connection <0.29204.156> (10.0.0.112:49114 -> 10.0.0.101:5672)

    =INFO REPORT==== 30-Nov-2019::13:32:36 ===
    Connection <0.29201.156> (10.0.0.112:49112 -> 10.0.0.101:5672) has a client-provided name: neutron-linuxbridge-agent:1935:0a424cb4-182b-4fdc-be57-e3c7f438ce92

    =ERROR REPORT==== 30-Nov-2019::13:32:36 ===
    Error on AMQP connection <0.29201.156> (10.0.0.112:49112 -> 10.0.0.101:5672, state: starting):
    AMQPLAIN login refused: user 'openstack' - invalid credentials

------------------------------------------------------------------------------

What should I do to fix this ?!

> PS: I did not change RabbitMQ user credentials at all, and I double check for RabbitMQ credentials in any config files and Database.