Hi All,
I'm using Ocata version. I deployed several compute nodes, one network node and two active/backup controller nodes with running neutron-server and rabbitmq. after I switched the role of the controller nodes. the dhcp-agent disabled its report state. here is dhcp-agent log:
2018-01-23 21:07:59.542 2903 ERROR oslo.messaging._drivers.impl_rabbit [req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -] [f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout: timed out
2018-01-23 21:08:01.489 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:01.554 2903 ERROR oslo.messaging._drivers.impl_rabbit [req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -] [f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:02.481 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:05.503 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:05.568 2903 ERROR oslo.messaging._drivers.impl_rabbit [req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -] [f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:06.494 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:08:11.547 2903 INFO oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] Reconnected to AMQP server on 100.7.38.246:5672 via [amqp] client with port 47254.
2018-01-23 21:08:11.599 2903 INFO oslo.messaging._drivers.impl_rabbit [req-d3517c03-3118-43db-ae21-dc28e29ae79b - - - - -] [f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] Reconnected to AMQP server on 100.7.38.246:5672 via [amqp] client with port 47256.
2018-01-23 21:08:12.541 2903 INFO oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] Reconnected to AMQP server on 100.7.38.246:5672 via [amqp] client with port 47258.
2018-01-23 21:08:32.154 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:08:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:09:32.154 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:09:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:10:32.154 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:10:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:10:53.561 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. Client port: None: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2018-01-23 21:10:53.570 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. Client port: None: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2018-01-23 21:10:54.584 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:54.586 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:56.608 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:56.609 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:10:56.623 2903 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2018-01-23 21:10:57.621 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:10:59.634 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:11:00.621 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:11:08.537 2903 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe
2018-01-23 21:11:08.648 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:11:11.635 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: timed out. Trying again in 8 seconds. Client port: None: timeout: timed out
2018-01-23 21:11:14.659 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout: timed out
2018-01-23 21:11:16.635 2903 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: timeout: timed out
2018-01-23 21:11:21.673 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: timed out. Trying again in 4 seconds. Client port: None: timeout: timed out
2018-01-23 21:11:22.709 2903 INFO oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] Reconnected to AMQP server on 100.7.38.246:5672 via [amqp] client with port 47434.
2018-01-23 21:11:23.524 2903 ERROR oslo.messaging._drivers.impl_rabbit [req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -] [f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] AMQP server on 100.7.38.246:5672 is unreachable: basic_publish: connection closed. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: basic_publish: connection closed
2018-01-23 21:11:24.558 2903 INFO oslo.messaging._drivers.impl_rabbit [req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -] [f4c046e5-9ddf-4f67-b4f2-3064e47fc9d8] Reconnected to AMQP server on 100.7.38.246:5672 via [amqp] client with port 47458.
2018-01-23 21:11:25.714 2903 INFO oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] Reconnected to AMQP server on 100.7.38.246:5672 via [amqp] client with port 47460.
2018-01-23 21:11:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:11:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent [req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID e3054e3b13b244af895ee8b2335ddc85
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 733, in _report_state
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent ctx, self.agent_state, True)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 87, in report_state
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent return method(context, 'report_state', **kwargs)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent retry=self.retry)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent timeout=timeout, retry=retry)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent retry=retry)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in _send
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent result = self._waiter.wait(msg_id, timeout)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent message = self.waiters.get(msg_id, timeout=timeout)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 347, in get
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent 'to message ID %s' % msg_id)
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent MessagingTimeout: Timed out waiting for a reply to message ID e3054e3b13b244af895ee8b2335ddc85
2018-01-23 21:12:24.563 2903 ERROR neutron.agent.dhcp.agent
2018-01-23 21:12:24.565 2903 WARNING oslo.service.loopingcall [req-968a5271-5b02-43a9-93a8-8e1565c2e9b3 - - - - -] Function 'neutron.agent.dhcp.agent.DhcpAgentWithStateReport._report_state' run outlasted interval by 36.04 sec
2018-01-23 21:12:24.656 2903 INFO neutron.agent.dhcp.agent [req-6ae984dc-fcf6-45a7-8a00-800d9e6a0eb1 - - - - -] Agent has just been revived. Scheduling full sync
2018-01-23 21:12:29.132 2903 DEBUG neutron.agent.dhcp.agent [req-58690281-b59d-4a71-a190-d0a11dde6e0f - - - - -] resync (*): ['Agent has just been revived'] _periodic_resync_helper /usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py:261
2018-01-23 21:12:29.133 2903 INFO neutron.agent.dhcp.agent [req-58690281-b59d-4a71-a190-d0a11dde6e0f - - - - -] Synchronizing state
2018-01-23 21:12:29.518 2903 INFO neutron.agent.dhcp.agent [req-a4f75792-796e-474a-9bac-5d13fbb31197 - - - - -] All active networks have been fetched through RPC.
2018-01-23 21:12:29.519 2903 INFO neutron.agent.dhcp.agent [req-a4f75792-796e-474a-9bac-5d13fbb31197 - - - - -] Synchronizing state complete
2018-01-23 21:12:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:12:32.156 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:13:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:13:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:13:59.991 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. Client port: None: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2018-01-23 21:13:59.996 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. Client port: None: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2018-01-23 21:14:01.015 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:01.016 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:01.653 2903 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2018-01-23 21:14:03.029 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:03.041 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:04.041 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:06.055 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:07.055 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:08.586 2903 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 32] Broken pipe
2018-01-23 21:14:15.069 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:18.068 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: timed out. Trying again in 8 seconds. Client port: None: timeout: timed out
2018-01-23 21:14:21.083 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout: timed out
2018-01-23 21:14:21.666 2903 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: timeout: timed out
2018-01-23 21:14:28.096 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: timed out. Trying again in 4 seconds. Client port: None: timeout: timed out
2018-01-23 21:14:28.597 2903 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: timeout: timed out
2018-01-23 21:14:29.086 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:29.577 2903 ERROR oslo.messaging._drivers.impl_rabbit [req-1bc1ad73-fc9d-4911-b170-f94c2f62514c - - - - -] Failed to publish message to topic 'neutron': 'NoneType' object has no attribute 'itervalues'
2018-01-23 21:14:29.578 2903 WARNING neutron.agent.dhcp.agent [req-1bc1ad73-fc9d-4911-b170-f94c2f62514c - - - - -] Neutron server does not support state report. State report for this agent will be disabled.
2018-01-23 21:14:30.099 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:32.122 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error>
2018-01-23 21:14:32.123 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:14:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-23 21:14:33.134 2903 ERROR oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] AMQP server on 100.7.38.246:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-01-23 21:14:35.175 2903 INFO oslo.messaging._drivers.impl_rabbit [-] [5e001526-90fb-41b4-a5fd-eef0ac96ee00] Reconnected to AMQP server on 100.7.38.246:5672 via [amqp] client with port 47694.
2018-01-23 21:14:36.191 2903 INFO oslo.messaging._drivers.impl_rabbit [-] [04989bd8-915f-4b83-89e2-e267fe118ce0] Reconnected to AMQP server on 100.7.38.246:5672 via [amqp] client with port 47696.
2018-01-23 21:15:32.155 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-23 21:15:32.156 2903 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.001s inner /usr/lib/pytho
dhcp-agent log