According to logs Neutron server sends 1 message upon router deletion (Fanout notify agent at l3_agent the message router_deleted on router 5534c2e4-d1c1-4882-9ac7-816798f9fb5d). Neutron L3 agent receives the message twice: 2015-01-20 10:19:15.377 32009 DEBUG neutron.agent.l3_agent [req-88d745e8-1142-493f-b21b-54088db94d1c None] Got router deleted notification for 5534c2e4-d1c1-4882-9ac7-816798f9fb5d and 2015-01-20 10:19:15.378 32009 DEBUG neutron.agent.l3_agent [req-88d745e8-1142-493f-b21b-54088db94d1c None] Got router deleted notification for 5534c2e4-d1c1-4882-9ac7-816798f9fb5d (the difference is 0.001 sec). In the traffic capture one message is seen as send to MQ, but forwarded with the dup. ~~SERVER~~ 2015-01-20T10:19:13.495063+00:00 debug: 2015-01-20 10:19:13.494 458 DEBUG routes.middleware [req-88d745e8-1142-493f-b21b-54088 db94d1c ] Matched DELETE /routers/5534c2e4-d1c1-4882-9ac7-816798f9fb5d __call__ /usr/lib/python2.7/dist-packages/routes/middlew are.py:100 2015-01-20T10:19:13.495848+00:00 debug: 2015-01-20 10:19:13.495 458 DEBUG routes.middleware [req-88d745e8-1142-493f-b21b-54088 db94d1c ] Route path: '/routers/:(id)', defaults: {'action': u'delete', 'controller': >} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102 2015-01-20T10:19:13.498761+00:00 debug: 2015-01-20 10:19:13.498 458 DEBUG routes.middleware [req-88d745e8-1142-493f-b21b-54088 db94d1c ] Match dict: {'action': u'delete', 'controller': >, 'id' : u'5534c2e4-d1c1-4882-9ac7-816798f9fb5d'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103 2015-01-20T10:19:13.584286+00:00 debug: 2015-01-20 10:19:13.583 32708 DEBUG neutron.scheduler.l3_agent_scheduler [req-427c036d -c0cc-4a78-b845-e23dcc86fc1b None] Router 5534c2e4-d1c1-4882-9ac7-816798f9fb5d has already been hosted by L3 agent 00180b07-2ee 8-4aad-b603-637474278162 filter_unscheduled_routers /usr/lib/python2.7/dist-packages/neutron/scheduler/l3_agent_scheduler.py:73 2015-01-20T10:19:15.355620+00:00 debug: 2015-01-20 10:19:15.355 458 DEBUG neutron.api.rpc.agentnotifiers.l3_rpc_agent_api [req-88d745e8-1142-493f-b21b-54088db94d1c None] Fanout notify agent at l3_agent the message router_deleted on router 5534c2e4-d1c1-4882-9ac7-816798f9fb5d _notification_fanout /usr/lib/python2.7/dist-packages/neutron/api/rpc/agentnotifiers/l3_rpc_agent_api.py:127 2015-01-20T10:19:15.357249+00:00 debug: 2015-01-20 10:19:15.356 458 DEBUG neutron.common.rpc [req-88d745e8-1142-493f-b21b-54088db94d1c None] neutron.api.rpc.agentnotifiers.l3_rpc_agent_api.L3AgentNotifyAPI method fanout_cast called with arguments (, {'args': {'router_id': u'5534c2e4-d1c1-4882-9ac7-816798f9fb5d'}, 'namespace': None, 'method': 'router_deleted'}) {'topic': 'l3_agent'} wrapper /usr/lib/python2.7/dist-packages/neutron/common/log.py:33 2015-01-20T10:19:15.359490+00:00 debug: 2015-01-20 10:19:15.359 458 DEBUG oslo.messaging._drivers.amqp [req-88d745e8-1142-493f-b21b-54088db94d1c ] UNIQUE_ID is 08e90f7ab1954e7c9efb54414689e1a1. _add_unique_id /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqp.py:246 2015-01-20T10:19:15.367637+00:00 debug: 2015-01-20 10:19:15.367 458 DEBUG oslo.messaging._drivers.impl_rabbit [req-88d745e8-1142-493f-b21b-54088db94d1c ] Publisher.send: sending message None to {'oslo.message': '{"_context_roles": ["_member_", "SwiftOperator"], "_context_project_name": "TestNetworkAdvancedServerOps-731648348", "_context_read_deleted": "no", "_context_request_id": "req-88d745e8-1142-493f-b21b-54088db94d1c", "_context_user_name": "TestNetworkAdvancedServerOps-531236036", "_context_auth_token": "66b3fcb5ba6543c99692e77a3944a551", "args": {"router_id": "5534c2e4-d1c1-4882-9ac7-816798f9fb5d"}, "_context_tenant": "2c12c8194ebd491ab26fe933fc7be44e", "_unique_id": "08e90f7ab1954e7c9efb54414689e1a1", "_context_is_admin": false, "version": "1.0", "_context_timestamp": "2015-01-20 10:19:13.485931", "_context_tenant_name": "TestNetworkAdvancedServerOps-731648348", "_context_user": "e206392996c2444ab5db00d918de6a77", "_context_user_id": "e206392996c2444ab5db00d918de6a77", "_context_tenant_id": "2c12c8194ebd491ab26fe933fc7be44e", "method": "router_deleted", "_context_project_id": "2c12c8194ebd491ab26fe933fc7be44e"}', 'oslo.version': '2.0'} with routing key l3_agent_fanout send /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py:353 2015-01-20T10:19:15.393743+00:00 info: 2015-01-20 10:19:15.391 458 INFO neutron.wsgi [req-88d745e8-1142-493f-b21b-54088db94d1c None] 192.168.0.2 - - [20/Jan/2015 10:19:15] "DELETE /v2.0/routers/5534c2e4-d1c1-4882-9ac7-816798f9fb5d HTTP/1.1" 204 168 1.923437 ~~L3 AGENT~~ 2015-01-20T10:19:15.380166+00:00 debug: 2015-01-20 10:19:15.375 32009 DEBUG oslo.messaging._drivers.impl_rabbit [-] RabbitMess age.acknowledge: message {u'args': {u'router_id': u'5534c2e4-d1c1-4882-9ac7-816798f9fb5d'}, u'version': u'1.0', u'method': u'ro uter_deleted'} acknowledge /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py:138 2015-01-20T10:19:15.380166+00:00 debug: 2015-01-20 10:19:15.376 32009 DEBUG neutron.context [req-88d745e8-1142-493f-b21b-54088 db94d1c None] Arguments dropped when creating context: {u'project_id': u'2c12c8194ebd491ab26fe933fc7be44e', u'project_name': u' TestNetworkAdvancedServerOps-731648348', u'user': u'e206392996c2444ab5db00d918de6a77', u'tenant': u'2c12c8194ebd491ab26fe933fc7 be44e'} __init__ /usr/lib/python2.7/dist-packages/neutron/context.py:83 2015-01-20T10:19:15.380166+00:00 debug: 2015-01-20 10:19:15.377 32009 DEBUG neutron.agent.l3_agent [req-88d745e8-1142-493f-b21 b-54088db94d1c None] Got router deleted notification for 5534c2e4-d1c1-4882-9ac7-816798f9fb5d router_deleted /usr/lib/python2.7 /dist-packages/neutron/agent/l3_agent.py:1699 2015-01-20T10:19:15.380166+00:00 debug: 2015-01-20 10:19:15.377 32009 DEBUG neutron.context [req-88d745e8-1142-493f-b21b-54088 db94d1c None] Arguments dropped when creating context: {u'project_id': u'2c12c8194ebd491ab26fe933fc7be44e', u'project_name': u' TestNetworkAdvancedServerOps-731648348', u'user': u'e206392996c2444ab5db00d918de6a77', u'tenant': u'2c12c8194ebd491ab26fe933fc7 be44e'} __init__ /usr/lib/python2.7/dist-packages/neutron/context.py:83 2015-01-20T10:19:15.380490+00:00 debug: 2015-01-20 10:19:15.378 32009 DEBUG neutron.agent.l3_agent [req-88d745e8-1142-493f-b21 b-54088db94d1c None] Got router deleted notification for 5534c2e4-d1c1-4882-9ac7-816798f9fb5d router_deleted /usr/lib/python2.7 /dist-packages/neutron/agent/l3_agent.py:1699 2015-01-20T10:19:15.461592+00:00 debug: 2015-01-20 10:19:15.460 32009 DEBUG oslo.messaging._drivers.impl_rabbit [-] RabbitMess age.Init: message {u'_unique_id': u'66857b1105774105bc2fee15d90fd3d5', u'failure': None, u'result': [{u'status': u'ACTIVE', u'e xternal_gateway_info': None, u'ha_vr_id': 0, u'name': u'router-smoke-1657318176', u'gw_port_id': None, u'admin_state_up': True, u'tenant_id': u'2c12c8194ebd491ab26fe933fc7be44e', u'distributed': False, u'routes': [], u'ha': False, u'id': u'5534c2e4-d1c1- 4882-9ac7-816798f9fb5d'}], u'_msg_id': u'ff47c246cc674e99a544f734fe47acc0'} __init__ /usr/lib/python2.7/dist-packages/oslo/mess aging/_drivers/impl_rabbit.py:134 2015-01-20T10:19:15.461765+00:00 debug: 2015-01-20 10:19:15.460 32009 DEBUG oslo.messaging._drivers.impl_rabbit [-] RabbitMess age.acknowledge: message {u'_unique_id': u'66857b1105774105bc2fee15d90fd3d5', u'failure': None, u'result': [{u'status': u'ACTIV E', u'external_gateway_info': None, u'ha_vr_id': 0, u'name': u'router-smoke-1657318176', u'gw_port_id': None, u'admin_state_up' : True, u'tenant_id': u'2c12c8194ebd491ab26fe933fc7be44e', u'distributed': False, u'routes': [], u'ha': False, u'id': u'5534c2e 4-d1c1-4882-9ac7-816798f9fb5d'}], u'_msg_id': u'ff47c246cc674e99a544f734fe47acc0'} acknowledge /usr/lib/python2.7/dist-packages /oslo/messaging/_drivers/impl_rabbit.py:138 2015-01-20T10:19:15.899252+00:00 debug: 2015-01-20 10:19:15.640 32009 DEBUG neutron.agent.l3_agent [-] Starting router update for 5534c2e4-d1c1-4882-9ac7-816798f9fb5d _process_router_update /usr/lib/python2.7/dist-packages/neutron/agent/l3_agent.py:1814 2015-01-20T10:19:15.899252+00:00 debug: 2015-01-20 10:19:15.641 32009 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-5534c2e4-d1c1-4882-9ac7-816798f9fb5d', 'ip', '-o', '-d', 'link', 'list'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:46 2015-01-20T10:19:15.899252+00:00 debug: 2015-01-20 10:19:15.760 32009 DEBUG neutron.agent.linux.utils [-] Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-5534c2e4-d1c1-4882-9ac7-816798f9fb5d', 'ip', '-o', '-d', 'link', 'list'] Exit code: 0 Stdout: '1: lo: mtu 65536 qdisc noqueue state UNKNOWN \\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\n' Stderr: '' execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:81 2015-01-20T10:19:15.899252+00:00 debug: 2015-01-20 10:19:15.763 32009 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'kill', '-9', '3118'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:46 2015-01-20T10:19:15.899252+00:00 debug: 2015-01-20 10:19:15.828 32009 DEBUG neutron.agent.linux.utils [-] Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'kill', '-9', '3118'] Exit code: 0 Stdout: '' Stderr: '' execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:81 2015-01-20T10:19:15.899552+00:00 debug: 2015-01-20 10:19:15.829 32009 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-5534c2e4-d1c1-4882-9ac7-816798f9fb5d', 'ip', '-o', '-d', 'link', 'list'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:46 2015-01-20T10:19:15.899552+00:00 debug: 2015-01-20 10:19:15.893 32009 DEBUG neutron.agent.linux.utils [-] Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-5534c2e4-d1c1-4882-9ac7-816798f9fb5d', 'ip', '-o', '-d', 'link', 'list'] Exit code: 0 Stdout: '1: lo: mtu 65536 qdisc noqueue state UNKNOWN \\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\n' Stderr: '' execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:81 2015-01-20T10:19:15.899552+00:00 debug: 2015-01-20 10:19:15.894 32009 DEBUG neutron.agent.l3_agent [-] Starting router update for 5534c2e4-d1c1-4882-9ac7-816798f9fb5d _process_router_update /usr/lib/python2.7/dist-packages/neutron/agent/l3_agent.py:1814 2015-01-20T10:19:15.899552+00:00 warning: 2015-01-20 10:19:15.895 32009 WARNING neutron.agent.l3_agent [-] Info for router 5534c2e4-d1c1-4882-9ac7-816798f9fb5d were not found. Skipping router removal ~~ TRAFFIC ~~ amqp.payload contains 30:38:65:39:30:66:37:61:62 6967 13:19:15.368768 127.0.0.1 127.0.0.1 AMQP 1114 Content-Body Transmission Control Protocol, Src Port: 54310 (54310), Dst Port: jms (5673), Seq: 22589, Ack: 1223, Len: 1048 {"oslo.message": "{\"_context_roles\": [\"_member_\", \"SwiftOperator\"], \"_context_project_name\": \"TestNetworkAdvancedServerOps-731648348\", \"_context_read_deleted\": \"no\", \"_context_request_id\": \"req-88d745e8-1142-493f-b21b-54088db94d1c\", \"_context_user_name\": \"TestNetworkAdvancedServerOps-531236036\", \"_context_auth_token\": \"66b3fcb5ba6543c99692e77a3944a551\", \"args\": {\"router_id\": \"5534c2e4-d1c1-4882-9ac7-816798f9fb5d\"}, \"_context_tenant\": \"2c12c8194ebd491ab26fe933fc7be44e\", \"_unique_id\": \"08e90f7ab1954e7c9efb54414689e1a1\", \"_context_is_admin\": false, \"version\": \"1.0\", \"_context_timestamp\": \"2015-01-20 10:19:13.485931\", \"_context_tenant_name\": \"TestNetworkAdvancedServerOps-731648348\", \"_context_user\": \"e206392996c2444ab5db00d918de6a77\", \"_context_user_id\": \"e206392996c2444ab5db00d918de6a77\", \"_context_tenant_id\": \"2c12c8194ebd491ab26fe933fc7be44e\", \"method\": \"router_deleted\", \"_context_project_id\": \"2c12c8194ebd491ab26fe933fc7be44e\"}", "oslo.version": "2.0"} 6968 13:19:15.369634 127.0.0.1 127.0.0.1 AMQP 1205 Basic.DeliverContent-Header Content-Body Transmission Control Protocol, Src Port: jms (5673), Dst Port: 49540 (49540), Seq: 2311, Ack: 60, Len: 1139 {"oslo.message": "{\"_context_roles\": [\"_member_\", \"SwiftOperator\"], \"_context_project_name\": \"TestNetworkAdvancedServerOps-731648348\", \"_context_read_deleted\": \"no\", \"_context_request_id\": \"req-88d745e8-1142-493f-b21b-54088db94d1c\", \"_context_user_name\": \"TestNetworkAdvancedServerOps-531236036\", \"_context_auth_token\": \"66b3fcb5ba6543c99692e77a3944a551\", \"args\": {\"router_id\": \"5534c2e4-d1c1-4882-9ac7-816798f9fb5d\"}, \"_context_tenant\": \"2c12c8194ebd491ab26fe933fc7be44e\", \"_unique_id\": \"08e90f7ab1954e7c9efb54414689e1a1\", \"_context_is_admin\": false, \"version\": \"1.0\", \"_context_timestamp\": \"2015-01-20 10:19:13.485931\", \"_context_tenant_name\": \"TestNetworkAdvancedServerOps-731648348\", \"_context_user\": \"e206392996c2444ab5db00d918de6a77\", \"_context_user_id\": \"e206392996c2444ab5db00d918de6a77\", \"_context_tenant_id\": \"2c12c8194ebd491ab26fe933fc7be44e\", \"method\": \"router_deleted\", \"_context_project_id\": \"2c12c8194ebd491ab26fe933fc7be44e\"}", "oslo.version": "2.0"} 6969 13:19:15.369691 127.0.0.1 127.0.0.1 AMQP 1205 Basic.DeliverContent-Header Content-Body Transmission Control Protocol, Src Port: jms (5673), Dst Port: 49538 (49538), Seq: 2311, Ack: 52, Len: 1139 {"oslo.message": "{\"_context_roles\": [\"_member_\", \"SwiftOperator\"], \"_context_project_name\": \"TestNetworkAdvancedServerOps-731648348\", \"_context_read_deleted\": \"no\", \"_context_request_id\": \"req-88d745e8-1142-493f-b21b-54088db94d1c\", \"_context_user_name\": \"TestNetworkAdvancedServerOps-531236036\", \"_context_auth_token\": \"66b3fcb5ba6543c99692e77a3944a551\", \"args\": {\"router_id\": \"5534c2e4-d1c1-4882-9ac7-816798f9fb5d\"}, \"_context_tenant\": \"2c12c8194ebd491ab26fe933fc7be44e\", \"_unique_id\": \"08e90f7ab1954e7c9efb54414689e1a1\", \"_context_is_admin\": false, \"version\": \"1.0\", \"_context_timestamp\": \"2015-01-20 10:19:13.485931\", \"_context_tenant_name\": \"TestNetworkAdvancedServerOps-731648348\", \"_context_user\": \"e206392996c2444ab5db00d918de6a77\", \"_context_user_id\": \"e206392996c2444ab5db00d918de6a77\", \"_context_tenant_id\": \"2c12c8194ebd491ab26fe933fc7be44e\", \"method\": \"router_deleted\", \"_context_project_id\": \"2c12c8194ebd491ab26fe933fc7be44e\"}", "oslo.version": "2.0"} ~~ NODE-5 ~~ root@node-5:~# ps aux | grep l3 neutron 32009 1.5 0.2 112064 43312 ? S 10:00 0:44 /usr/bin/python /usr/bin/neutron-l3-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/l3_agent.ini --log-file=/var/log/neutron/l3-agent.log root 32365 0.0 0.0 9388 940 pts/17 S+ 10:47 0:00 grep --color=auto l3 root@node-5:~# lsof -i 4 | grep 32009 neutron-l 32009 neutron 7u IPv4 10517866 0t0 TCP localhost:49534->localhost:5673 (ESTABLISHED) neutron-l 32009 neutron 8u IPv4 10517882 0t0 TCP localhost:49535->localhost:5673 (ESTABLISHED) neutron-l 32009 neutron 9u IPv4 10518076 0t0 TCP localhost:49538->localhost:5673 (ESTABLISHED) neutron-l 32009 neutron 10u IPv4 10518080 0t0 TCP localhost:49539->localhost:5673 (ESTABLISHED) neutron-l 32009 neutron 11u IPv4 10518084 0t0 TCP localhost:49540->localhost:5673 (ESTABLISHED)