neutron

NeutronServerTrytoFindL3agentOnComputeNodewhenWeUseLinuxBridge

Bug #1737917 reported by Aaasidncza on 2017-12-13

This bug affects 13 people

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Medium	Slawek Kaplonski

Bug Description

Hi all Cool Developer of Openstack,I followed the pike install doc and successful running openstack now,and In
Neutron configuration section,Verify operation Networking Option 2: Self-service networks(I choosed Self-Servrice network to deploy on Centos7x64 everything up to date)there we see:
The output should indicate four agents on the controller node and one agent on each compute node.

$ openstack network agent list

seems running normally just like what we got in offical docs.and ip netns on controller:
[root@controller neutron]# ip netns exec qrouter-89466cea-7d58-4a45-92da-e636c0
958358 ping -c 5 www.bing.com
PING cn-0001.cn-msedge.net (202.89.233.100) 56(84) bytes of data.
64 bytes from 202.89.233.100 (202.89.233.100): icmp_seq=1 ttl=115 time=31.6 ms
64 bytes from 202.89.233.100 (202.89.233.100): icmp_seq=2 ttl=115 time=31.5 ms
64 bytes from 202.89.233.100 (202.89.233.100): icmp_seq=3 ttl=115 time=31.4 ms
64 bytes from 202.89.233.100 (202.89.233.100): icmp_seq=4 ttl=115 time=32.4 ms
64 bytes from 202.89.233.100 (202.89.233.100): icmp_seq=5 ttl=115 time=31.8 ms

--- cn-0001.cn-msedge.net ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4003ms
rtt min/avg/max/mdev = 31.490/31.810/32.405/0.397 ms
[root@controller neutron]# ip netns
qrouter-89466cea-7d58-4a45-92da-e636c0958358 (id: 2)
qdhcp-fca97929-aed3-45dc-9f19-f7ead767fbc3 (id: 0)
qdhcp-c08c44ed-d71e-4acb-b1a5-3ebc4715a01d (id: 1)

and every virtual machine on openstack runs perfectly(self network or provider network,access to Internet),except one
day I log into compute node,and found the linuxbridge.log seems too big(over 300mb),quickly I use grep to filter out most INFO information and got so many ERROR log like this(also the only one type of error happens every fix period):

2017-12-13 17:23:16.030 1334 INFO neutron.agent.securitygroups_rpc [req-7fbf230a
-bf45-4817-a7e8-447005e1700a - - - - -] Security group member updated [u'626f761
f-7451-42cf-afbf-724da51190c0']
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt [req-7fbf230a-bf45-4817-a7e8-447005e1700a - - - - -] Error occurred while rem
oving port tapa97e89f4-2d: RemoteError: Remote error: AgentNotFoundByTypeHost Ag
ent with agent_type=L3 agent and host=compute1 could not be found
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-pack
ages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n res = se
lf.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/o
slo_messaging/rpc/dispatcher.py", line 213, in dispatch\n return self._do_dis
patch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packag
es/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n result = fu
nc(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/neutron/plug
ins/ml2/rpc.py", line 234, in update_device_down\n n_const.PORT_STATUS_DOWN,
host)\n', u' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py"
, line 331, in notify_l2pop_port_wiring\n l2pop_driver.obj.update_port_down(p
ort_context)\n', u' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/
drivers/l2pop/mech_driver.py", line 253, in update_port_down\n admin_context,
agent_host, [port[\'device_id\']]):\n', u' File "/usr/lib/python2.7/site-packa
ges/neutron/db/l3_agentschedulers_db.py", line 303, in list_router_ids_on_host\n
context, constants.AGENT_TYPE_L3, host)\n', u' File "/usr/lib/python2.7/sit
e-packages/neutron/db/agents_db.py", line 291, in _get_agent_by_type_and_host\n
host=host)\n', u'AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and
host=compute1 could not be found\n'].
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt Traceback (most recent call last):
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_c
ommon_agent.py", line 336, in treat_devices_removed
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt cfg.CONF.host)
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 139, in
update_device_down
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt agent_id=agent_id, host=host)
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 162, in
call
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt return self._original_context.call(ctxt, method, **kwargs)
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line
169, in call
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt retry=self.retry)
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 1
23, in _send
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt timeout=timeout, retry=retry)
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.p
y", line 578, in send
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt retry=retry)
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.p
y", line 569, in _send
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt raise result
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt RemoteError: Remote error: AgentNotFoundByTypeHost Agent with agent_type=L3 a
gent and host=compute1 could not be found
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-p
ackages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n res =
self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-package
s/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch\n return self._do_
dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-pac
kages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n result =
func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/neutron/p
lugins/ml2/rpc.py", line 234, in update_device_down\n n_const.PORT_STATUS_DOW
N, host)\n', u' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.
py", line 331, in notify_l2pop_port_wiring\n l2pop_driver.obj.update_port_dow
n(port_context)\n', u' File "/usr/lib/python2.7/site-packages/neutron/plugins/m
l2/drivers/l2pop/mech_driver.py", line 253, in update_port_down\n admin_conte
xt, agent_host, [port[\'device_id\']]):\n', u' File "/usr/lib/python2.7/site-pa
ckages/neutron/db/l3_agentschedulers_db.py", line 303, in list_router_ids_on_hos
t\n context, constants.AGENT_TYPE_L3, host)\n', u' File "/usr/lib/python2.7/
site-packages/neutron/db/agents_db.py", line 291, in _get_agent_by_type_and_host
\n host=host)\n', u'AgentNotFoundByTypeHost: Agent with agent_type=L3 agent a
nd host=compute1 could not be found\n'].
2017-12-13 17:23:16.172 1334 ERROR neutron.plugins.ml2.drivers.agent._common_age
nt
2017-12-13 17:23:16.174 1334 INFO neutron.plugins.ml2.drivers.agent._common_agen
t [req-7fbf230a-bf45-4817-a7e8-447005e1700a - - - - -] Attachment tap8d780fbe-84
removed
2017-12-13 17:23:16.218 1334 INFO neutron.plugins.ml2.drivers.agent._common_agen
t [req-7fbf230a-bf45-4817-a7e8-447005e1700a - - - - -] Port tap8d780fbe-84 updat
ed.
2017-12-13 17:23:16.218 1334 INFO neutron.plugins.ml2.drivers.agent._common_agen
t [req-7fbf230a-bf45-4817-a7e8-447005e1700a - - - - -] Attachment tap3ca9698e-61
removed

Here we see

"AgentNotFoundByTypeHost: Agent with agent_type=L3 agent a
nd host=compute1 could not be found\n"

So I come back to the compute node neutron configration section to check if there was setup l3 agent on compute node I was missing,But checked and found there never mentions that we need to install l3 agent on compute node.

I googled these keywords and found only a little posts about this problem,and most posts was about dhcp agent cannot found or old release of openstack,so Is there a bug on neutron what should I do to avoid this,should I install l3 agent on compute node or just ignore this error info since all VMs runns as usual?

-----------------------------------
Release: 11.0.3.dev21 on 2017-12-11 21:40
SHA: 1caba2d2e0f5c9752a858f28957d7dbb33640180
Source: https://git.openstack.org/cgit/openstack/neutron/tree/doc/source/install/verify-option2.rst
URL: https://docs.openstack.org/neutron/pike/install/verify-option2.html

Tags:

Revision history for this message

Aaasidncza (aascigsz2) wrote on 2017-12-13:

linuxbridgelogOnComputeNode Edit (2.2 MiB, application/zip)

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2017-12-13:

For me it looks more like some error in l2 population mech driver because error message is IMHO returned to Linuxbridge agent from server as a response to rpc calls.
Can You check also what You have in neutron-server logs for same time?
And do You have l2pop enabled on Your cluster?

tags:	added: linuxbridge removed: doc
tags:	added: l2-pop

Revision history for this message

Aaasidncza (aascigsz2) wrote on 2017-12-13:

Download full text (21.5 KiB)

Hi Slawek Kaplonski (slaweq),thanks for help.I'm not very sure about it's my fault or a bug of neutron,here is
server.log on controller node:
[HEAD]
[root@controller neutron]# head -n 50 server.log
2017-12-13 03:30:04.153 1862 INFO neutron.notifiers.nova [-] Nova event response
: {u'status': u'completed', u'tag': u'1b6e1b2c-aea4-41b1-9a98-9bee24ee8a0e', u'n
ame': u'network-vif-plugged', u'server_uuid': u'08044f19-3184-4f01-8c6f-5a2f5f03
f215', u'code': 200}
2017-12-13 03:30:04.153 1862 INFO neutron.notifiers.nova [-] Nova event response
: {u'status': u'completed', u'tag': u'f28b362c-0f15-41be-b08c-ce8f161fc8d4', u'n
ame': u'network-vif-plugged', u'server_uuid': u'97eff4de-19a2-4558-8562-0a1471bb
73c8', u'code': 200}
2017-12-13 03:30:04.153 1862 INFO neutron.notifiers.nova [-] Nova event response
: {u'status': u'completed', u'tag': u'd1b3dab4-fdf6-4cac-ac78-c1b33686655a', u'n
ame': u'network-vif-plugged', u'server_uuid': u'77722c79-e584-4200-b4c4-230bdecd
9a1a', u'code': 200}
2017-12-13 03:30:09.689 1862 INFO neutron.notifiers.nova [-] Nova event response
: {u'status': u'completed', u'tag': u'31f89863-e597-4f96-bb06-c092d9126775', u'n
ame': u'network-vif-plugged', u'server_uuid': u'ea492dec-12eb-4282-8196-97182cd6
1e08', u'code': 200}
2017-12-13 03:30:09.689 1862 INFO neutron.notifiers.nova [-] Nova event response
: {u'status': u'completed', u'tag': u'430af11e-a111-418d-b46e-1bdec38d62a7', u'n
ame': u'network-vif-plugged', u'server_uuid': u'87480b8a-ce25-45ef-8bed-785ac321
172d', u'code': 200}
2017-12-13 03:30:09.689 1862 INFO neutron.notifiers.nova [-] Nova event response
: {u'status': u'completed', u'tag': u'fdb0d573-1ddc-4228-86c7-d1b2e0ee9f14', u'n
ame': u'network-vif-plugged', u'server_uuid': u'b406c12e-95c9-4fa9-ae0b-14e30089
b02f', u'code': 200}
2017-12-13 03:30:10.241 1858 INFO neutron.wsgi [req-b5734451-603f-450b-a33b-4c54
efb7448d fc93769465504021a04b29bee9b93e8b c957b73284424c1bbe7aee5a8eefa075 - def
ault default] 10.88.1.10 "GET /v2.0/ports?tenant_id=2b17a96ea1054d8e8f67f808e0fd
7fe5&device_id=ea492dec-12eb-4282-8196-97182cd61e08 HTTP/1.1" status: 200  len:
1074 time: 0.0387630
2017-12-13 03:30:10.321 1858 INFO neutron.wsgi [req-762bc966-a6e6-4e01-8b76-b79a
86b930e8 fc93769465504021a04b29bee9b93e8b c957b73284424c1bbe7aee5a8eefa075 - def
ault default] 10.88.1.10 "GET /v2.0/networks?id=c08c44ed-d71e-4acb-b1a5-3ebc4715
a01d HTTP/1.1" status: 200  len: 891 time: 0.0778539
2017-12-13 03:30:10.339 1858 INFO neutron.wsgi [req-34ea4f7c-9f91-4648-ad82-c57e
26243d20 fc93769465504021a04b29bee9b93e8b c957b73284424c1bbe7aee5a8eefa075 - def
ault default] 10.88.1.10 "GET /v2.0/floatingips?fixed_ip_address=10.99.1.49&port
_id=31f89863-e597-4f96-bb06-c092d9126775 HTTP/1.1" status: 200  len: 217 time: 0
.0128708
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server [req-7fbf230a-bf45-
4817-a7e8-447005e1700a - - - - -] Exception during message handling: AgentNotFou
ndByTypeHost: Agent with agent_type=L3 agent and host=compute1 could not be foun
d
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server Traceback (most rec
ent call last):
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incom
ing
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server     res = self.disp
atcher.dispatch(message)
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server     return self._do
_dispatch(endpoint, method, ctxt, args)
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispat
ch
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server     result = func(c
txt, **new_args)
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/plugins/ml2/rpc.py", line 234, in update_device_do
wn
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server     n_const.PORT_ST
ATUS_DOWN, host)
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/plugins/ml2/rpc.py", line 331, in notify_l2pop_por
t_wiring
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server     l2pop_driver.ob
j.update_port_down(port_context)
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 25
3, in update_port_down
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server     admin_context,
agent_host, [port['device_id']]):
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 303, in list_ro
uter_ids_on_host
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server     context, consta
nts.AGENT_TYPE_L3, host)
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/db/agents_db.py", line 291, in _get_agent_by_type_
and_host
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server     host=host)
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server AgentNotFoundByType
Host: Agent with agent_type=L3 agent and host=compute1 could not be found
2017-12-13 03:30:10.411 1862 ERROR oslo_messaging.rpc.server
2017-12-13 03:30:10.503 1858 INFO neutron.wsgi [req-d7e77dd7-a876-45c7-b061-938b
1d3e635b fc93769465504021a04b29bee9b93e8b c957b73284424c1bbe7aee5a8eefa075 - def
ault default] 10.88.1.10 "GET /v2.0/subnets?id=2d13271b-b6bf-466e-8942-5f4f5f855
a0a HTTP/1.1" status: 200  len: 821 time: 0.1606300
2017-12-13 03:30:10.546 1858 INFO neutron.wsgi [req-b52abca0-b88f-490d-a31e-ec2a
f7f8efe9 fc93769465504021a04b29bee9b93e8b c957b73284424c1bbe7aee5a8eefa075 - def
ault default] 10.88.1.10 "GET /v2.0/ports?network_id=c08c44ed-d71e-4acb-b1a5-3eb
c4715a01d&device_owner=network%3Adhcp HTTP/1.1" status: 200  len: 1072 time: 0.0
387509
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server [req-7fbf230a-bf45-
4817-a7e8-447005e1700a - - - - -] Exception during message handling: AgentNotFou
ndByTypeHost: Agent with agent_type=L3 agent and host=compute1 could not be foun
d
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server Traceback (most rec
ent call last):
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incom
ing
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server     res = self.disp
atcher.dispatch(message)
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server     return self._do
_dispatch(endpoint, method, ctxt, args)
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispat
ch
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server     result = func(c
txt, **new_args)
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/plugins/ml2/rpc.py", line 234, in update_device_do
wn
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server     n_const.PORT_ST
ATUS_DOWN, host)
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/plugins/ml2/rpc.py", line 331, in notify_l2pop_por
t_wiring
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server     l2pop_driver.ob
j.update_port_down(port_context)
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 25
3, in update_port_down
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server     admin_context,
agent_host, [port['device_id']]):
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 303, in list_ro
uter_ids_on_host
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server     context, consta
nts.AGENT_TYPE_L3, host)
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/db/agents_db.py", line 291, in _get_agent_by_type_
and_host
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server     host=host)
2017-12-13 03:30:10.610 1862 ERROR oslo_messaging.rpc.server AgentNotFoundByType
Host: Agent with agent_type=L3 agent and host=compute1 could not be found

[TAIL]
[root@controller neutron]# tail -n 50 server.log
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server Traceback (most rec
ent call last):
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incom
ing
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server     res = self.disp
atcher.dispatch(message)
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server     return self._do
_dispatch(endpoint, method, ctxt, args)
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispat
ch
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server     result = func(c
txt, **new_args)
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/plugins/ml2/rpc.py", line 234, in update_device_do
wn
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server     n_const.PORT_ST
ATUS_DOWN, host)
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/plugins/ml2/rpc.py", line 331, in notify_l2pop_por
t_wiring
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server     l2pop_driver.ob
j.update_port_down(port_context)
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 25
3, in update_port_down
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server     admin_context,
agent_host, [port['device_id']]):
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 303, in list_ro
uter_ids_on_host
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server     context, consta
nts.AGENT_TYPE_L3, host)
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/db/agents_db.py", line 291, in _get_agent_by_type_
and_host
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server     host=host)
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server AgentNotFoundByType
Host: Agent with agent_type=L3 agent and host=compute1 could not be found
2017-12-13 19:03:34.709 1862 ERROR oslo_messaging.rpc.server
2017-12-13 19:03:35.469 1862 INFO neutron.plugins.ml2.plugin [req-7fbf230a-bf45-
4817-a7e8-447005e1700a - - - - -] No ports have port_id starting with 9b256b8b-2
8
2017-12-13 19:03:35.608 1862 INFO neutron.notifiers.nova [-] Nova event response
: {u'status': u'completed', u'tag': u'a97e89f4-2dc2-468c-9aff-08cbb098ca2f', u'n
ame': u'network-vif-unplugged', u'server_uuid': u'd7494a79-7572-4922-817b-9a0712
609bb0', u'code': 200}
2017-12-13 19:03:35.968 1862 INFO neutron.plugins.ml2.plugin [req-7fbf230a-bf45-
4817-a7e8-447005e1700a - - - - -] No ports have port_id starting with 85c9082f-9
f
2017-12-13 19:03:36.231 1862 INFO neutron.plugins.ml2.plugin [req-7fbf230a-bf45-
4817-a7e8-447005e1700a - - - - -] No ports have port_id starting with 89fd7204-7
d
2017-12-13 19:03:36.760 1862 INFO neutron.plugins.ml2.plugin [req-7fbf230a-bf45-
4817-a7e8-447005e1700a - - - - -] No ports have port_id starting with 3ca9698e-6
1
2017-12-13 19:03:36.774 1862 INFO neutron.plugins.ml2.plugin [req-7fbf230a-bf45-
4817-a7e8-447005e1700a - - - - -] No ports have port_id starting with 10aeccf0-a
c
2017-12-13 19:03:37.035 1862 INFO neutron.plugins.ml2.plugin [req-7fbf230a-bf45-
4817-a7e8-447005e1700a - - - - -] No ports have port_id starting with 33f80bf6-9
e
2017-12-13 19:03:37.045 1862 INFO neutron.plugins.ml2.plugin [req-7fbf230a-bf45-
4817-a7e8-447005e1700a - - - - -] No ports have port_id starting with 7d767afe-0
d
2017-12-13 19:03:37.055 1862 INFO neutron.plugins.ml2.plugin [req-7fbf230a-bf45-
4817-a7e8-447005e1700a - - - - -] No ports have port_id starting with 7e69c0c1-1
9
2017-12-13 19:03:37.063 1862 INFO neutron.plugins.ml2.plugin [req-7fbf230a-bf45-
4817-a7e8-447005e1700a - - - - -] No ports have port_id starting with abed15b0-e
f
2017-12-13 19:03:37.638 1862 INFO neutron.notifiers.nova [-] Nova event response
: {u'status': u'completed', u'tag': u'b487ed3f-792a-432b-80ed-bdd2b139a0e8', u'n
ame': u'network-vif-plugged', u'server_uuid': u'82b8302c-08ed-42aa-9ac3-e8bcf714
cbc6', u'code': 200}
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server [req-7fbf230a-bf45-
4817-a7e8-447005e1700a - - - - -] Exception during message handling: AgentNotFou
ndByTypeHost: Agent with agent_type=L3 agent and host=compute1 could not be foun
d
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server Traceback (most rec
ent call last):
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incom
ing
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server     res = self.disp
atcher.dispatch(message)
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server     return self._do
_dispatch(endpoint, method, ctxt, args)
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispat
ch
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server     result = func(c
txt, **new_args)
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/plugins/ml2/rpc.py", line 234, in update_device_do
wn
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server     n_const.PORT_ST
ATUS_DOWN, host)
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/plugins/ml2/rpc.py", line 331, in notify_l2pop_por
t_wiring
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server     l2pop_driver.ob
j.update_port_down(port_context)
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 25
3, in update_port_down
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server     admin_context,
agent_host, [port['device_id']]):
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 303, in list_ro
uter_ids_on_host
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server     context, consta
nts.AGENT_TYPE_L3, host)
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server   File "/usr/lib/py
thon2.7/site-packages/neutron/db/agents_db.py", line 291, in _get_agent_by_type_
and_host
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server     host=host)
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server AgentNotFoundByType
Host: Agent with agent_type=L3 agent and host=compute1 could not be found
2017-12-13 19:03:38.047 1862 ERROR oslo_messaging.rpc.server

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

And here are My basic network info:
--------------------------------------------------------------------------------------------------------------
controller:
eth0   10.88.1.2  Gateway 10.88.1.1 Management network
eth1   10.99.1.2  Gateway 10.99.1.1 Provider
compute1:
eth3   10.88.1.10  Gateway 10.88.1.1 Management network
eth2   10.99.1.10  Gateway 10.99.1.1 Provider
And Here are all the configuration file on Controller Node and Compute Node:
##########################################################################################################
<Controller Node>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
--------------------------------------------------------------------------------------------------------------
/etc/neutron/neutron.conf:

[root@controller neutron]# cat neutron.conf | egrep -v "^$|^#"
[DEFAULT]
core_plugin = ml2
service_plugins = router
allow_overlapping_ips = true
transport_url = rabbit://openstack:openstack@controller
auth_strategy = keystone
notify_nova_on_port_status_changes = true
notify_nova_on_port_data_changes = true
[agent]
[cors]
[database]
connection = mysql+pymysql://neutron:neutron@controller/neutron
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = neutron
password = neutron
[matchmaker_redis]
[nova]
auth_url = http://controller:35357
auth_type = password
project_domain_name = default
user_domain_name = default
region_name = RegionOne
project_name = service
username = nova
password = nova
[oslo_concurrency]
lock_path = /var/lib/neutron/tmp
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
[oslo_messaging_zmq]
[oslo_middleware]
[oslo_policy]
[quotas]
[ssl]
[root@controller neutron]#

--------------------------------------------------------------------------------------------------------------
/etc/neutron/l3_agent.ini:

[root@controller neutron]# cat l3_agent.ini | egrep -v "^$|^#"
[DEFAULT]
interface_driver = linuxbridge
[agent]
[ovs]
[root@controller neutron]#
--------------------------------------------------------------------------------------------------------------
/etc/neutron/metadata_agent.ini:

[root@controller neutron]# cat metadata_agent.ini | egrep -v "^$|^#"
[DEFAULT]
nova_metadata_host = controller
metadata_proxy_shared_secret = metadata
[agent]
[cache]
[root@controller neutron]#
--------------------------------------------------------------------------------------------------------------
/etc/neutron/dhcp_agent.ini:

[root@controller neutron]# cat dhcp_agent.ini | egrep -v "^$|^#"
[DEFAULT]
interface_driver = linuxbridge
dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq
enable_isolated_metadata = true
[agent]
[ovs]
[root@controller neutron]#
--------------------------------------------------------------------------------------------------------------
/etc/neutron/plugins/ml2/linuxbridge_agent.ini:

[root@controller ml2]# cat linuxbridge_agent.ini |egrep -v "^$|^#"
[DEFAULT]
[agent]
[linux_bridge]
physical_interface_mappings = provider:eth1
[securitygroup]
enable_security_group = true
firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver
[vxlan]
enable_vxlan = true
local_ip = 10.88.1.2
l2_population = true
[root@controller ml2]#

--------------------------------------------------------------------------------------------------------------
/etc/neutron/plugins/ml2/ml2_conf.ini:

[root@controller ml2]# cat ml2_conf.ini |egrep -v "^$|^#"
[DEFAULT]
[l2pop]
[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vxlan
mechanism_drivers = linuxbridge,l2population
extension_drivers = port_security
[ml2_type_flat]
flat_networks = provider
[ml2_type_geneve]
[ml2_type_gre]
[ml2_type_vlan]
[ml2_type_vxlan]
vni_ranges = 1:1000
[securitygroup]
enable_ipset = true
[root@controller ml2]#

--------------------------------------------------------------------------------------------------------------
/etc/neutron/plugins/ml2/sriov_agent.ini:

[root@controller ml2]# cat sriov_agent.ini |egrep -v "^$|^#"
[DEFAULT]
[agent]
[sriov_nic]
[root@controller ml2]#

<Compute Node>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
--------------------------------------------------------------------------------------------------------------
/etc/neutron/neutron/

[root@compute1 neutron]# cat neutron.conf | egrep -v "^#|^$"
[DEFAULT]
transport_url = rabbit://openstack:openstack@controller
auth_strategy = keystone
[agent]
[cors]
[database]
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = neutron
password = neutron
[matchmaker_redis]
[nova]
[oslo_concurrency]
lock_path = /var/lib/neutron/tmp
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
[oslo_messaging_zmq]
[oslo_middleware]
[oslo_policy]
[quotas]
[ssl]
[root@compute1 neutron]#

--------------------------------------------------------------------------------------------------------------
/etc/neutron/plugins/ml2/linuxbridge_agent.ini:

[root@compute1 ml2]# cat linuxbridge_agent.ini | egrep -v "^#|^$"
[DEFAULT]
[agent]
[linux_bridge]
physical_interface_mappings = provider:eth2
[securitygroup]
enable_security_group = true
firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver
[vxlan]
enable_vxlan = true
local_ip = 10.88.1.10
l2_population = true
[root@compute1 ml2]#

Aaasidncza (aascigsz2) on 2017-12-14

summary:

- Networking Option 2: Self-service networks in neutron
+ NeutronServerTrytoFindL3agentOnComputeNodewhenWeUseLinuxBridge

Revision history for this message

Avtar singh (avtarsingh12015) wrote on 2017-12-19:

Download full text (13.3 KiB)

I am facing the same issue after the update from Centos 7.3 to Centos 7.4. Here are the compute linuxbridge logs :-

=================================
==> neutron/linuxbridge-agent.log <==
2017-12-19 06:15:52.481 39537 INFO neutron.agent.securitygroups_rpc [req-f8952149-d0bd-4a2a-862a-3bd6447bf853 - - - - -] Security group member updated [u'c0580b27-a250-4b63-94f2-81cf32f75518']
 2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-f8952149-d0bd-4a2a-862a-3bd6447bf853 - - - - -] Error occurred while removing port tap7f94b3a4-1c: RemoteError: Remote error: AgentNotFoundByTypeHost Agent with agent_type=L3 agent and host=compute1 could not be found
[u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 234, in update_device_down\n    n_const.PORT_STATUS_DOWN, host)\n', u'  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 331, in notify_l2pop_port_wiring\n    l2pop_driver.obj.update_port_down(port_context)\n', u'  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 253, in update_port_down\n    admin_context, agent_host, [port[\'device_id\']]):\n', u'  File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 303, in list_router_ids_on_host\n    context, constants.AGENT_TYPE_L3, host)\n', u'  File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 291, in _get_agent_by_type_and_host\n    host=host)\n', u'AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and host=compute1 could not be found\n'].
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 336, in treat_devices_removed
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent     cfg.CONF.host)
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 139, in update_device_down
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent     agent_id=agent_id, host=host)
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 162, in call
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return self._original_context.call(ctxt, method, **kwargs)
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent     retry=self.retry)
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent     timeout=timeout, retry=retry)
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent     retry=retry)
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent     raise result
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent RemoteError: Remote error: AgentNotFoundByTypeHost Agent with agent_type=L3 agent and host=compute1 could not be found
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 234, in update_device_down\n    n_const.PORT_STATUS_DOWN, host)\n', u'  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 331, in notify_l2pop_port_wiring\n    l2pop_driver.obj.update_port_down(port_context)\n', u'  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 253, in update_port_down\n    admin_context, agent_host, [port[\'device_id\']]):\n', u'  File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 303, in list_router_ids_on_host\n    context, constants.AGENT_TYPE_L3, host)\n', u'  File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 291, in _get_agent_by_type_and_host\n    host=host)\n', u'AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and host=compute1 could not be found\n'].
2017-12-19 06:15:52.633 39537 ERROR neutron.plugins.ml2.drivers.agent._common_agent 
2017-12-19 06:15:52.670 39537 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-f8952149-d0bd-4a2a-862a-3bd6447bf853 - - - - -] Linux bridge agent Agent out of sync with plugin!
2017-12-19 06:15:52.671 39537 INFO neutron.agent.securitygroups_rpc [req-f8952149-d0bd-4a2a-862a-3bd6447bf853 - - - - -] Preparing filters for devices set(['tap7f94b3a4-1c', 'tapb0f5adaa-7f', 'tap51122b50-f3', 'tap2674a607-48'])
2017-12-19 06:15:53.061 39537 INFO neutron.agent.securitygroups_rpc [req-f8952149-d0bd-4a2a-862a-3bd6447bf853 - - - - -] Security group member updated [u'c0580b27-a250-4b63-94f2-81cf32f75518']
2017-12-19 06:15:53.317 39537 INFO neutron.agent.securitygroups_rpc [req-f8952149-d0bd-4a2a-862a-3bd6447bf853 - - - - -] Security group member updated [u'aa68a99f-e13e-4f7d-accb-a7dea4178e0d']
2017-12-19 06:15:53.477 39537 INFO neutron.agent.securitygroups_rpc [req-f8952149-d0bd-4a2a-862a-3bd6447bf853 - - - - -] Security group member updated [u'6282dc55-b631-4617-8787-b36d30c79d4d']
2017-12-19 06:15:53.580 39537 INFO neutron.agent.securitygroups_rpc [req-f8952149-d0bd-4a2a-862a-3bd6447bf853 - - - - -] Security group member updated [u'6282dc55-b631-4617-8787-b36d30c79d4d']
2017-12-19 06:15:53.596 39537 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-f8952149-d0bd-4a2a-862a-3bd6447bf853 - - - - -] Port tap7f94b3a4-1c updated. Details: {u'profile': {}, u'network_qos_policy_id': None, u'qos_policy_id': None, u'allowed_address_pairs': [{u'ip_address': u'10.100.0.0/16', u'mac_address': u'fa:16:3e:9b:a1:13'}], u'admin_state_up': True, u'network_id': u'57cfefb3-f5f2-44c6-b346-44c8b87d8f99', u'segmentation_id': 47, u'mtu': 1450, u'device_owner': u'compute:nova', u'physical_network': None, u'mac_address': u'fa:16:3e:9b:a1:13', u'device': u'tap7f94b3a4-1c', u'port_security_enabled': True, u'port_id': u'7f94b3a4-1c1f-40bc-8d23-521eee15059c', u'fixed_ips': [{u'subnet_id': u'df5f3161-38e4-4775-95d1-a0a9194c509b', u'ip_address': u'10.0.0.10'}], u'network_type': u'vxlan'}
2017-12-19 06:15:54.146 39537 INFO neutron.agent.securitygroups_rpc [req-f8952149-d0bd-4a2a-862a-3bd6447bf853 - - - - -] Security group member updated [u'c0580b27-a250-4b63-94f2-81cf32f75518']
2017-12-19 06:15:54.430 39537 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-f8952149-d0bd-4a2a-862a-3bd6447bf853 - - - - -] Port tapb0f5adaa-7f updated. Details: {u'profile': {}, u'network_qos_policy_id': None, u'qos_policy_id': None, u'allowed_address_pairs': [{u'ip_address': u'10.100.0.0/16', u'mac_address': u'fa:16:3e:bc:1d:ec'}], u'admin_state_up': True, u'network_id': u'57cfefb3-f5f2-44c6-b346-44c8b87d8f99', u'segmentation_id': 47, u'mtu': 1450, u'device_owner': u'compute:nova', u'physical_network': None, u'mac_address': u'fa:16:3e:bc:1d:ec', u'device': u'tapb0f5adaa-7f', u'port_security_enabled': True, u'port_id': u'b0f5adaa-7f91-4d4b-9181-0af7a0d5d2b1', u'fixed_ips': [{u'subnet_id': u'df5f3161-38e4-4775-95d1-a0a9194c509b', u'ip_address': u'10.0.0.3'}], u'network_type': u'vxlan'}

==========================

Controller
==========================
==> server.log <==
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server [req-f8952149-d0bd-4a2a-862a-3bd6447bf853 - - - - -] Exception during message handling: AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and host=compute1 could not be found
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 234, in update_device_down
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server     n_const.PORT_STATUS_DOWN, host)
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 331, in notify_l2pop_port_wiring
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server     l2pop_driver.obj.update_port_down(port_context)
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 253, in update_port_down
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server     admin_context, agent_host, [port['device_id']]):
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 303, in list_router_ids_on_host
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server     context, constants.AGENT_TYPE_L3, host)
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 291, in _get_agent_by_type_and_host
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server     host=host)
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and host=compute1 could not be found
2017-12-19 06:17:50.299 22721 ERROR oslo_messaging.rpc.server

================================

Any pointer will be helpful

Revision history for this message

Aaasidncza (aascigsz2) wrote on 2017-12-20:

wonder if it's a same bug as https://bugs.launchpad.net/neutron/+bug/1456822

Revision history for this message

Georgy (suquant) wrote on 2017-12-30:

After restart

neutron-linuxbridge-cleanup.service
neutron-linuxbridge-agent.service
nova-compute.service

the problem may disappear!

Revision history for this message

Aalaesar (aalaesar) wrote on 2018-01-17:

Hi There
hitting the very same issue each time OpenStack stop an instance for migration or for stop
on ubuntu 16.04, followed Pike's installation guide for self-service network

the error appears on a compute host, each time the linux bridge agent needs to delete a port.
- it happen only once if the instance is destroyed
- it happen in loop if the instance is stopped or migrated.
The loop can be stopped by restarting neutron-linuxbridge-cleanup.service or neutron-linuxbridge-agent.service. But the error come back as soon as another instance is stopped or migrated.

I had also noted (but note tested) some network performances drop on the instances in the same compute host during the error loops.

No errors logged on the neutron controller side...

I'm searching for configuration references to see if it's a config issue, but no luck yet.

Any hint ?
Regards,
Aal

Revision history for this message

Chris Apsey (bitskrieg) wrote on 2018-01-29:

All,

Also seeing this on Pike (UCA/Ubuntu 16.04) w/ linuxbridge and all neutron agents on dedicated network nodes. Errors about missing l3agents on compute nodes appears in the logs on both compute nodes and the dedicated network nodes, although it is much more severe on the compute nodes (if I ignore it for too long, vif plugging starts to fail for new instances unless I restart linuxbridge-agent). I have never installed l3agent on compute nodes, and can find no references to it anywhere within the database, so I'm not sure why neutron thinks there should be l3agents on compute nodes.

Is this a new best practice/requirement that we missed?

Revision history for this message

Robert Putt (robert-putt) wrote on 2018-02-14:

Hi,

Also experiencing this with a fresh Pike deployment in my OpenStack playground :-(. It is unfortunate as it means if someone is new to Pike and follow the installation guide online their OpenStack deployment is fundamentally broken. It would be great if we could get some more information about this. I am not going to bother posting my logs as they are just like the ones from the previous posts.

Best Regards,

Rob

Revision history for this message

Chris Apsey (bitskrieg) wrote on 2018-03-15:

#10

Just want to add that this is still happening on Queens with UCA packages on 16.04.

Revision history for this message

Lucas Costa Beyeler (lucas.beyeler) wrote on 2018-03-16:

#11

A new information: because of this bug I'm losing my instances when I try to shutdown/start. And like Chris said, the bugs still happens in Queens.

Revision history for this message

zhaobo (zhaobo6) wrote on 2018-03-19:

#12

For me, I think it's a Neutron Bug. The check in http://git.openstack.org/cgit/openstack/neutron/tree/neutron/plugins/ml2/drivers/l2pop/mech_driver.py#n246 should be change.

Changed in neutron:
status:	New → Confirmed
importance:	Undecided → Medium

Slawek Kaplonski (slaweq) on 2018-03-19

Changed in neutron:
assignee:	nobody → Slawek Kaplonski (slaweq)

Revision history for this message

Kevin Tibi (ktibi) wrote on 2018-03-26:

#13

Same bug on fresh pike install. And I use OVS.

Revision history for this message

Georgii (georgii.sytov) wrote on 2018-03-29:

#14

OS: Centos
Fresh install Pike.
Same bug.

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2018-03-29:

#15

Thx guys for confirming that this issue happens for many of You. I started checking it and I hope I will push fix in next few days (it will be probably next week because of Easter) :)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-30: Fix proposed to neutron (master)

#16

Fix proposed to branch: master
Review: https://review.openstack.org/557908

Changed in neutron:
status:	Confirmed → In Progress

Revision history for this message

Kevin Tibi (ktibi) wrote on 2018-03-30:

#17

Hi Slawek,

I use kolla with pike version. I use three controllers. I patch the first (ctrl01) with your patch but I have always the error log on each controller.

"Hostname": "ctrl01","Payload": "Failed to update device cb5f787c-8cd9-4995-a423-80dd6ea5838c down: AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and host=compute02 could not be found"

"Hostname": "ctrl02","Payload": "Failed to update device cb5f787c-8cd9-4995-a423-80dd6ea5838c down: AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and host=compute02 could not be found"

"Hostname": "ctrl03","Payload": "Failed to update device cb5f787c-8cd9-4995-a423-80dd6ea5838c down: AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and host=compute02 could not be found"

patch apply on ctrl01 on "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py" :

from neutron_lib.exceptions import agent as agent_exc
......
    def list_router_ids_on_host(self, context, host, router_ids=None):
        try:
            agent = self._get_agent_by_type_and_host(
                context, constants.AGENT_TYPE_L3, host)
        except agent_exc.AgentNotFoundByTypeHost:
            LOG.debug("L3 Agent not found on host %s", host)
            return []
        if not agentschedulers_db.services_available(agent.admin_state_up):
            return []
        return self._get_router_ids_for_agent(context, agent, router_ids)

Th for your time ;)

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2018-04-02:

#18

Hi Kevin,

Can You send me whole log from neutron-server and L3 agent where You have still this issue?

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2018-04-04:

#19

@Kevin: are You sure that Your patch is applied properly? I compared for example fullstack logs from any (random) patch without my change to fix this issue and there is plenty of such error messages there, see e.g.: http://logs.openstack.org/77/558677/1/check/neutron-fullstack/ebea841/logs/dsvm-fullstack-logs/TestSecurityGroupsSameNetwork.test_securitygroup_linuxbridge-iptables_/neutron-server--2018-04-04--09-22-02-354194.txt.gz?level=ERROR
in fullstack logs from same test from my patch proposed to fix this issue there is no any of such errors in logs: http://logs.openstack.org/08/557908/1/check/neutron-fullstack/8e35c0b/logs/dsvm-fullstack-logs/TestSecurityGroupsSameNetwork.test_securitygroup_linuxbridge-iptables_/neutron-server--2018-03-30--09-19-31-057586.txt.gz - there are only debug messages which my patch adds in such case.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-04-09: Fix merged to neutron (master)

#20

Reviewed: https://review.openstack.org/557908
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7b0f6330d6f877f3d2093a64c2bca4c14334574c
Submitter: Zuul
Branch: master

commit 7b0f6330d6f877f3d2093a64c2bca4c14334574c
Author: Sławek Kapłoński <email address hidden>
Date: Fri Mar 30 10:24:59 2018 +0200

Handle AgentNotFoundByTypeHost exception properly

    During listing router_ids on host it is possible that on some hosts
    there are no L3 agents.
    In such case AgentNotFoundByTypeHost exception is raised in
    neutron.db.agents_db module in _get_agent_by_type_and_host() method.
    Now this exception is properly handled during listing routers on host.

Change-Id: Ia5ff1b57ef63c98b4ada4f2d46c45336e413be3d
Closes-Bug: #1737917

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-04-10: Fix proposed to neutron (stable/queens)

#21

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/559932

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-04-13: Fix merged to neutron (stable/queens)

#22

Reviewed: https://review.openstack.org/559932
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7a67987a646e64255984fa13cd7433351099c46d
Submitter: Zuul
Branch: stable/queens

commit 7a67987a646e64255984fa13cd7433351099c46d
Author: Sławek Kapłoński <email address hidden>
Date: Fri Mar 30 10:24:59 2018 +0200

Handle AgentNotFoundByTypeHost exception properly

    Change-Id: Ia5ff1b57ef63c98b4ada4f2d46c45336e413be3d
    Closes-Bug: #1737917
    (cherry picked from commit 7b0f6330d6f877f3d2093a64c2bca4c14334574c)

tags:

added: in-stable-queens

Revision history for this message

Andreas (fattony666) wrote on 2018-04-19:

#23

Hi,

Issue as well on Queens, Ubuntu 16.04.4 LTS.
Is there a workaround or fix already available that has been released?

We have "neutron-l3-agent/xenial-updates,xenial-updates,now 2:12.0.0-0ubuntu1.4~cloud0 all [installed]" installed, so far there is no update available for neutron-l3-agent package.

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2018-04-19:

#24

Fix for stable/queens should be already merged: https://review.openstack.org/#/c/559932/
Do You still have this issue with this patch applied?

Revision history for this message

Andreas (fattony666) wrote on 2018-04-19:

#25

I applied the changes from https://review.openstack.org/#/c/559932/2/neutron/db/l3_agentschedulers_db.py and https://review.openstack.org/#/c/559932/2/neutron/tests/unit/db/test_agentschedulers_db.py, error messages are now gone after rebooting instance.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-04-19: Fix included in openstack/neutron 13.0.0.0b1

#26

This issue was fixed in the openstack/neutron 13.0.0.0b1 development milestone.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-04-27: Fix proposed to neutron (stable/pike)

#27

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/564755

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-08: Fix included in openstack/neutron 12.0.2

#28

This issue was fixed in the openstack/neutron 12.0.2 release.

Revision history for this message

Crazik (crazik) wrote on 2018-06-14:

#29

Fingers crossed and waiting for package for Pike.

Revision history for this message

Crazik (crazik) wrote on 2018-06-25:

#30

@slaweq Looks like you made a typo for Pike commit:

You have:

`from neutron_lib.exceptions import agent as agent_exc`

it should be (analogy to Queens' patch)

`from neutron.extensions import agent as agent_exc`

Revision history for this message

Crazik (crazik) wrote on 2018-06-26:

#31

Sorry for the mess, I was looking at other commit.
Patch is ok, issue fixed in Pike release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-06-27: Fix merged to neutron (stable/pike)

#32

Reviewed: https://review.openstack.org/564755
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cb0afda244f87ef1e84b6b4b7ed6e759962d95fa
Submitter: Zuul
Branch: stable/pike

commit cb0afda244f87ef1e84b6b4b7ed6e759962d95fa
Author: Sławek Kapłoński <email address hidden>
Date: Fri Mar 30 10:24:59 2018 +0200

Handle AgentNotFoundByTypeHost exception properly

    Change-Id: Ia5ff1b57ef63c98b4ada4f2d46c45336e413be3d
    Closes-Bug: #1737917
    (cherry picked from commit 7b0f6330d6f877f3d2093a64c2bca4c14334574c)