neutron-server hangs at initializing driver for type vlan

[Jerry Zhao]

i used nova boot to launch about 50 vms. some of them were active eventually while the rest were in error state.
I later found that neutron-server was not returning anything for wget neutron-host:9696
in the neutron-server log:
Dec 17 14:14:57 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:14:57.099 13651 WARNING neutron.db.l3_agentschedulers_db [-] Time since last L3 agent reschedule check has exceeded the interval between checks. Waiting before check to allow agents to send a heartbeat in case there was a clock adjustment.

That's it. the last log that was printed.

then i tried to restart neutron-server with service neutron-server restart.However, nothing was printed.

i later enabled debug level for the log and the following logs were printed:

Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.852 25846 DEBUG neutron.openstack.common.lockutils [-] Created new semaphore "manager" internal_lock /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py:206
Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.852 25846 DEBUG neutron.openstack.common.lockutils [-] Acquired semaphore "manager" lock /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py:229
Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.853 25846 DEBUG neutron.openstack.common.lockutils [-] Got semaphore / lock "_create_instance" inner /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py:271
Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.853 25846 INFO neutron.manager [-] Loading core plugin: neutron.plugins.ml2.plugin.Ml2Plugin
Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.992 25846 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'gre', 'vxlan']
Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.994 25846 INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['datacentre']
Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.998 25846 INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.003 25846 INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: {'datacentre': []}
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.003 25846 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'gre', 'local', 'vxlan', 'vlan']
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.003 25846 INFO neutron.plugins.ml2.managers [-] Registered types: ['vlan', 'flat', 'gre', 'local', 'vxlan']
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.004 25846 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['gre']
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.004 25846 INFO neutron.plugins.ml2.managers [-] Configured extension driver names: []
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.004 25846 INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: []
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.004 25846 INFO neutron.plugins.ml2.managers [-] Registered extension drivers: []
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.005 25846 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['openvswitch']
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.006 25846 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['openvswitch']
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.006 25846 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['openvswitch']
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.019 25846 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'vlan'

The neutron server seems to be stuck at initializing driver for type vlan.

[Eugene Nikanorov (enikanorov)]

What vlan range have you configured for vlans?

[Jerry Zhao (zhaoxinyu)]

the net i created is flat provider network, not vlan mode.

[Eugene Nikanorov (enikanorov)]

So, you claim that neutron-server hangs at startup, correct?
Could you paste your neutron.conf and ml2_conf.ini to paste.openstack.org?

[Jerry Zhao (zhaoxinyu)]

neutron.conf http://paste.openstack.org/show/153822/
ml2_conf.ini http://paste.openstack.org/show/153823/

the neutron server could successfully start at first, and vm created was accessible, but after launching a few more vms straight in a few period of time, the neutron-server began to stuck and after executing service restart, it hang at loading driver for type vlan

[Jerry Zhao (zhaoxinyu)]

i was just able to reproduce the scenario, after creating 40 vms in a row, the neutron server hung again. with neutron-server log complaining no more dhcp agent.

Dec 23 08:45:36 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:36.516 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-70de0656-24f4-4cec-96ed-ceb47e0ad6b7 None] No more DHCP agents
Dec 23 08:45:38 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:38.698 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-1145f488-1fb1-46a1-80f6-85650ad90667 None] No more DHCP agents
Dec 23 08:45:41 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:41.985 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-cb2111ec-392d-4834-95bf-32d0953af03c None] No more DHCP agents
Dec 23 08:45:43 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:43.728 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-93eeeea6-a547-487d-a0b3-9415cee8bf33 None] No more DHCP agents
Dec 23 08:45:45 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:45.098 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-44ae62c2-8d09-44ba-bb5b-ad98eb1e27e3 None] No more DHCP agents
Dec 23 08:45:46 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:46.890 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-b6c118cf-c941-4053-8696-f1f152b56d9a None] No more DHCP agents
Dec 23 08:45:48 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:48.587 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-8a66b40e-5a69-4586-a097-dd4feb4016c9 None] No more DHCP agents
Dec 23 08:45:51 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:51.404 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-238f8a12-cf96-4fac-a15a-22b1168d07d7 None] No more DHCP agents
Dec 23 08:45:57 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:57.779 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-fccbdfaa-2013-4838-9168-b5598b72cee1 None] No more DHCP agents
Dec 23 08:45:59 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:59.234 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-c2a291c9-8ed5-46b9-af32-1358279d6315 None] No more DHCP agents

neutron dhcp agent log:
o AMQP server on 162.3.121.108:5672
Dec 18 04:56:20 ci-overcloud-controller0-oxzkjphwfyw3 neutron-dhcp-agent: 2014-12-18 04:56:20.762 8518 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
Dec 18 04:56:20 ci-overcloud-controller0-oxzkjphwfyw3 neutron-dhcp-agent: 2014-12-18 04:56:20.762 8518 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
Dec 18 04:56:20 ci-overcloud-controller0-oxzkjphwfyw3 neutron-dhcp-agent: 2014-12-18 04:56:20.762 8518 TRACE oslo.messaging._drivers.impl_rabbit File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
Dec 18 04:56:20 ci-overcloud-controller0-oxzkjphwfyw3 neutron-dhcp-agent: 2014-12-18 04:56:20.762 8518 TRACE o...

[Jerry Zhao (zhaoxinyu)]

the problem seems to be related to a pecona mysql exception.after restarting mysql, neutron can be started successfully.
but the root cause is still unknown.

[Gary Kotton (garyk)]

Does this problem still occur?

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]