neutron-server hangs at initializing driver for type vlan

Bug #1403528 reported by Jerry Zhao
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Expired
Medium
Unassigned

Bug Description

i used nova boot to launch about 50 vms. some of them were active eventually while the rest were in error state.
I later found that neutron-server was not returning anything for wget neutron-host:9696
in the neutron-server log:
Dec 17 14:14:57 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:14:57.099 13651 WARNING neutron.db.l3_agentschedulers_db [-] Time since last L3 agent reschedule check has exceeded the interval between checks. Waiting before check to allow agents to send a heartbeat in case there was a clock adjustment.

That's it. the last log that was printed.

then i tried to restart neutron-server with service neutron-server restart.However, nothing was printed.

i later enabled debug level for the log and the following logs were printed:

Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.852 25846 DEBUG neutron.openstack.common.lockutils [-] Created new semaphore "manager" internal_lock /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py:206
Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.852 25846 DEBUG neutron.openstack.common.lockutils [-] Acquired semaphore "manager" lock /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py:229
Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.853 25846 DEBUG neutron.openstack.common.lockutils [-] Got semaphore / lock "_create_instance" inner /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py:271
Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.853 25846 INFO neutron.manager [-] Loading core plugin: neutron.plugins.ml2.plugin.Ml2Plugin
Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.992 25846 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'gre', 'vxlan']
Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.994 25846 INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['datacentre']
Dec 17 14:34:42 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:42.998 25846 INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.003 25846 INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: {'datacentre': []}
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.003 25846 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'gre', 'local', 'vxlan', 'vlan']
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.003 25846 INFO neutron.plugins.ml2.managers [-] Registered types: ['vlan', 'flat', 'gre', 'local', 'vxlan']
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.004 25846 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['gre']
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.004 25846 INFO neutron.plugins.ml2.managers [-] Configured extension driver names: []
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.004 25846 INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: []
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.004 25846 INFO neutron.plugins.ml2.managers [-] Registered extension drivers: []
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.005 25846 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['openvswitch']
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.006 25846 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['openvswitch']
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.006 25846 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['openvswitch']
Dec 17 14:34:43 ci-overcloud-controller1-mqrd73cq4fgs neutron-server: 2014-12-17 14:34:43.019 25846 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'vlan'

The neutron server seems to be stuck at initializing driver for type vlan.

Tags: ml2
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

What vlan range have you configured for vlans?

Changed in neutron:
status: New → Incomplete
Revision history for this message
Jerry Zhao (zhaoxinyu) wrote :

the net i created is flat provider network, not vlan mode.

Changed in neutron:
status: Incomplete → New
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

So, you claim that neutron-server hangs at startup, correct?
Could you paste your neutron.conf and ml2_conf.ini to paste.openstack.org?

Changed in neutron:
status: New → Incomplete
importance: Undecided → Medium
tags: added: ml2
Revision history for this message
Jerry Zhao (zhaoxinyu) wrote :

neutron.conf http://paste.openstack.org/show/153822/
ml2_conf.ini http://paste.openstack.org/show/153823/

the neutron server could successfully start at first, and vm created was accessible, but after launching a few more vms straight in a few period of time, the neutron-server began to stuck and after executing service restart, it hang at loading driver for type vlan

Changed in neutron:
status: Incomplete → New
Revision history for this message
Jerry Zhao (zhaoxinyu) wrote :
Download full text (7.0 KiB)

i was just able to reproduce the scenario, after creating 40 vms in a row, the neutron server hung again. with neutron-server log complaining no more dhcp agent.

Dec 23 08:45:36 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:36.516 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-70de0656-24f4-4cec-96ed-ceb47e0ad6b7 None] No more DHCP agents
Dec 23 08:45:38 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:38.698 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-1145f488-1fb1-46a1-80f6-85650ad90667 None] No more DHCP agents
Dec 23 08:45:41 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:41.985 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-cb2111ec-392d-4834-95bf-32d0953af03c None] No more DHCP agents
Dec 23 08:45:43 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:43.728 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-93eeeea6-a547-487d-a0b3-9415cee8bf33 None] No more DHCP agents
Dec 23 08:45:45 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:45.098 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-44ae62c2-8d09-44ba-bb5b-ad98eb1e27e3 None] No more DHCP agents
Dec 23 08:45:46 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:46.890 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-b6c118cf-c941-4053-8696-f1f152b56d9a None] No more DHCP agents
Dec 23 08:45:48 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:48.587 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-8a66b40e-5a69-4586-a097-dd4feb4016c9 None] No more DHCP agents
Dec 23 08:45:51 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:51.404 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-238f8a12-cf96-4fac-a15a-22b1168d07d7 None] No more DHCP agents
Dec 23 08:45:57 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:57.779 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-fccbdfaa-2013-4838-9168-b5598b72cee1 None] No more DHCP agents
Dec 23 08:45:59 ci-overcloud-controller0-oxzkjphwfyw3 neutron-server: 2014-12-23 08:45:59.234 17796 WARNING neutron.scheduler.dhcp_agent_scheduler [req-c2a291c9-8ed5-46b9-af32-1358279d6315 None] No more DHCP agents

neutron dhcp agent log:
o AMQP server on 162.3.121.108:5672
Dec 18 04:56:20 ci-overcloud-controller0-oxzkjphwfyw3 neutron-dhcp-agent: 2014-12-18 04:56:20.762 8518 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
Dec 18 04:56:20 ci-overcloud-controller0-oxzkjphwfyw3 neutron-dhcp-agent: 2014-12-18 04:56:20.762 8518 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
Dec 18 04:56:20 ci-overcloud-controller0-oxzkjphwfyw3 neutron-dhcp-agent: 2014-12-18 04:56:20.762 8518 TRACE oslo.messaging._drivers.impl_rabbit File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
Dec 18 04:56:20 ci-overcloud-controller0-oxzkjphwfyw3 neutron-dhcp-agent: 2014-12-18 04:56:20.762 8518 TRACE o...

Read more...

Revision history for this message
Jerry Zhao (zhaoxinyu) wrote :

the problem seems to be related to a pecona mysql exception.after restarting mysql, neutron can be started successfully.
but the root cause is still unknown.

Revision history for this message
Gary Kotton (garyk) wrote :

Does this problem still occur?

Changed in neutron:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.