Errors in neutron-server logs for HA, CentOS: neutron.openstack.common.rpc.amqp [-] Exception during message handling

[Anastasia Palkina]

ISO #192
build_id": "2013-12-23_00-01-41",
"ostf_sha": "2a59b9a850f5c612ef4e8617395a60c27305283f",
"build_number": "192",
"nailgun_sha": "b6b89d2f145d243a570e86c914a14ab09cd5d84e",
"fuelmain_sha": "2e4b3e9b173050def41264feaeeeed47efc9b718",
"astute_sha": "a9cb3aad39b0fe9fc8c8399ef6d6e75850a1ed6d",
"release": "4.0",
"fuellib_sha": "a898b49da10bf5daddb069f60506b65d4c0b2f68"

1. Create new environment (CentOS, HA mode)
2. Choose VLAN segmentation and installing Ceilometer
3. Add 3 controllers, 1 compute and 1 cinder nodes
4. Start deployment. It was successful
5. But in logs neutron0server was many errors
neutron.openstack.common.rpc.amqp [-] Exception during message handling

3 OpenSwitch for first controller was in XXX state

Logs here (huge archive)
https://drive.google.com/a/mirantis.com/uc?id=0B6SjzarTGFxaN1p3Nk5IVm9ram8&export=download

[Sergey Vasilenko (xenolog)]

Some times after deploy we have 3 records about neutron-ovs-agent in neutron-db for one node.
Neutron server report about 3 dead agents instead 1 alive.

steps to reproduce this -- not clear.

In neutron-server logs we have exception, look like:

neutron.openstack.common.rpc.amqp [-] Exception during message handling
neutron.openstack.common.rpc.amqp Traceback (most recent call last):
neutron.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
neutron.openstack.common.rpc.amqp **args)
neutron.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/neutron/common/rpc.py", line 44, in dispatch
neutron.openstack.common.rpc.amqp neutron_ctxt, version, method, namespace, **kwargs)
neutron.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
neutron.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
neutron.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 186, in report_state
neutron.openstack.common.rpc.amqp self.plugin.create_or_update_agent(context, agent_state)
neutron.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 149, in create_or_update_agent
neutron.openstack.common.rpc.amqp context, agent['agent_type'], agent['host'])
neutron.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 132, in _get_agent_by_type_and_host
neutron.openstack.common.rpc.amqp host=host)
neutron.openstack.common.rpc.amqp MultipleAgentFoundByTypeHost: Multiple agents with agent_type=Open vSwitch agent and host=node-4.domain.tld found

[Sergey Vasilenko (xenolog)]

controller's logs