Test instances are not removed in Horizon

[Anastasia Palkina]

"build_id": "2014-01-08_01-17-41",
"ostf_sha": "05b1bfc92fa40728966518992c94176030c25f35",
"build_number": "21",
"nailgun_sha": "ea5dc470d3b789ad8dc0be2c256b9c7ec6ac8500",
"fuelmain_sha": "73fb5d8bba72cb032b7ed4d034827b75042d0c47",
"astute_sha": "419e864c77681c355043c21e76b3f5a940bef9ea",
"release": "4.1",
"fuellib_sha": "f661314923ce584ad84c40c7a99e80a8feb4e713"

1. Create new environment (CentOS, simple mode)
2. Choose Neutron VLAN segmentation
3. Add controller, compute and cinder
4. Start deployment. It was successful
5. Start OSTF tests. It was successful
6. After it test instances are not removed (see screen)

[Tatyanka (tatyana-leontovich)]

Security group do not deleted, according instance that use sec group failed to delete. As result Deletion of security group failed with error
BadRequest: 409-{u'NeutronError': {u'message': u'Security Group 25d7e92e-cb82-4307-a3c8-5aca3144cee8 in use.', u'type': u'SecurityGroupInUse', u'detail': u''}} (HTTP 400) (Request-ID: req-0e310564-77c0-4cc8-8d3e-3210da4317f8)

[Tatyanka (tatyana-leontovich)]

Also there are a lot of erros like:
2014-01-09T14:14:06.992459+00:00 emerg: ERROR: Failed storing info cache
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 75, in update_instance_cache_with_nw_info
    nw_info = api._get_instance_nw_info(context, instance)
  File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 465, in _get_instance_nw_info
    nw_info = self._build_network_info_model(context, instance, networks)
  File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 1011, in _build_network_info_model
    subnets)
  File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 964, in _nw_info_build_network
    label=network_name,
UnboundLocalError: local variable 'network_name' referenced before assignment

we have bug for this trace, but I can not find them(

[Tatyanka (tatyana-leontovich)]

Also as we can see in logs we failed here when neutron agent try to fetch metadata (neutron.agent.metadata.agent)
failed with the same error as described above

[Tatyanka (tatyana-leontovich)]

and in nova-nova.network-api.netron2.log warning about 2014-01-09T14:26:18.492004+00:00 emerg: WARNING: No network configured!

[Tatyanka (tatyana-leontovich)]

On the second deployment on the same host - 2014-01-09T13:13:35.962098+00:00 debug: 2014-01-09 13:13:26.272 20875 INFO neutron.agent.l3_agent [-] L3 agent started
2014-01-09T13:13:35.962098+00:00 debug: 2014-01-09 13:13:26.273 20875 INFO neutron.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.0.5:5672
2014-01-09T13:13:35.962106+00:00 debug: 2014-01-09 13:13:26.477 20875 ERROR neutron.agent.l3_agent [-] Failed synchronizing routers
2014-01-09T13:13:35.962106+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent Traceback (most recent call last):
2014-01-09T13:13:35.962110+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent File "/usr/lib/python2.6/site-packages/neutron/agent/l3_agent.py", line 723, in _sync_routers_task
2014-01-09T13:13:35.962110+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent context, router_ids)
2014-01-09T13:13:35.962223+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent File "/usr/lib/python2.6/site-packages/neutron/agent/l3_agent.py", line 76, in get_routers
2014-01-09T13:13:35.962330+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent topic=self.topic)
2014-01-09T13:13:35.962473+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/proxy.py", line 126, in call
2014-01-09T13:13:35.962672+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent result = rpc.call(context, real_topic, msg, timeout)
2014-01-09T13:13:35.962800+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/__init__.py", line 140, in call
2014-01-09T13:13:35.962961+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent return _get_impl().call(CONF, context, topic, msg, timeout)
2014-01-09T13:13:35.962961+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 796, in call
2014-01-09T13:13:35.964346+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent rpc_amqp.get_connection_pool(conf, Connection))
2014-01-09T13:13:35.964346+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 551, in call
2014-01-09T13:13:35.964358+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent rv = list(rv)
2014-01-09T13:13:35.964358+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 516, in __iter__
2014-01-09T13:13:35.964372+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent raise result
2014-01-09T13:13:35.964372+00:00 debug: 2014-01-09 13:13:26.477 20875 TRACE neutron.agent.l3_agent RemoteError: Remote error: AgentNotFoundByTypeHost Agent with agent_type=L3 agent and host=node-4.domain.tld could not be found
2014-01-09T13:13:35.96446...

[Sergey Vasilenko (xenolog)]

Exception "neutron.agent.l3_agent RemoteError: Remote error: AgentNotFoundByTypeHost" not matter in this case.
I think we got this exception because log level TRACE is enabled.

What happens? At first start l3 agent send (by amqp) message to the Neutron server. But in database not found records about any l3-agent, and server got exception. This is normal situation. But log-level=trace and this traceback logged.

[Dmitry Pyzhov (dpyzhov)]

Not reproducible

[Tatyanka (tatyana-leontovich)]

Nastya, could you please re-verify if issue is still here

[Anastasia Palkina (apalkina)]

Cannot reproduce on ISO #56