Neutron firewall_l3_agent failed to create firewall through Attribute Error. firewall stuck in "PENDING_CREATE" state and in "PENDING_DELETE" while deleting the same.

Tests through error as firewall policy is still in use with failed firewall.


Console log-

2014-10-13 23:15:21.180 |     2014-10-13 22:43:28,278 25997 INFO     [tempest.common.rest_client] Request (FWaaSExtensionTestJSON:test_create_show_delete_firewall): 201 POST http://127.0.0.1:9696/v2.0/fw/firewalls 0.061s
2014-10-13 23:15:21.180 |     2014-10-13 22:43:28,278 25997 DEBUG    [tempest.common.rest_client] Request (FWaaSExtensionTestJSON:test_create_show_delete_firewall): 201 POST http://127.0.0.1:9696/v2.0/fw/firewalls 0.061s
2014-10-13 23:15:21.181 |         Request - Headers: {'Content-Type': 'application/json', 'X-Auth-Token': '<omitted>', 'Accept': 'application/json'}
2014-10-13 23:15:21.181 |             Body: {"firewall": {"firewall_policy_id": "1af273a9-b299-4e5c-81cf-580398ae4ec7", "name": "firewall-1257536380"}}
2014-10-13 23:15:21.181 |         Response - Headers: {'content-type': 'application/json; charset=UTF-8', 'date': 'Mon, 13 Oct 2014 22:43:28 GMT', 'content-length': '273', 'status': '201', 'connection': 'close', 'x-openstack-request-id': 'req-5a6b5a71-ee4f-4c89-a90b-0e17a0b7f929'}
2014-10-13 23:15:21.181 |             Body: {"firewall": {"status": "PENDING_CREATE", "name": "firewall-1257536380", "admin_state_up": true, "tenant_id": "b51b3b4153794b18933ac1c3b26f11bd", "firewall_policy_id": "1af273a9-b299-4e5c-81cf-580398ae4ec7", "id": "5b12dcb1-5805-4123-833b-6cdc1b123a62", "description": ""}}

After 5 min (network build timeout)

2014-10-13 23:15:21.689 |     2014-10-13 22:48:27,791 25997 DEBUG    [tempest.common.rest_client] Request (FWaaSExtensionTestJSON:test_create_show_delete_firewall): 200 GET http://127.0.0.1:9696/v2.0/fw/firewalls/5b12dcb1-5805-4123-833b-6cdc1b123a62 0.370s
2014-10-13 23:15:21.689 |         Request - Headers: {'Content-Type': 'application/json', 'X-Auth-Token': '<omitted>', 'Accept': 'application/json'}
2014-10-13 23:15:21.689 |             Body: None
2014-10-13 23:15:21.689 |         Response - Headers: {'content-type': 'application/json; charset=UTF-8', 'date': 'Mon, 13 Oct 2014 22:48:27 GMT', 'content-length': '273', 'status': '200', 'content-location': 'http://127.0.0.1:9696/v2.0/fw/firewalls/5b12dcb1-5805-4123-833b-6cdc1b123a62', 'connection': 'close', 'x-openstack-request-id': 'req-e95acaaa-81c3-4415-b4df-917c8427a160'}
2014-10-13 23:15:21.689 |             Body: {"firewall": {"status": "PENDING_CREATE", "name": "firewall-1257536380", "admin_state_up": true, "tenant_id": "b51b3b4153794b18933ac1c3b26f11bd", "firewall_policy_id": "1af273a9-b299-4e5c-81cf-580398ae4ec7", "id": "5b12dcb1-5805-4123-833b-6cdc1b123a62", "description": ""}}


screen-q-vpn logs-

2014-10-13 22:43:29.445 21229 ERROR neutron.services.firewall.agents.l3reference.firewall_l3_agent [req-5a6b5a71-ee4f-4c89-a90b-0e17a0b7f929 None] FWaaS RPC failure in create_firewall for fw: 5b12dcb1-5805-4123-833b-6cdc1b123a62
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent Traceback (most recent call last):
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/opt/stack/new/neutron/neutron/services/firewall/agents/l3reference/firewall_l3_agent.py", line 122, in _invoke_driver_for_plugin_api
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     routers = self.plugin_rpc.get_routers(context)
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/opt/stack/new/neutron/neutron/agent/l3_agent.py", line 108, in get_routers
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     router_ids=router_ids))
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/opt/stack/new/neutron/neutron/common/log.py", line 34, in wrapper
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     return method(*args, **kwargs)
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/opt/stack/new/neutron/neutron/common/rpc.py", line 161, in call
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     context, msg, rpc_method='call', **kwargs)
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/opt/stack/new/neutron/neutron/common/rpc.py", line 187, in __call_rpc_method
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     return func(context, msg['method'], **msg['args'])
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 389, in call
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     return self.prepare().call(ctxt, method, **kwargs)
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     retry=self.retry)
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     timeout=timeout, retry=retry)
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     retry=retry)
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 399, in _send
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     raise result
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent AttributeError: _oslo_messaging_localcontext_5d77b2ce83274736837731b4a4be3ff1
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent Traceback (most recent call last):
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent 
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     incoming.message))
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent 
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 179, in _dispatch
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     localcontext.clear_local_context()
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent 
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/localcontext.py", line 55, in clear_local_context
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     delattr(_STORE, _KEY)
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent 
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent AttributeError: _oslo_messaging_localcontext_5d77b2ce83274736837731b4a4be3ff1
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent 
2014-10-13 22:43:29.445 21229 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent