neutron

fwaas v2 hit driver internal error when restart l3 agent

Bug #1635156 reported by zhaobo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
zhaobo

Bug Description

There is a router serviced by l3 agent. And a firewall group located a port which serviced by the router, such as an interface of an internal subnet. Then I restart l3 agent, it will sync the server side router info and refresh the all related router info, include firewall iptables rules. It hit driver internal error during this period.

The trace like:
2016-10-20 15:05:11.587 DEBUG neutron.agent.linux.iptables_manager [-] IPTablesManager.apply completed with success. 16 iptables commands were issued from (pid=81933) _apply_synchronized /opt/stack/neutron/neutron/agent/linux/iptables_manager.py:533
2016-10-20 15:05:11.588 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "iptables-qrouter-cc5ab5a3-ef25-4496-87c7-5063cd167ce6" from (pid=81933) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
2016-10-20 15:05:11.589 DEBUG neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 [-] Process router update, router_id: cc5ab5a3-ef25-4496-87c7-5063cd167ce6 tenant: 488da3aab0ff45df9e85e17e7f89fedd. from (pid=81933) _process_router_update /opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py:239
2016-10-20 15:05:11.589 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'list'] from (pid=81933) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
2016-10-20 15:05:11.593 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141
2016-10-20 15:05:14.358 DEBUG oslo_policy._cache_handler [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloading cached file /etc/neutron/policy.json from (pid=81933) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:38
2016-10-20 15:05:14.470 DEBUG oslo_policy.policy [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloaded policy file: /etc/neutron/policy.json from (pid=81933) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:584
2016-10-20 15:05:14.471 DEBUG oslo_policy._cache_handler [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloading cached file /etc/neutron/policy.d/bgpvpn.conf from (pid=81933) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:38
2016-10-20 15:05:14.484 DEBUG oslo_policy.policy [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloaded policy file: /etc/neutron/policy.d/bgpvpn.conf from (pid=81933) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:584
2016-10-20 15:05:14.484 DEBUG oslo_policy._cache_handler [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloading cached file /etc/neutron/policy.d/dynamic_routing.conf from (pid=81933) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:38
2016-10-20 15:05:14.491 DEBUG oslo_policy.policy [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloaded policy file: /etc/neutron/policy.d/dynamic_routing.conf from (pid=81933) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:584
2016-10-20 15:05:14.491 DEBUG oslo_policy._cache_handler [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloading cached file /etc/neutron/policy.d/neutron-fwaas.json from (pid=81933) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:38
2016-10-20 15:05:14.502 DEBUG oslo_policy.policy [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloaded policy file: /etc/neutron/policy.d/neutron-fwaas.json from (pid=81933) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:584
2016-10-20 15:05:14.503 DEBUG neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Fetch firewall groups from plugin from (pid=81933) get_firewall_groups_for_project /opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py:43
2016-10-20 15:05:14.504 DEBUG oslo_messaging._drivers.amqpdriver [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] CALL msg_id: 2d47fe084441434db22bbc5e21861abc exchange 'neutron' topic 'q-firewall-plugin' from (pid=81933) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-10-20 15:05:14.505 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141
2016-10-20 15:05:14.505 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141
2016-10-20 15:05:14.506 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141
2016-10-20 15:05:14.506 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141
2016-10-20 15:05:14.507 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141
2016-10-20 15:05:14.508 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1662f95-21a5-494c-b14b-a6249e7179b7', 'ip6tables-save'] from (pid=81933) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
2016-10-20 15:05:14.525 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 2d47fe084441434db22bbc5e21861abc from (pid=81933) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-10-20 15:05:17.668 DEBUG neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Updating firewall 2932b3d9-3a7b-48a1-a16c-bf9f7b2751a5 for tenant 488da3aab0ff45df9e85e17e7f89fedd from (pid=81933) update_firewall_group /opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas_v2.py:131
2016-10-20 15:05:17.668 ERROR neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Failed to update firewall: 2932b3d9-3a7b-48a1-a16c-bf9f7b2751a5
2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 Traceback (most recent call last):
2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas_v2.py", line 139, in update_firewall_group
2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 apply_list, firewall)
2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas_v2.py", line 304, in _remove_conntrack_new_firewall
2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 routers_list = list(set([apply_info[0] for apply_info in apply_list]))
2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 KeyError: 0
2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2
2016-10-20 15:05:17.669 ERROR neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] FWaaS driver error on ACTIVE for firewall group: 2932b3d9-3a7b-48a1-a16c-bf9f7b2751a5
2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 Traceback (most recent call last):
2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py", line 192, in _invoke_driver_for_sync_from_plugin
2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 if firewall_group['status'] == n_const.PENDING_DELETE:
2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas_v2.py", line 147, in update_firewall_group
2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 raise fw_ext.FirewallInternalDriverError(driver=FWAAS_DRIVER_NAME)
2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 FirewallInternalDriverError: Fwaas iptables driver: Internal driver error.
2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2
2016-10-20 15:05:17.670 DEBUG neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Set firewall groups from plugin from (pid=81933) set_firewall_group_status /opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py:67
2016-10-20 15:05:17.671 DEBUG oslo_messaging._drivers.amqpdriver [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] CALL msg_id: f68ec0a4772941ebaacf410f5187c130 exchange 'neutron' topic 'q-firewall-plugin' from (pid=81933) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448

Tags: fwass
zhaobo (zhaobo6)
Changed in neutron:
assignee: nobody → zhaobo (zhaobo6)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-fwaas (master)

Fix proposed to branch: master
Review: https://review.openstack.org/389058

Changed in neutron:
status: New → In Progress
Anindita Das (anindita-das)
tags: added: fwass
Anindita Das (anindita-das)
Changed in neutron:
importance: Undecided → Medium
importance: Medium → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-fwaas (master)

Reviewed: https://review.openstack.org/389058
Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=7de84412592562e1a1b9a0f67caaaad665c9bc9b
Submitter: Jenkins
Branch: master

commit 7de84412592562e1a1b9a0f67caaaad665c9bc9b
Author: ZhaoBo <email address hidden>
Date: Thu Oct 20 15:57:07 2016 +0800

    Fix fwaas_v2 driver internal error when restart l3 agent

    The fwaas_v2 driver could not process the input 'apply_list'
    just include port_id. The expected input should be like
    [(router_info object, port_id),..]

    This patch fixes the l3 agent sync process with server. Make the
    input value meet the required.

    Change-Id: I7891b6c65524afd9eec89aec06d1e6bf6b0f0fae
    Closes-Bug: #1635156

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron-fwaas 10.0.0.0b1

This issue was fixed in the openstack/neutron-fwaas 10.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.