Ubuntu
neutron package

l3-agent unable to parse output from ip netns list (iproute2 >= 4.0)

Bug #1497309 reported by James Page
30
This bug affects 4 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
James Page
Kilo
Fix Released
Undecided
Unassigned
neutron 2015.1.3
neutron (Ubuntu)
Fix Released
Undecided
James Page

Bug Description

When run through sudo, ip netns has some extra output on Ubuntu wily:

$ sudo ip netns
qdhcp-35fc068a-750d-4add-b1d2-af392dbd8790 (id: 1)
qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)

and from l3-agent:

2015-09-18 14:15:47.889 26554 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 59f744cafcab474baee68232c4cf70e9 _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392
2015-09-18 14:15:47.895 26554 DEBUG neutron.agent.l3.agent [-] Starting _process_routers_loop _process_routers_loop /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:509
2015-09-18 14:15:47.896 26554 DEBUG oslo_service.loopingcall [-] Fixed interval looping call 'neutron.agent.l3.agent.L3NATAgentWithStateReport._report_state' sleeping for 29.93 seconds _run_loop /usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py:121
2015-09-18 14:15:47.916 26554 DEBUG neutron.agent.l3.agent [-] Starting periodic_sync_routers_task - fullsync:True periodic_sync_routers_task /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:521
2015-09-18 14:15:47.953 26554 DEBUG neutron.agent.linux.utils [-]
Command: ['ip', 'netns', 'list']
Exit code: 0
 execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:151
2015-09-18 14:15:47.954 26554 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is d04be6386ef7495ebeb3cb656fb330a8 _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392
2015-09-18 14:15:48.268 26554 DEBUG neutron.agent.l3.agent [-] Processing :[{u'status': u'ACTIVE', u'_interfaces': [{u'status': u'DOWN', u'subnets': [{u'ipv6_ra_mode': None, u'cidr': u'192.168.21.0/24', u'gateway_ip': u'192.168.21.1', u'id': u'242acaef-22b7-4044-ab1f-788bd31ad1da', u'subnetpool_id': None}], u'binding:host_id': u'juju-devel3-machine-12', u'name': u'', u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'35fc068a-750d-4add-b1d2-af392dbd8790', u'dns_name': u'', u'extra_dhcp_opts': [], u'mac_address': u'fa:16:3e:3f:4a:90', u'binding:vif_details': {u'port_filter': True, u'ovs_hybrid_plug': True}, u'binding:vif_type': u'ovs', u'device_owner': u'network:router_interface', u'tenant_id': u'85d6051d040347e5bbd689348405faf0', u'extra_subnets': [], u'binding:profile': {}, u'binding:vnic_type': u'normal', u'fixed_ips': [{u'subnet_id': u'242acaef-22b7-4044-ab1f-788bd31ad1da', u'prefixlen': 24, u'ip_address': u'192.168.21.1'}], u'id': u'bfc7b6e2-29a7-428b-b7fd-0675e9bf5df8', u'security_groups': [], u'device_id': u'49c6d7b1-8399-4944-81ad-093b6e786db0'}], u'enable_snat': True, u'ha_vr_id': 0, u'gw_port_host': None, u'gw_port_id': u'7cca3db9-5502-43be-b193-59d523e3c81b', u'admin_state_up': True, u'tenant_id': u'85d6051d040347e5bbd689348405faf0', u'gw_port': {u'status': u'DOWN', u'subnets': [{u'ipv6_ra_mode': None, u'cidr': u'10.5.0.0/16', u'gateway_ip': u'10.5.0.1', u'id': u'431e736d-04d1-4817-b3c1-c9579b4b51f0', u'subnetpool_id': None}], u'binding:host_id': u'juju-devel3-machine-12', u'name': u'', u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'00ef84fe-880a-45c6-ae3d-967089b161ef', u'dns_name': u'', u'extra_dhcp_opts': [], u'mac_address': u'fa:16:3e:6f:24:28', u'binding:vif_details': {u'port_filter': True, u'ovs_hybrid_plug': True}, u'binding:vif_type': u'ovs', u'device_owner': u'network:router_gateway', u'tenant_id': u'', u'extra_subnets': [], u'binding:profile': {}, u'binding:vnic_type': u'normal', u'fixed_ips': [{u'subnet_id': u'431e736d-04d1-4817-b3c1-c9579b4b51f0', u'prefixlen': 16, u'ip_address': u'10.5.150.0'}], u'id': u'7cca3db9-5502-43be-b193-59d523e3c81b', u'security_groups': [], u'device_id': u'49c6d7b1-8399-4944-81ad-093b6e786db0'}, u'distributed': False, u'_snat_router_interfaces': [], u'_floatingip_agent_interfaces': [], u'_floatingips': [{u'router_id': u'49c6d7b1-8399-4944-81ad-093b6e786db0', u'status': u'DOWN', u'tenant_id': u'85d6051d040347e5bbd689348405faf0', u'floating_network_id': u'00ef84fe-880a-45c6-ae3d-967089b161ef', u'fixed_ip_address': u'192.168.21.3', u'floating_ip_address': u'10.5.150.1', u'port_id': u'a5d45770-98ea-4a2b-b839-5274e783abca', u'id': u'7067f627-91ea-4917-ab31-48570d3c397b'}], u'routes': [], u'external_gateway_info': {u'network_id': u'00ef84fe-880a-45c6-ae3d-967089b161ef', u'enable_snat': True, u'external_fixed_ips': [{u'subnet_id': u'431e736d-04d1-4817-b3c1-c9579b4b51f0', u'ip_address': u'10.5.150.0'}]}, u'ha': False, u'id': u'49c6d7b1-8399-4944-81ad-093b6e786db0', u'name': u'provider-router'}] fetch_and_sync_all_routers /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:555
2015-09-18 14:15:48.272 26554 DEBUG neutron.agent.l3.agent [-] periodic_sync_routers_task successfully completed fetch_and_sync_all_routers /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:570
2015-09-18 14:15:48.273 26554 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/external/pids/49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0).pid get_value_from_file /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:240
2015-09-18 14:15:48.274 26554 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/external/pids/49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0).pid get_value_from_file /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:240
2015-09-18 14:15:48.276 26554 DEBUG neutron.agent.linux.external_process [-] No process started for 49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0) disable /usr/lib/python2.7/dist-packages/neutron/agent/linux/external_process.py:118
2015-09-18 14:15:48.287 26554 DEBUG neutron.agent.l3.agent [-] Starting router update for 49c6d7b1-8399-4944-81ad-093b6e786db0, action None, priority 1 _process_router_update /usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:456
2015-09-18 14:15:48.326 26554 DEBUG neutron.agent.linux.utils [-]
Command: ['ip', '-o', 'link', 'show', 'br-ex']
Exit code: 0
 execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:151
2015-09-18 14:15:48.329 26554 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 1502ca45d5e844e3937f095d9af27875 _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392
2015-09-18 14:15:48.359 26554 ERROR neutron.agent.linux.utils [-]
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
Exit code: 1
Stdin:
Stdout:
Stderr: Cannot open network namespace "qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)": No such file or directory

2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager [-] Failed to destroy stale namespace <neutron.agent.l3.namespaces.RouterNamespace object at 0x7f6d668b7110>
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Traceback (most recent call last):
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/namespace_manager.py", line 143, in _cleanup
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager ns.delete()
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/namespaces.py", line 101, in delete
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager for d in ns_ip.get_devices(exclude_loopback=True):
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 112, in get_devices
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager log_fail_as_error=self.log_fail_as_error
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 154, in execute
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager raise RuntimeError(m)
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager RuntimeError:
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Exit code: 1
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Stdin:
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Stdout:
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager Stderr: Cannot open network namespace "qrouter-49c6d7b1-8399-4944-81ad-093b6e786db0 (id: 0)": No such file or directory
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager
2015-09-18 14:15:48.360 26554 ERROR neutron.agent.l3.namespace_manager

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: neutron-l3-agent 2:7.0.0~b3-0ubuntu2
ProcVersionSignature: Ubuntu 4.2.0-7.7-generic 4.2.0
Uname: Linux 4.2.0-7-generic x86_64
ApportVersion: 2.18.1-0ubuntu1
Architecture: amd64
Date: Fri Sep 18 14:16:35 2015
JournalErrors:
 -- Logs begin at Fri 2015-09-18 14:01:29 UTC, end at Fri 2015-09-18 14:16:43 UTC. --
 Sep 18 14:06:25 hostname kernel: conntrack: generic helper won't handle protocol 47. Please consider loading the specific helper module.
PackageArchitecture: all
SourcePackage: neutron
UpgradeStatus: No upgrade log present (probably fresh install)

See original description
Revision history for this message
James Page (james-page) wrote :
James Page (james-page)
description: updated
summary: - l3-agent unable to parse output from ip netns list
+ l3-agent unable to parse output from ip netns list (iproute2 >= 4.0)
James Page (james-page)
Changed in neutron (Ubuntu):
assignee: nobody → James Page (james-page)
Changed in neutron:
assignee: nobody → James Page (james-page)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/225206

Changed in neutron:
status: New → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:7.0.0~b3-0ubuntu3

---------------
neutron (2:7.0.0~b3-0ubuntu3) wily; urgency=medium

  * d/p/iproute2-compat.patch: Cherry pick inflight fix for iproute2 >= v4
    compatibility (LP: #1497309).
  * d/p/*: Refresh.

 -- James Page <email address hidden> Mon, 21 Sep 2015 09:29:10 +0100

Changed in neutron (Ubuntu):
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/225206
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3aefdf4de76fdcdc02093bc631e339f9ecd4c707
Submitter: Jenkins
Branch: master

commit 3aefdf4de76fdcdc02093bc631e339f9ecd4c707
Author: James Page <email address hidden>
Date: Fri Sep 18 16:38:47 2015 +0100

    Add compatibility with iproute2 >= 4.0

    The ip netns list command adds additional id data in more recent
    versions of iproute2 of the format:

      qdhcp-35fc068a-750d-4add-b1d2-af392dbd8790 (id: 1)

    Update parsing to deal with old and new formats.

    Change-Id: I0d3fc4262284172f5ad31e4f2f78ae1fb33b4228
    Closes-Bug: 1497309

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/neutron 8.0.0.0b1

This issue was fixed in the openstack/neutron 8.0.0.0b1 development milestone.

Doug Hellmann (doug-hellmann)
Changed in neutron:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/258493

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/258503

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/258503
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d65576d1665c3b7a8d6b2e23749a3c703ee79c76
Submitter: Jenkins
Branch: stable/kilo

commit d65576d1665c3b7a8d6b2e23749a3c703ee79c76
Author: James Page <email address hidden>
Date: Fri Sep 18 16:38:47 2015 +0100

    Add compatibility with iproute2 >= 4.0

    The ip netns list command adds additional id data in more recent
    versions of iproute2 of the format:

      qdhcp-35fc068a-750d-4add-b1d2-af392dbd8790 (id: 1)

    Update parsing to deal with old and new formats.

    Conflicts:
            neutron/tests/functional/agent/test_l3_agent.py

    Change-Id: I0d3fc4262284172f5ad31e4f2f78ae1fb33b4228
    Closes-Bug: 1497309
    (cherry picked from commit 3aefdf4de76fdcdc02093bc631e339f9ecd4c707)

tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/liberty)

Reviewed: https://review.openstack.org/258493
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1d8aff33a6b153c54760d2a141128ef4c045308e
Submitter: Jenkins
Branch: stable/liberty

commit 1d8aff33a6b153c54760d2a141128ef4c045308e
Author: James Page <email address hidden>
Date: Fri Sep 18 16:38:47 2015 +0100

    Add compatibility with iproute2 >= 4.0

    The ip netns list command adds additional id data in more recent
    versions of iproute2 of the format:

      qdhcp-35fc068a-750d-4add-b1d2-af392dbd8790 (id: 1)

    Update parsing to deal with old and new formats.

    Change-Id: I0d3fc4262284172f5ad31e4f2f78ae1fb33b4228
    Closes-Bug: 1497309
    (cherry picked from commit 3aefdf4de76fdcdc02093bc631e339f9ecd4c707)

tags: added: in-stable-liberty
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/neutron 7.0.2

This issue was fixed in the openstack/neutron 7.0.2 release.

Revision history for this message
hertzyang (hertz-7) wrote :

how to get neutron 7.0.2 , thanks.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.